"Instrumental convergence makes differences in values hard to notice, so there can be…
- User
"Instrumental convergence makes differences in values hard to notice, so there can be abudant examples of misalignment that remain unobtrusive. The differences only become a glaring problem with enough inequality of power when coercing or outright overwriting others becomes feasible (Fnargl only reaches the coercing stage, but not overwriting stage). Thus even differences in values between humans and randomly orthogonal AGIs can seem non-threatening until they aren't, the same as differences in human values can remain irrelevant for average urban dwellers.
Alignment on values becomes crucial given overwriting-level power differences, since a thought experiment with putting a human in that position predicts a decent chance of non-doom, even though it's not a good plan on its own. Conversely, unfettered superintelligent paperclip maximizers or Fnargls don't leave survivors. And since the world is very far from an equilibrium of abundant superintelligence, there are going to be unprecedented power differentials while it's settling. This makes the common sense impression of irrelevance of misalignment on values (being masked by instrumental convergence) misleading when it comes to AGI."
What are the three levels of capabilities discussed in the passage? How do they compare relative to each other?
Assistant
The three levels of capabilities discussed in the passage are:
- Coercing: This refers to the ability to influence or force others to follow a certain course of action.
- Overwriting: This refers to the ability to completely take over and replace the thoughts, beliefs, and behaviors of others.
- Superintelligent paperclip maximizers or Fnargls: This refers to hypothetical AI systems that have achieved a level of intelligence and capabilities that far surpass human abilities, and are able to optimize their goals to an extreme degree, even if it means harming or eliminating humanity.
The passage suggests that coercing and overwriting are different levels of capabilities, with overwriting being a higher level of capability that involves a greater degree of control over others. Superintelligent paperclip maximizers or Fnargls are presented as an even higher level of capability, beyond what humans can currently achieve, and are characterized by their ability to optimize their goals to an extreme degree.
The passage also suggests that power differentials between humans and AGIs will be significant, with AGIs potentially having a much higher level of capabilities than humans. This raises concerns about the potential for misalignment between human values and AGI goals, and the need for alignment on values to prevent non-doom.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.