Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-sonnet-4-20250514-thinking-32k· AnthropicHugging Face LMArena Human Preference 140K

I'm reading a post advocating for development of 'Wise AI advisors', who advise humans…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219-thinking-32k’s answer.
  1. User

    I'm reading a post advocating for development of 'Wise AI advisors', who advise humans on how to make better decisions, and therefore help in making conventional AI development more aligned. In response to a question 'I'm mostly worried about inner alignment. Does this proposal address this?', they wrote: 'Yes 😊.

    Inner alignment is an extremely challenging issue. If only we had some... Wise AI Advisors to help us navigate this problem.

    "But surely this doesn't solve the problem", you might object, "this assumes that these advisors are aligned themselves".

    This takes us to the other way this approach could potentially help address inner alignment.

    I suspect that the wise AI advisors approach has less exposure to these kinds of risks as it allows us to achieve certain goals at a lower base model capability level.

    • Firstly, wise people don't always have huge amounts of intellectual firepower. So I believe that we will be able to achieve a lot without necessarily using the most powerful models. • Secondly, the approach of combined human-AI teams allows the humans to compensate for any weaknesses present in the AIs.

    In summary, this approach might help in two ways: by reducing exposure and advising us on how to navigate the issue.'

    I'm not comfortable that this argument is tight, but I can't put my finger on why. What are your thoughts?

  2. Claude Assistant

    Your instinct is right - there are several issues with this argument that make it feel unsatisfying.

    The most fundamental problem is circular reasoning. The proposal essentially says "we'll solve inner alignment by building aligned AI advisors to help us with inner alignment." But this just pushes the problem back one level - how do we ensure these AI advisors are actually aligned? The author acknowledges this circularity but doesn't resolve it.

    The "lower capability" argument is also questionable. Inner alignment problems aren't just about raw intellectual power - they can emerge at various capability levels when systems develop internal objectives that differ from their training objectives. If these AI advisors are sophisticated enough to give genuinely useful advice about complex AI development decisions, they're already quite capable systems where inner alignment could be a concern.

    There's also an assumption about human oversight that may not hold. The argument relies on humans being able to detect when AI advisors are giving problematic advice and compensate for their weaknesses. But inner alignment failures could be subtle - the AI might be optimizing for something slightly different than intended in ways that aren't obviously wrong to human observers. If humans could easily catch these problems, inner alignment would be much less concerning in the first place.

    Finally, there's a selection pressure issue - if we're using AI systems to advise on AI development, there could be implicit pressures favoring advisors that recommend building more capable AI systems, potentially including more capable versions of themselves.

    The proposal essentially assumes away the hard part of the alignment problem rather than solving it. It's a bit like saying "we'll solve nuclear safety by building safe nuclear reactors to advise us on nuclear safety" - it doesn't address how we build the first safe reactor.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.