Skip to content
The Chat Archive
sonar-reasoning-pro-high LMArena Search Arena 24K

What’s llama4’s drama?

  1. User

    What’s llama4’s drama?

  2. Assistant

    Llama 4, Meta's latest family of multimodal AI models released in April 2025, has been surrounded by controversy since its launch. The drama centers around several key issues:

    Benchmark Manipulation Allegations

    The most significant controversy involves allegations that Meta misrepresented Llama 4's capabilities in benchmarks:

    • LM Arena controversy: Meta was accused of using a different model version for benchmarks than what was released to the public. LM Arena had to issue a statement saying "meta's interpretation of our policy did not match what we expect from model providers"[1].

    • Evidence emerged showing that "Llama 4 Maverick Experimental" (used for benchmarks) produced significantly different and more comprehensive responses than the publicly released version[6].

    • AI professor Ethan Molik compared answers from the arena model to the released model and found "they aren't close at all," suggesting benchmark manipulation[6].

    Disappointing Performance

    Despite impressive claims, many users reported underwhelming real-world performance:

    • Reddit users expressed significant disappointment, with one stating that "Llama-4-Maverick, the 402B parameter model, performs roughly on par with Qwen-QwQ-32B in terms of coding ability"[7].

    • Many users questioned why they would use such a large model when smaller competitors performed similarly or better[7].

    • Users described it as "foolish" and "atrocious" for coding tasks, with some suggesting it felt like "a late April Fools' joke"[7].

    Internal Issues at Meta

    Reports suggest Meta faced significant challenges developing Llama 4:

    • An anonymous post claimed Meta's Gen AI organization went into "panic mode" after DeepSeek V3's release, which reportedly outperformed Llama 4 despite a much smaller training budget[6].

    • There were allegations that "engineers are moving frantically to dissect Deep Seek and copy anything and everything we can from it"[6].

    • Another report claimed that "despite repeated training efforts, the internal models performance still fall short of open-source state-of-the-art benchmarks"[6].

    Meta's Response

    Meta addressed some of these concerns in an official statement:

    • They acknowledged "reports of mixed quality across different services" but attributed this to implementation issues that would take "several days for the public implementations to get dialed in"[6].

    • They directly denied benchmark manipulation: "we've heard claims that we trained on test sets and that's simply not true and we would never do that"[6].

    • They reaffirmed their belief that "Llama 4 models are a significant advancement"[6].

    Technical Reality vs. Claims

    The actual Llama 4 family includes:

    • Scout (109B total parameters, 17B active) with an unprecedented 10 million token context window[3][4].
    • Maverick (400B total parameters, 17B active) designed for advanced multimodal tasks[3][4].
    • Behemoth (2 trillion parameters), an internal model not yet publicly released[4].

    While Meta claimed these models outperform competitors on various benchmarks, the community's experience has been mixed, with some suggesting that Meta prioritized size over optimization and efficiency[7].

    This drama highlights the increasingly competitive AI landscape, where benchmark performance, real-world capabilities, and development transparency have become critical points of contention.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.