Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-search-preview· OpenAIHugging Face LMArena Search Arena 24K

Is there any real release of a Mamba language model in 2025?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-search-preview-high-loc’s answer.
  1. User

    Is there any real release of a Mamba language model in 2025?

  2. Assistant

    Yes, there have been several notable releases of Mamba-based language models in 2025. One prominent example is Llamba, a family of efficient recurrent language models introduced in February 2025. Llamba models, including Llamba-1B, Llamba-3B, and Llamba-8B, are distilled from Llama-3.x into the Mamba architecture. They achieve higher inference throughput and handle significantly larger batch sizes than Transformer-based models while maintaining comparable benchmark performance. Additionally, Llamba demonstrates the effectiveness of cross-architecture distillation using MOHAWK, achieving these results with less than 0.1% of the training data typically used for models of similar size. An optimized implementation of Llamba is provided for resource-constrained devices such as smartphones and edge platforms, offering a practical and memory-efficient alternative to Transformers[1].

    Another significant release is Falcon Mamba 7B, unveiled in October 2024 by the Technology Innovation Institute (TII). This model is a pure Mamba-based architecture trained on 5.8 trillion tokens. It surpasses leading open-weight Transformer-based models such as Mistral 7B, Llama3.1 8B, and Falcon2 11B. Falcon Mamba 7B is recognized as the best-performing Mamba model at this scale, offering faster inference and requiring less memory for long-sequence generation. The model's weights are publicly available under a permissive license[2].

    Additionally, AI21 Labs released Jamba in March 2024, the world's first production-grade Mamba-based model. Jamba integrates Mamba Structured State Space model (SSM) technology with elements of the traditional Transformer architecture, offering a 256K context window and demonstrating remarkable gains in throughput and efficiency. Notably, Jamba outperforms or matches other state-of-the-art models in its size class on a wide range of benchmarks. The model is released with open weights under Apache 2.0, inviting further discoveries and optimizations[3].

    These developments highlight the growing adoption and advancement of Mamba-based architectures in the field of language modeling.

  3. User

    Is that Llamba a deployable model? What else do we know about Llamba?

  4. Assistant

    Yes, Llamba is designed for deployment, particularly on resource-constrained devices such as smartphones and edge platforms. The Llamba family comprises three models: Llamba-1B, Llamba-3B, and Llamba-8B, each tailored for efficient language processing.

    Key Features of Llamba:

    • Efficient Architecture: Llamba models are distilled from Llama-3.x into the Mamba architecture, replacing self-attention mechanisms with Mamba-2 layers. This design enhances inference throughput and allows handling of larger batch sizes compared to traditional Transformer-based models, while maintaining comparable benchmark performance[1].

    • Data Efficiency: Utilizing the MOHAWK framework for cross-architecture distillation, Llamba achieves high performance with significantly reduced training data. For instance, Llamba-3.1-8B was distilled using just 12 billion tokens—less than 0.1% of the data typically required for models of similar size[1].

    • On-Device Deployment: Optimized implementations of Llamba are available for deployment on devices with limited memory and computational resources. These implementations leverage GPU parallelism and support quantization to 4-bit precision, reducing memory usage without compromising performance[1].

    • Model Specifications:

      • Llamba-1B: 16 residual Mamba-2 blocks, hidden dimension of 2048.
      • Llamba-3B: 28 residual Mamba-2 blocks, hidden dimension of 3072.
      • Llamba-8B: 32 residual Mamba-2 blocks, hidden dimension of 4096.

    These models share the tokenizer and vocabulary of Llama-3.1, facilitating seamless integration into existing workflows[1].

    In summary, Llamba offers a practical and memory-efficient alternative to traditional Transformer models, making high-quality language models more accessible and deployable across various platforms.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.