Is there any real release of a Mamba language model in 2025?
- User
Is there any real release of a Mamba language model in 2025?
- Assistant
Yes, there have been several notable releases of Mamba-based language models in 2025. One prominent example is Llamba, a family of efficient recurrent language models introduced in February 2025. Llamba models, including Llamba-1B, Llamba-3B, and Llamba-8B, are distilled from Llama-3.x into the Mamba architecture. They achieve higher inference throughput and handle significantly larger batch sizes than Transformer-based models while maintaining comparable benchmark performance. Additionally, Llamba demonstrates the effectiveness of cross-architecture distillation using MOHAWK, achieving these results with less than 0.1% of the training data typically used for models of similar size. An optimized implementation of Llamba is provided for resource-constrained devices such as smartphones and edge platforms, offering a practical and memory-efficient alternative to Transformers[1].
Another significant release is Falcon Mamba 7B, unveiled in October 2024 by the Technology Innovation Institute (TII). This model is a pure Mamba-based architecture trained on 5.8 trillion tokens. It surpasses leading open-weight Transformer-based models such as Mistral 7B, Llama3.1 8B, and Falcon2 11B. Falcon Mamba 7B is recognized as the best-performing Mamba model at this scale, offering faster inference and requiring less memory for long-sequence generation. The model's weights are publicly available under a permissive license[2].
Additionally, AI21 Labs released Jamba in March 2024, the world's first production-grade Mamba-based model. Jamba integrates Mamba Structured State Space model (SSM) technology with elements of the traditional Transformer architecture, offering a 256K context window and demonstrating remarkable gains in throughput and efficiency. Notably, Jamba outperforms or matches other state-of-the-art models in its size class on a wide range of benchmarks. The model is released with open weights under Apache 2.0, inviting further discoveries and optimizations[3].
These developments highlight the growing adoption and advancement of Mamba-based architectures in the field of language modeling.
- User
Is that Llamba a deployable model? What else do we know about Llamba?
- Assistant
Yes, Llamba is designed for deployment, particularly on resource-constrained devices such as smartphones and edge platforms. The Llamba family comprises three models: Llamba-1B, Llamba-3B, and Llamba-8B, each tailored for efficient language processing.
Key Features of Llamba:
-
Efficient Architecture: Llamba models are distilled from Llama-3.x into the Mamba architecture, replacing self-attention mechanisms with Mamba-2 layers. This design enhances inference throughput and allows handling of larger batch sizes compared to traditional Transformer-based models, while maintaining comparable benchmark performance[1].
-
Data Efficiency: Utilizing the MOHAWK framework for cross-architecture distillation, Llamba achieves high performance with significantly reduced training data. For instance, Llamba-3.1-8B was distilled using just 12 billion tokens—less than 0.1% of the data typically required for models of similar size[1].
-
On-Device Deployment: Optimized implementations of Llamba are available for deployment on devices with limited memory and computational resources. These implementations leverage GPU parallelism and support quantization to 4-bit precision, reducing memory usage without compromising performance[1].
-
Model Specifications:
- Llamba-1B: 16 residual Mamba-2 blocks, hidden dimension of 2048.
- Llamba-3B: 28 residual Mamba-2 blocks, hidden dimension of 3072.
- Llamba-8B: 32 residual Mamba-2 blocks, hidden dimension of 4096.
These models share the tokenizer and vocabulary of Llama-3.1, facilitating seamless integration into existing workflows[1].
In summary, Llamba offers a practical and memory-efficient alternative to traditional Transformer models, making high-quality language models more accessible and deployable across various platforms.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.