how do mixture of experts LLMs work?
- User
how do mixture of experts LLMs work?
- Assistant
Mixture-of-Experts (MoE) in large-language models is a way to massively increase model capacity without blowing up inference cost, by activating only a small “mixture” of sub-modules (experts) per token. Here’s how it works:
1. Overall Architecture
- Shared layers
• Embeddings, attention blocks, … - MoE layers (interleaved)
• Instead of one big feed-forward network (FFN), you have E parallel FFNs (“experts”)
• A lightweight gating network routes each token to a subset of experts
Diagram (simplified):
┌────────────┐ Input →│ Self-Attn │ └────────────┘ ↓ ┌────────────┐ │ Gating │──► Expert #3 ──┐ └────────────┘ ├─ sum & norm → Output ↓ │ …│ └─ Expert #17 ──┘
2. The Gating Network
- Takes the token’s representation (x) as input.
- Computes logits (g_i = w_i^\top x) for each expert (i=1…E).
- Applies a Softmax (or Top-k Softmax) to pick the strongest experts:
[ p_i = \frac{\exp(g_i)}{\sum_j \exp(g_j)}, \quad\text{then keep top-k experts per token.} ] - Routes (x) to those k experts, weighting their outputs by (p_i).
3. Sparse, Conditional Computation
- Sparse activation: only k≪E experts fire → big compute savings.
- Conditional: each token sees different experts, letting the model specialize.
Benefits
• More total parameters (billions of expert weights)
• Inference cost ∝ active experts (small)
• Experts can specialize (linguistic phenomena, domain, etc.)
4. Load Balancing & Training
Challenge: avoid “expert collapse” (all tokens choose the same expert).
Solution: add auxiliary losses to encourage even routing:
[ \mathcal{L}{\text{load}} = \lambda \sum{i=1}^E \text{Load}(i) \log \text{Load}(i), ]
where Load(i) is the fraction of tokens routed to expert i.
5. Real-World MoE Models
- Google Switch Transformer: Top-1 gating, ~1T total params, 1/3 compute.
- GLaM (Google): Mixture of Experts with Top-k routing (k=2).
- GShard: Scales MoE across TPU pods.
6. Pros & Cons
Pros
• Huge model capacity for free
• Expert specialization (e.g. language, style, knowledge)Cons
• More complex implementation (distributed routing)
• Communication overhead (sending tokens to experts on different devices)
• Load balancing tuning
7. What Next?
- Research on better routing (dynamic k, reinforcement learning)
- Fusing experts for faster inference
- Domain-adaptive expert fine-tuning
Anything in particular you’d like to dive deeper into? 😊
- Shared layers
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.