Skip to content
TheChatArchiveThe Chat Archive
o4-mini-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

how do mixture of experts LLMs work?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with o3-2025-04-16’s answer.
  1. User

    how do mixture of experts LLMs work?

  2. Assistant

    Mixture-of-Experts (MoE) in large-language models is a way to massively increase model capacity without blowing up inference cost, by activating only a small “mixture” of sub-modules (experts) per token. Here’s how it works:


    1. Overall Architecture

    1. Shared layers
      • Embeddings, attention blocks, …
    2. MoE layers (interleaved)
      • Instead of one big feed-forward network (FFN), you have E parallel FFNs (“experts”)
      • A lightweight gating network routes each token to a subset of experts

    Diagram (simplified):

           ┌────────────┐
    Input →│ Self-Attn  │
           └────────────┘
                 ↓
           ┌────────────┐
           │   Gating   │──► Expert #3 ──┐
           └────────────┘                ├─ sum & norm → Output
                 ↓                       │
                                        …│
                                        └─ Expert #17 ──┘
    

    2. The Gating Network

    • Takes the token’s representation (x) as input.
    • Computes logits (g_i = w_i^\top x) for each expert (i=1…E).
    • Applies a Softmax (or Top-k Softmax) to pick the strongest experts:
      [ p_i = \frac{\exp(g_i)}{\sum_j \exp(g_j)}, \quad\text{then keep top-k experts per token.} ]
    • Routes (x) to those k experts, weighting their outputs by (p_i).

    3. Sparse, Conditional Computation

    • Sparse activation: only k≪E experts fire → big compute savings.
    • Conditional: each token sees different experts, letting the model specialize.

    Benefits
    • More total parameters (billions of expert weights)
    • Inference cost ∝ active experts (small)
    • Experts can specialize (linguistic phenomena, domain, etc.)


    4. Load Balancing & Training

    Challenge: avoid “expert collapse” (all tokens choose the same expert).
    Solution: add auxiliary losses to encourage even routing:
    [ \mathcal{L}{\text{load}} = \lambda \sum{i=1}^E \text{Load}(i) \log \text{Load}(i), ]
    where Load(i) is the fraction of tokens routed to expert i.


    5. Real-World MoE Models

    • Google Switch Transformer: Top-1 gating, ~1T total params, 1/3 compute.
    • GLaM (Google): Mixture of Experts with Top-k routing (k=2).
    • GShard: Scales MoE across TPU pods.

    6. Pros & Cons

    Pros
    • Huge model capacity for free
    • Expert specialization (e.g. language, style, knowledge)

    Cons
    • More complex implementation (distributed routing)
    • Communication overhead (sending tokens to experts on different devices)
    • Load balancing tuning


    7. What Next?

    • Research on better routing (dynamic k, reinforcement learning)
    • Fusing experts for faster inference
    • Domain-adaptive expert fine-tuning

    Anything in particular you’d like to dive deeper into? 😊

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.