Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-5-haiku-20241022· AnthropicHugging Face LMArena Human Preference 140K

Great. We're done with RoPE. Now, using the same principles as before, I want you to…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-sonnet-4-20250514’s answer.
  1. User

    Great. We're done with RoPE.

    Now, using the same principles as before, I want you to write the MOST clean, short, pythonic, yet efficient and "wow" implementation of a Mixture of Experts (MoE) layer in JAX & Flax. You must write 3 separate classes : MoE, Expert, and Gate. Do not write the implementation for any boilerplate layers like MLP or RMSNorm, but you are free to use them as if they were defined elsewhere.

  2. Claude Assistant

    Here's a concise, efficient, and elegant MoE implementation:

    import jax
    import jax.numpy as jnp
    import flax.linen as nn
    from typing import Callable, Sequence
    
    class Expert(nn.Module):
        """Individual expert module with configurable architecture."""
        features: Sequence[int]
        activation: Callable = nn.gelu
        
        @nn.compact
        def __call__(self, x):
            for feat in self.features[:-1]:
                x = nn.Dense(feat)(x)
                x = self.activation(x)
            return nn.Dense(self.features[-1])(x)
    
    class Gate(nn.Module):
        """Routing gate with top-k routing and soft routing."""
        num_experts: int
        k: int = 2
        temperature: float = 1.0
        
        @nn.compact
        def __call__(self, x):
            # Compute routing logits
            logits = nn.Dense(self.num_experts)(x)
            
            # Top-k routing with differentiable approximation
            if self.k > 1:
                top_k_indices = jax.lax.top_k(logits, k=self.k)[0]
                
                # Create sparse routing mask with soft top-k
                routing_weights = jax.nn.softmax(logits / self.temperature)
                mask = jnp.zeros_like(routing_weights)
                mask = mask.at[jnp.arange(x.shape[0])[:, None], top_k_indices].set(1.0)
                routing_weights *= mask
            else:
                routing_weights = jax.nn.softmax(logits)
            
            return routing_weights
    
    class MoE(nn.Module):
        """Efficient Mixture of Experts layer."""
        num_experts: int
        expert_config: Sequence[int]
        k: int = 2
        
        @nn.compact
        def __call__(self, x):
            # Create experts
            experts = [Expert(self.expert_config) for _ in range(self.num_experts)]
            gate = Gate(num_experts=self.num_experts, k=self.k)
            
            # Compute expert outputs
            expert_outputs = jnp.stack([expert(x) for expert in experts], axis=1)
            
            # Compute routing weights
            routing_weights = gate(x)
            
            # Weighted combination of expert outputs
            output = jnp.einsum('bne,bn->be', expert_outputs, routing_weights)
            
            return output
    
    # Usage example
    moe = MoE(num_experts=8, expert_config=[512, 2048, 512], k=2)
    x = jnp.ones((32, 512))  # [batch, features]
    output = moe.init_with_output(jax.random.key(0), x)[0]
    

    Key "Wow" Features:

    1. Soft Top-K Routing: Uses differentiable top-k routing with temperature
    2. Flexible Expert Configuration: Configurable expert architecture
    3. Efficient Computation: Uses jnp.einsum for fast routing
    4. Minimal Boilerplate: Concise implementation focusing on core MoE mechanics
    5. Pythonic Design: Clean, type-hinted, modular architecture

    The implementation provides:

    • Dynamic expert selection
    • Soft routing weights
    • Configurable number of experts
    • Flexible expert architectures
    • Efficient JAX/Flax computation

    Potential extensions could include adding load balancing loss or more advanced routing strategies.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.