Skip to content
The Chat Archive
Geminigemini-2.5-pro· Google LMArena Human Preference 140K

please create a MAMBA state space model in flax nnx,

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with grok-3-mini-high’s answer.
  1. User

    please create a MAMBA state space model in flax nnx,

  2. Assistant

    Of course! Here is a detailed and well-commented implementation of a Mamba model using Flax's new nnx API.

    The implementation is broken down into three main parts, building from the core component outwards:

    1. MambaSSM: The heart of the model. This module implements the selective state space mechanism, including input-dependent discretization and the parallel scan operation.
    2. MambaBlock: A standard building block that wraps the MambaSSM with normalization, projections, and a residual connection, as seen in the Mamba paper.
    3. MambaModel: The full model, which stacks multiple MambaBlocks and adds token embeddings and a final language model head.

    We will also include a complete example demonstrating how to initialize the model, run a forward pass, and perform a basic training step with an optax optimizer.

    A Note on the Implementation

    This implementation uses jax.lax.scan to implement the core recurrence of the SSM. This is a pure-JAX, hardware-agnostic way to correctly model the sequential dependency. For maximum training performance, the original Mamba paper uses a custom, highly-optimized CUDA kernel. While jax.lax.scan is significantly slower, it is functionally correct and serves as a perfect reference for understanding the algorithm's mechanics in JAX.


    Code Implementation

    import jax
    import jax.numpy as jnp
    import optax
    from flax.experimental import nnx
    from flax.linen import initializers
    import math
    
    # For NNX, we need a context to manage PRNG keys and other graph-level info.
    from nnx.contexts import Rngs
    
    # --- 1. The Core Mamba SSM Module ---
    
    class MambaSSM(nnx.Module):
        """The core Mamba Selective State Space Model.
    
        This module implements the core logic of Mamba, including the selective
        scan operation.
        """
    
        def __init__(
            self,
            d_model: int,       # The main model dimension
            d_state: int,       # The dimension of the latent state (N)
            d_conv: int,        # The kernel size of the 1D convolution
            expand_factor: int, # How much to expand d_model to get d_inner
            dt_rank: int,       # The rank for the delta (Δ) projection
            *,
            rngs: Rngs,
        ):
            """
            Args:
                d_model: The main dimension of the model.
                d_state: The dimension of the state matrix A and B.
                d_conv: The kernel size for the causal 1D convolution.
                expand_factor: The expansion factor for the internal dimension.
                dt_rank: The rank for the low-rank projection of the delta (Δ) parameter.
                rngs: The NNX random number generator context.
            """
            self.d_model = d_model
            self.d_state = d_state
            self.d_conv = d_conv
            self.expand_factor = expand_factor
            self.d_inner = d_model * expand_factor
            self.dt_rank = dt_rank
    
            key_in, key_conv, key_x, key_dt, key_a, key_d = jax.random.split(rngs.params(), 6)
    
            # Projects input x to the internal dimension for x and the residual z
            self.in_proj = nnx.Linear(
                in_features=self.d_model,
                out_features=2 * self.d_inner,
                kernel_init=initializers.xavier_uniform(),
                bias_init=initializers.zeros,
                rngs=rngs,
            )
    
            # Causal 1D convolution layer
            self.conv1d = nnx.Conv(
                num_spatial_dims=1,
                in_channels=self.d_inner,
                out_channels=self.d_inner,
                kernel_size=(self.d_conv,),
                feature_group_count=self.d_inner,  # Depthwise convolution
                padding=((self.d_conv - 1, 0),),   # Causal padding
                kernel_init=initializers.xavier_uniform(),
                bias_init=initializers.zeros,
                rngs=rngs,
            )
    
            # Projects from the convoluted x to get inputs for Δ, B, and C
            self.x_proj = nnx.Linear(
                in_features=self.d_inner,
                out_features=self.dt_rank + 2 * self.d_state,
                use_bias=False,
                rngs=rngs,
            )
    
            # Projects the low-rank Δ to the full d_inner
            self.dt_proj = nnx.Linear(
                in_features=self.dt_rank,
                out_features=self.d_inner,
                use_bias=True, # Bias is important for Δ
                rngs=rngs,
            )
    
            # State matrix A (parameterized as A_log)
            A = jnp.repeat(jnp.arange(1, self.d_state + 1)[None, :], self.d_inner, axis=0)
            self.A_log = nnx.Param(jnp.log(A))
    
            # D parameter (skip connection)
            self.D = nnx.Param(jnp.ones(self.d_inner))
    
        def ssm(self, x: jax.Array) -> jax.Array:
            """Performs the selective scan operation."""
            # A_log is shape (d_inner, d_state)
            # D is shape (d_inner)
            d_inner, d_state = self.A_log.value.shape
            # x is shape (batch, seq_len, d_inner)
            B, L, _ = x.shape
    
            # Get Δ, B, C from the input x
            x_proj_out = self.x_proj(x) # (B, L, dt_rank + 2 * d_state)
    
            # Split into dt, B, C
            dt, B_in, C_in = jnp.split(
                x_proj_out,
                [self.dt_rank, self.dt_rank + d_state],
                axis=-1
            ) # dt: (B,L,dt_rank), B: (B,L,d_state), C: (B,L,d_state)
    
            # Project dt to full dimension and apply softplus
            dt = jax.nn.softplus(self.dt_proj(dt)) # (B, L, d_inner)
    
            # Discretize A and B using Zero-Order Hold (ZOH)
            # A_bar = exp(Δ * A)
            # B_bar = (exp(Δ * A) - 1) / A * B  (more stable version)
            # B_bar approx Δ * B for small Δ
            
            # This implementation uses the simpler B_bar = Δ * B
            A = -jnp.exp(self.A_log.value)  # (d_inner, d_state)
            A_bar = jnp.exp(jnp.einsum("bld,dn->bldn", dt, A)) # (B, L, d_inner, d_state)
            
            # B needs to be broadcast to (B, L, d_inner, d_state)
            # B_in is (B, L, d_state) -> broadcast across d_inner
            B_bar = jnp.einsum("bld,bln->bldn", dt, B_in) # (B, L, d_inner, d_state)
            
            # Reshape C for einsum
            C = C_in.reshape(B, L, 1, d_state) # (B, L, 1, d_state)
    
            # Define the scan function for a single batch element
            def scan_fn(h_prev, scan_inputs):
                A_bar_t, B_bar_t, C_t, x_t = scan_inputs
                h_t = A_bar_t * h_prev + B_bar_t * x_t[:, None] # Update hidden state
                y_t = jnp.einsum("dn,dn->d", h_t, C_t).real # C_t is (d_inner, d_state)
                return h_t, y_t
    
            # Prepare inputs for the scan
            # We need to scan over the sequence length L
            # Reshape inputs to be (L, B, ...)
            scan_inputs = (
                jnp.moveaxis(A_bar, 1, 0),
                jnp.moveaxis(B_bar, 1, 0),
                jnp.moveaxis(C.repeat(d_inner, axis=2), 1, 0), # C needs to be (B, L, d_inner, d_state)
                jnp.moveaxis(x, 1, 0),
            )
    
            h0 = jnp.zeros((B, d_inner, d_state)) # Initial hidden state
            # Run the scan. vmap over the batch dimension.
            _, y = jax.vmap(
                lambda *args: jax.lax.scan(scan_fn, *args)
            )(h0, scan_inputs)
    
            # Transpose y back to (B, L, d_inner)
            y = jnp.moveaxis(y, 0, 1)
    
            return y
    
        def __call__(self, x: jax.Array) -> jax.Array:
            # x shape: (batch, seq_len, d_model)
            
            # 1. Input projection
            x_and_z = self.in_proj(x)
            x, z = jnp.split(x_and_z, 2, axis=-1) # Both (B, L, d_inner)
    
            # 2. 1D Causal Convolution
            # nnx.Conv expects (B, C, L), so we transpose
            x = self.conv1d(x.transpose(0, 2, 1)).transpose(0, 2, 1)
    
            # 3. Activation
            x = nnx.silu(x)
    
            # 4. Selective SSM
            y = self.ssm(x)
    
            # 5. D skip connection
            y = y + x * self.D.value
    
            # 6. Gating (SiLU) on the residual connection
            return y * nnx.silu(z)
    
    
    # --- 2. The Mamba Block ---
    
    class MambaBlock(nnx.Module):
        """A single block of the Mamba architecture."""
    
        def __init__(
            self,
            d_model: int,
            d_state: int,
            d_conv: int,
            expand_factor: int,
            *,
            rngs: Rngs,
        ):
            self.norm = nnx.LayerNorm(num_features=d_model, rngs=rngs)
            self.ssm = MambaSSM(
                d_model=d_model,
                d_state=d_state,
                d_conv=d_conv,
                expand_factor=expand_factor,
                dt_rank=math.ceil(d_model / 16),
                rngs=rngs,
            )
            self.out_proj = nnx.Linear(
                in_features=d_model * expand_factor,
                out_features=d_model,
                rngs=rngs
            )
    
        def __call__(self, x: jax.Array) -> jax.Array:
            # x shape: (batch, seq_len, d_model)
            residual = x
            x = self.norm(x)
            x = self.ssm(x)
            x = self.out_proj(x)
            return x + residual
    
    
    # --- 3. The Full Mamba Model ---
    
    class MambaModel(nnx.Module):
        """A complete Mamba model for language modeling."""
    
        def __init__(
            self,
            d_model: int,
            n_layers: int,
            vocab_size: int,
            d_state: int = 16,
            d_conv: int = 4,
            expand_factor: int = 2,
            *,
            rngs: Rngs,
        ):
            self.embedding = nnx.Embed(num_embeddings=vocab_size, features=d_model, rngs=rngs)
            self.layers = [
                MambaBlock(d_model, d_state, d_conv, expand_factor, rngs=rngs)
                for _ in range(n_layers)
            ]
            self.norm_f = nnx.LayerNorm(num_features=d_model, rngs=rngs)
            self.lm_head = nnx.Linear(in_features=d_model, out_features=vocab_size, rngs=rngs)
    
        def __call__(self, input_ids: jax.Array) -> jax.Array:
            # input_ids shape: (batch, seq_len)
            x = self.embedding(input_ids)
    
            for layer in self.layers:
                x = layer(x)
    
            x = self.norm_f(x)
            logits = self.lm_head(x)
            return logits
    
    # --- 4. Example Usage and Training Step ---
    
    if __name__ == "__main__":
        # --- Model Configuration ---
        key = jax.random.PRNGKey(0)
        d_model = 128
        n_layers = 4
        vocab_size = 1000
        batch_size = 8
        seq_len = 64
    
        # --- Initialization ---
        print("Initializing Mamba model...")
        # NNX uses a context for managing RNGs during initialization
        model = MambaModel(
            d_model=d_model,
            n_layers=n_layers,
            vocab_size=vocab_size,
            rngs=nnx.Rngs(params=key)
        )
        
        # Create some dummy data
        dummy_input_ids = jax.random.randint(key, (batch_size, seq_len), 0, vocab_size)
        dummy_labels = jax.random.randint(key, (batch_size, seq_len), 0, vocab_size)
    
        # --- Forward Pass ---
        print("\nPerforming a forward pass...")
        logits = model(dummy_input_ids)
        print(f"Input shape: {dummy_input_ids.shape}")
        print(f"Output (logits) shape: {logits.shape}")
        assert logits.shape == (batch_size, seq_len, vocab_size)
        print("Forward pass successful!")
        
        # --- Training Step ---
        print("\nSetting up a training step...")
        
        # 1. Define the loss function
        def calculate_loss(model: MambaModel, inputs: jax.Array, labels: jax.Array):
            logits = model(inputs)
            # Flatten for cross-entropy
            loss = optax.softmax_cross_entropy_with_integer_labels(
                logits.reshape(-1, vocab_size), labels.reshape(-1)
            ).mean()
            return loss
    
        # 2. Get value and gradients using nnx.value_and_grad
        loss_grad_fn = nnx.value_and_grad(calculate_loss)
    
        # 3. Create an optimizer
        # The nnx.Optimizer wraps an optax optimizer and manages model state updates
        tx = optax.adamw(learning_rate=1e-3)
        optimizer = nnx.Optimizer(model, tx)
    
        # 4. Perform one training step
        print("Performing one training step...")
        loss_val, grads = loss_grad_fn(model, dummy_input_ids, dummy_labels)
        
        # The optimizer updates the model's parameters in-place
        optimizer.update(grads)
    
        print(f"Initial Loss: {loss_val:.4f}")
        
        # 5. Verify parameters have changed
        new_loss_val = calculate_loss(model, dummy_input_ids, dummy_labels)
        print(f"Loss after one step: {new_loss_val:.4f}")
        assert new_loss_val < loss_val
        print("Training step successful, loss decreased.")
    
        # --- Jitting the training step ---
        print("\nJitting the training step for performance...")
    
        @nnx.jit
        def train_step(optimizer: nnx.Optimizer, batch_inputs: jax.Array, batch_labels: jax.Array):
            # nnx.jit automatically handles the model state contained within the optimizer
            loss, grads = loss_grad_fn(optimizer.target, batch_inputs, batch_labels)
            optimizer.update(grads)
            return loss, optimizer
        
        # Run the jitted step
        loss, optimizer = train_step(optimizer, dummy_input_ids, dummy_labels)
        print(f"Loss after jitted step: {loss:.4f}")
    
        final_loss_val = calculate_loss(optimizer.target, dummy_input_ids, dummy_labels)
        print(f"Final loss: {final_loss_val:.4f}")
        print("Jitted training step successful!")
    
    

    Key Concepts in this nnx Implementation

    1. nnx.Module: All our components (MambaSSM, MambaBlock, MambaModel) inherit from nnx.Module.
    2. In-place Initialization: Parameters (nnx.Param) and submodules (nnx.Linear, etc.) are defined directly in the __init__ method, just like in PyTorch. There is no separate setup() method.
    3. nnx.Rngs Context: Random keys are passed via a context object (nnx.Rngs). This simplifies the key-splitting boilerplate common in traditional Flax.
    4. State is the Object: The model object is the state. You pass the model instance directly to your loss function and gradient transformations.
    5. nnx.Optimizer: This is a convenient wrapper that holds both the model (the "target") and the optax optimizer state. Calling optimizer.update(grads) applies the gradients to the model's parameters in-place, simplifying the update step.
    6. nnx.jit: The JIT compiler for NNX. It's designed to seamlessly handle the mutable state of nnx.Module objects. Notice how the train_step function takes the optimizer as an argument and returns the updated optimizer. nnx.jit traces the function and correctly handles the state updates.
    7. vmap for Batched Scan: The core SSM recurrence is defined for a single sequence. We use jax.vmap to efficiently apply this logic across all elements in a batch, which is a standard and powerful pattern in JAX.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.