Skip to content
TheChatArchiveThe Chat Archive
grok-3-mini-high· xAIHugging Face LMArena Human Preference 140K

please create a MAMBA state space model in flax nnx,

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-pro’s answer.
  1. User

    please create a MAMBA state space model in flax nnx,

  2. Assistant

    Below is an implementation of a Mamba-inspired state space model (SSM) using Flax NNX. This implementation captures the core ideas of the Mamba architecture, including a selective SSM with data-dependent discretization parameters (e.g., delta and B). For simplicity, I've assumed a diagonal continuous A matrix for efficiency, as full matrix exponentiation per time step can be computationally expensive. The model includes:

    • A selective projection layer that computes the SSM input (u_t), discretization parameter (\Delta_t), and selective B parameter per time step.
    • A discretized SSM update using zero-order hold (ZOH) approximation.
    • Support for sequence modeling via jax.lax.scan.

    This is a simplified version and may not include all optimizations from the original Mamba paper (e.g., hardware-aware convolutions or advanced discretization approximations). You can extend it by adding residual connections, layer normalization, or stacking multiple layers.

    Code

    import flax.nnx as nnx
    import jax
    import jax.numpy as jnp
    from typing import Any
    
    class MambaSSM(nnx.Module):
        """A Mamba-inspired state space model implemented in Flax NNX.
    
        This model applies a selective state space model to input sequences.
        It uses a diagonal continuous A matrix for efficiency and includes
        data-dependent discretization via a selective projection.
    
        Args:
            input_size: Size of the input features.
            ssm_state_size: Size of the SSM state (hidden state dimension).
            ssm_input_size: Size of the SSM input (after projection).
            output_size: Size of the output features per time step.
            rngs: Random number generators for parameter initialization.
        """
        ssm_state_size: int
        ssm_input_size: int
        output_size: int
    
        def __init__(self, input_size: int, ssm_state_size: int, ssm_input_size: int, output_size: int, rngs: nnx.Rngs):
            super().__init__()
            self.ssm_state_size = ssm_state_size
            self.ssm_input_size = ssm_input_size
            self.output_size = output_size
    
            # Continuous A diagonal, initialized with negative values for stability
            self.A_cont_diag = nnx.Param(nnx.random.uniform(rngs.params(), (ssm_state_size,), minval=-0.5, maxval=-0.1))
    
            # C matrix for output projection, shape (output_size, ssm_state_size)
            self.C = nnx.Param(nnx.initializers.lecun_normal()(rngs.params(), (output_size, ssm_state_size)))
    
            # D matrix for skip connection, often small or zero, shape (output_size, ssm_input_size)
            self.D = nnx.Param(jnp.zeros((output_size, ssm_input_size)))
    
            # Selective projection: maps input to u_t, delta_t, and flattened B_cont_t
            # Output size: ssm_input_size (u_t) + 1 (delta_t) + ssm_state_size * ssm_input_size (B_cont flattened)
            selective_output_size = ssm_input_size + 1 + (ssm_state_size * ssm_input_size)
            self.selective_proj = nnx.Linear(input_size, selective_output_size, rngs=rngs)
    
        def __call__(self, inputs: jnp.ndarray) -> jnp.ndarray:
            """Forward pass for the Mamba SSM.
    
            Args:
                inputs: Input sequence of shape (sequence_length, batch_size, input_size).
    
            Returns:
                Output sequence of shape (sequence_length, batch_size, output_size).
            """
            batch_size = inputs.shape[1]
            initial_state = jnp.zeros((batch_size, self.ssm_state_size))  # Initial state is zero
    
            def scan_fn(carry, x_t):
                # x_t: input at time t, shape (batch_size, input_size)
                selective_output = self.selective_proj(x_t)  # Shape (batch_size, selective_output_size)
    
                # Split the selective output
                u_t = selective_output[:, :self.ssm_input_size]  # Shape (batch_size, ssm_input_size)
                delta_t = selective_output[:, self.ssm_input_size : self.ssm_input_size + 1]  # Shape (batch_size, 1)
                B_cont_flat = selective_output[:, self.ssm_input_size + 1 :]  # Shape (batch_size, ssm_state_size * ssm_input_size)
                B_cont_t = jnp.reshape(B_cont_flat, (batch_size, self.ssm_state_size, self.ssm_input_size))  # Shape (batch_size, ssm_state_size, ssm_input_size)
    
                # Discretize A and B using ZOH approximation
                A_bar_diag = jnp.exp(delta_t * self.A_cont_diag[None, :])  # Shape (batch_size, ssm_state_size)
                exp_A_delta = A_bar_diag  # Reused for B_bar computation
                B_bar = ((exp_A_delta - 1.0) / self.A_cont_diag[None, :])[:, :, None] * B_cont_t  # Shape (batch_size, ssm_state_size, ssm_input_size)
    
                # State update: x_new = A_bar * carry + B_bar @ u_t
                x_new = A_bar_diag * carry + jnp.einsum('bsi,bi->bs', B_bar, u_t)  # Shape (batch_size, ssm_state_size)
    
                # Output computation: y_t = C @ x_new + D @ u_t
                y_t = jnp.matmul(x_new, self.C.T) + jnp.matmul(u_t, self.D.T)  # Shape (batch_size, output_size)
    
                return x_new, y_t  # Return new state and output
    
            # Apply scan over the sequence dimension
            _, ys = jax.lax.scan(scan_fn, initial_state, inputs)  # ys shape (sequence_length, batch_size, output_size)
            return ys
    
    ### Explanation
    
    - **Parameters**:
      - `A_cont_diag`: A diagonal matrix for the continuous SSM, initialized with negative values to ensure stability (eigenvalues have negative real parts).
      - `C` and `D`: Matrices for the output computation in the SSM equations.
      - `selective_proj`: A linear layer that computes data-dependent parameters (\(u_t\), \(\Delta_t\), and \(B_{\text{cont},t}\)) for each time step. This makes the model "selective" as in Mamba.
    
    - **Forward Pass**:
      - The model uses `jax.lax.scan` to iterate over the sequence, updating the state and computing outputs at each step.
      - Discretization is done per time step using the ZOH method, with \(\Delta_t\) and \(B_t\) being data-dependent.
      - The state update and output computation follow the standard SSM equations:
        - \(x_t = \bar{A} x_{t-1} + \bar{B} u_t\)
        - \(y_t = C x_t + D u_t\)
      - The scan function handles batched inputs for efficiency.
    
    - **Assumptions and Simplifications**:
      - Diagonal \(A\) is used for computational efficiency. In the full Mamba model, \(A\) might be a full matrix with specific parameterizations.
      - \(D\) is initialized to zero but can be learned.
      - The selective projection outputs flattened \(B_{\text{cont},t}\), which is reshaped in the scan function to handle matrix multiplications.
      - This implementation assumes the input is a sequence tensor. If you need to handle non-sequence inputs or add more features (e.g., residual connections), you can modify the `__call__` method.
    
    - **Usage Example**:
      ```python
      import jax
      import jax.numpy as jnp
    
      # Define model parameters
      input_size = 64  # Example input feature size
      ssm_state_size = 16  # SSM state dimension
      ssm_input_size = 1  # SSM input size (simplified to scalar for efficiency)
      output_size = 64  # Output feature size
    
      # Create RNGs and initialize the model
      rngs = nnx.Rngs(0)  # Seed for reproducibility
      model = MambaSSM(input_size, ssm_state_size, ssm_input_size, output_size, rngs)
    
      # Generate dummy input data: shape (sequence_length, batch_size, input_size)
      key = jax.random.key(0)
      inputs = jax.random.normal(key, (10, 32, input_size))  # Sequence length 10, batch size 32
    
      # Run the model
      outputs = model(inputs)  # Shape (10, 32, output_size)
      print(outputs.shape)  # Should print (10, 32, 64)
    

    This code should work with JAX and Flax NNX. If you need enhancements (e.g., handling non-diagonal (A), advanced discretization, or integration into a larger transformer-like model), let me know!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.