please create a MAMBA state space model in flax nnx,
- User
please create a MAMBA state space model in flax nnx,
- Assistant
Below is an implementation of a Mamba-inspired state space model (SSM) using Flax NNX. This implementation captures the core ideas of the Mamba architecture, including a selective SSM with data-dependent discretization parameters (e.g., delta and B). For simplicity, I've assumed a diagonal continuous A matrix for efficiency, as full matrix exponentiation per time step can be computationally expensive. The model includes:
- A selective projection layer that computes the SSM input (u_t), discretization parameter (\Delta_t), and selective B parameter per time step.
- A discretized SSM update using zero-order hold (ZOH) approximation.
- Support for sequence modeling via
jax.lax.scan.
This is a simplified version and may not include all optimizations from the original Mamba paper (e.g., hardware-aware convolutions or advanced discretization approximations). You can extend it by adding residual connections, layer normalization, or stacking multiple layers.
Code
import flax.nnx as nnx import jax import jax.numpy as jnp from typing import Any class MambaSSM(nnx.Module): """A Mamba-inspired state space model implemented in Flax NNX. This model applies a selective state space model to input sequences. It uses a diagonal continuous A matrix for efficiency and includes data-dependent discretization via a selective projection. Args: input_size: Size of the input features. ssm_state_size: Size of the SSM state (hidden state dimension). ssm_input_size: Size of the SSM input (after projection). output_size: Size of the output features per time step. rngs: Random number generators for parameter initialization. """ ssm_state_size: int ssm_input_size: int output_size: int def __init__(self, input_size: int, ssm_state_size: int, ssm_input_size: int, output_size: int, rngs: nnx.Rngs): super().__init__() self.ssm_state_size = ssm_state_size self.ssm_input_size = ssm_input_size self.output_size = output_size # Continuous A diagonal, initialized with negative values for stability self.A_cont_diag = nnx.Param(nnx.random.uniform(rngs.params(), (ssm_state_size,), minval=-0.5, maxval=-0.1)) # C matrix for output projection, shape (output_size, ssm_state_size) self.C = nnx.Param(nnx.initializers.lecun_normal()(rngs.params(), (output_size, ssm_state_size))) # D matrix for skip connection, often small or zero, shape (output_size, ssm_input_size) self.D = nnx.Param(jnp.zeros((output_size, ssm_input_size))) # Selective projection: maps input to u_t, delta_t, and flattened B_cont_t # Output size: ssm_input_size (u_t) + 1 (delta_t) + ssm_state_size * ssm_input_size (B_cont flattened) selective_output_size = ssm_input_size + 1 + (ssm_state_size * ssm_input_size) self.selective_proj = nnx.Linear(input_size, selective_output_size, rngs=rngs) def __call__(self, inputs: jnp.ndarray) -> jnp.ndarray: """Forward pass for the Mamba SSM. Args: inputs: Input sequence of shape (sequence_length, batch_size, input_size). Returns: Output sequence of shape (sequence_length, batch_size, output_size). """ batch_size = inputs.shape[1] initial_state = jnp.zeros((batch_size, self.ssm_state_size)) # Initial state is zero def scan_fn(carry, x_t): # x_t: input at time t, shape (batch_size, input_size) selective_output = self.selective_proj(x_t) # Shape (batch_size, selective_output_size) # Split the selective output u_t = selective_output[:, :self.ssm_input_size] # Shape (batch_size, ssm_input_size) delta_t = selective_output[:, self.ssm_input_size : self.ssm_input_size + 1] # Shape (batch_size, 1) B_cont_flat = selective_output[:, self.ssm_input_size + 1 :] # Shape (batch_size, ssm_state_size * ssm_input_size) B_cont_t = jnp.reshape(B_cont_flat, (batch_size, self.ssm_state_size, self.ssm_input_size)) # Shape (batch_size, ssm_state_size, ssm_input_size) # Discretize A and B using ZOH approximation A_bar_diag = jnp.exp(delta_t * self.A_cont_diag[None, :]) # Shape (batch_size, ssm_state_size) exp_A_delta = A_bar_diag # Reused for B_bar computation B_bar = ((exp_A_delta - 1.0) / self.A_cont_diag[None, :])[:, :, None] * B_cont_t # Shape (batch_size, ssm_state_size, ssm_input_size) # State update: x_new = A_bar * carry + B_bar @ u_t x_new = A_bar_diag * carry + jnp.einsum('bsi,bi->bs', B_bar, u_t) # Shape (batch_size, ssm_state_size) # Output computation: y_t = C @ x_new + D @ u_t y_t = jnp.matmul(x_new, self.C.T) + jnp.matmul(u_t, self.D.T) # Shape (batch_size, output_size) return x_new, y_t # Return new state and output # Apply scan over the sequence dimension _, ys = jax.lax.scan(scan_fn, initial_state, inputs) # ys shape (sequence_length, batch_size, output_size) return ys ### Explanation - **Parameters**: - `A_cont_diag`: A diagonal matrix for the continuous SSM, initialized with negative values to ensure stability (eigenvalues have negative real parts). - `C` and `D`: Matrices for the output computation in the SSM equations. - `selective_proj`: A linear layer that computes data-dependent parameters (\(u_t\), \(\Delta_t\), and \(B_{\text{cont},t}\)) for each time step. This makes the model "selective" as in Mamba. - **Forward Pass**: - The model uses `jax.lax.scan` to iterate over the sequence, updating the state and computing outputs at each step. - Discretization is done per time step using the ZOH method, with \(\Delta_t\) and \(B_t\) being data-dependent. - The state update and output computation follow the standard SSM equations: - \(x_t = \bar{A} x_{t-1} + \bar{B} u_t\) - \(y_t = C x_t + D u_t\) - The scan function handles batched inputs for efficiency. - **Assumptions and Simplifications**: - Diagonal \(A\) is used for computational efficiency. In the full Mamba model, \(A\) might be a full matrix with specific parameterizations. - \(D\) is initialized to zero but can be learned. - The selective projection outputs flattened \(B_{\text{cont},t}\), which is reshaped in the scan function to handle matrix multiplications. - This implementation assumes the input is a sequence tensor. If you need to handle non-sequence inputs or add more features (e.g., residual connections), you can modify the `__call__` method. - **Usage Example**: ```python import jax import jax.numpy as jnp # Define model parameters input_size = 64 # Example input feature size ssm_state_size = 16 # SSM state dimension ssm_input_size = 1 # SSM input size (simplified to scalar for efficiency) output_size = 64 # Output feature size # Create RNGs and initialize the model rngs = nnx.Rngs(0) # Seed for reproducibility model = MambaSSM(input_size, ssm_state_size, ssm_input_size, output_size, rngs) # Generate dummy input data: shape (sequence_length, batch_size, input_size) key = jax.random.key(0) inputs = jax.random.normal(key, (10, 32, input_size)) # Sequence length 10, batch size 32 # Run the model outputs = model(inputs) # Shape (10, 32, output_size) print(outputs.shape) # Should print (10, 32, 64)This code should work with JAX and Flax NNX. If you need enhancements (e.g., handling non-diagonal (A), advanced discretization, or integration into a larger transformer-like model), let me know!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.