please create a MAMBA state space model in flax nnx,
- User
please create a MAMBA state space model in flax nnx,
- Assistant
Of course! Here is a detailed and well-commented implementation of a Mamba model using Flax's new
nnxAPI.The implementation is broken down into three main parts, building from the core component outwards:
MambaSSM: The heart of the model. This module implements the selective state space mechanism, including input-dependent discretization and the parallel scan operation.MambaBlock: A standard building block that wraps theMambaSSMwith normalization, projections, and a residual connection, as seen in the Mamba paper.MambaModel: The full model, which stacks multipleMambaBlocks and adds token embeddings and a final language model head.
We will also include a complete example demonstrating how to initialize the model, run a forward pass, and perform a basic training step with an
optaxoptimizer.A Note on the Implementation
This implementation uses
jax.lax.scanto implement the core recurrence of the SSM. This is a pure-JAX, hardware-agnostic way to correctly model the sequential dependency. For maximum training performance, the original Mamba paper uses a custom, highly-optimized CUDA kernel. Whilejax.lax.scanis significantly slower, it is functionally correct and serves as a perfect reference for understanding the algorithm's mechanics in JAX.
Code Implementation
import jax import jax.numpy as jnp import optax from flax.experimental import nnx from flax.linen import initializers import math # For NNX, we need a context to manage PRNG keys and other graph-level info. from nnx.contexts import Rngs # --- 1. The Core Mamba SSM Module --- class MambaSSM(nnx.Module): """The core Mamba Selective State Space Model. This module implements the core logic of Mamba, including the selective scan operation. """ def __init__( self, d_model: int, # The main model dimension d_state: int, # The dimension of the latent state (N) d_conv: int, # The kernel size of the 1D convolution expand_factor: int, # How much to expand d_model to get d_inner dt_rank: int, # The rank for the delta (Δ) projection *, rngs: Rngs, ): """ Args: d_model: The main dimension of the model. d_state: The dimension of the state matrix A and B. d_conv: The kernel size for the causal 1D convolution. expand_factor: The expansion factor for the internal dimension. dt_rank: The rank for the low-rank projection of the delta (Δ) parameter. rngs: The NNX random number generator context. """ self.d_model = d_model self.d_state = d_state self.d_conv = d_conv self.expand_factor = expand_factor self.d_inner = d_model * expand_factor self.dt_rank = dt_rank key_in, key_conv, key_x, key_dt, key_a, key_d = jax.random.split(rngs.params(), 6) # Projects input x to the internal dimension for x and the residual z self.in_proj = nnx.Linear( in_features=self.d_model, out_features=2 * self.d_inner, kernel_init=initializers.xavier_uniform(), bias_init=initializers.zeros, rngs=rngs, ) # Causal 1D convolution layer self.conv1d = nnx.Conv( num_spatial_dims=1, in_channels=self.d_inner, out_channels=self.d_inner, kernel_size=(self.d_conv,), feature_group_count=self.d_inner, # Depthwise convolution padding=((self.d_conv - 1, 0),), # Causal padding kernel_init=initializers.xavier_uniform(), bias_init=initializers.zeros, rngs=rngs, ) # Projects from the convoluted x to get inputs for Δ, B, and C self.x_proj = nnx.Linear( in_features=self.d_inner, out_features=self.dt_rank + 2 * self.d_state, use_bias=False, rngs=rngs, ) # Projects the low-rank Δ to the full d_inner self.dt_proj = nnx.Linear( in_features=self.dt_rank, out_features=self.d_inner, use_bias=True, # Bias is important for Δ rngs=rngs, ) # State matrix A (parameterized as A_log) A = jnp.repeat(jnp.arange(1, self.d_state + 1)[None, :], self.d_inner, axis=0) self.A_log = nnx.Param(jnp.log(A)) # D parameter (skip connection) self.D = nnx.Param(jnp.ones(self.d_inner)) def ssm(self, x: jax.Array) -> jax.Array: """Performs the selective scan operation.""" # A_log is shape (d_inner, d_state) # D is shape (d_inner) d_inner, d_state = self.A_log.value.shape # x is shape (batch, seq_len, d_inner) B, L, _ = x.shape # Get Δ, B, C from the input x x_proj_out = self.x_proj(x) # (B, L, dt_rank + 2 * d_state) # Split into dt, B, C dt, B_in, C_in = jnp.split( x_proj_out, [self.dt_rank, self.dt_rank + d_state], axis=-1 ) # dt: (B,L,dt_rank), B: (B,L,d_state), C: (B,L,d_state) # Project dt to full dimension and apply softplus dt = jax.nn.softplus(self.dt_proj(dt)) # (B, L, d_inner) # Discretize A and B using Zero-Order Hold (ZOH) # A_bar = exp(Δ * A) # B_bar = (exp(Δ * A) - 1) / A * B (more stable version) # B_bar approx Δ * B for small Δ # This implementation uses the simpler B_bar = Δ * B A = -jnp.exp(self.A_log.value) # (d_inner, d_state) A_bar = jnp.exp(jnp.einsum("bld,dn->bldn", dt, A)) # (B, L, d_inner, d_state) # B needs to be broadcast to (B, L, d_inner, d_state) # B_in is (B, L, d_state) -> broadcast across d_inner B_bar = jnp.einsum("bld,bln->bldn", dt, B_in) # (B, L, d_inner, d_state) # Reshape C for einsum C = C_in.reshape(B, L, 1, d_state) # (B, L, 1, d_state) # Define the scan function for a single batch element def scan_fn(h_prev, scan_inputs): A_bar_t, B_bar_t, C_t, x_t = scan_inputs h_t = A_bar_t * h_prev + B_bar_t * x_t[:, None] # Update hidden state y_t = jnp.einsum("dn,dn->d", h_t, C_t).real # C_t is (d_inner, d_state) return h_t, y_t # Prepare inputs for the scan # We need to scan over the sequence length L # Reshape inputs to be (L, B, ...) scan_inputs = ( jnp.moveaxis(A_bar, 1, 0), jnp.moveaxis(B_bar, 1, 0), jnp.moveaxis(C.repeat(d_inner, axis=2), 1, 0), # C needs to be (B, L, d_inner, d_state) jnp.moveaxis(x, 1, 0), ) h0 = jnp.zeros((B, d_inner, d_state)) # Initial hidden state # Run the scan. vmap over the batch dimension. _, y = jax.vmap( lambda *args: jax.lax.scan(scan_fn, *args) )(h0, scan_inputs) # Transpose y back to (B, L, d_inner) y = jnp.moveaxis(y, 0, 1) return y def __call__(self, x: jax.Array) -> jax.Array: # x shape: (batch, seq_len, d_model) # 1. Input projection x_and_z = self.in_proj(x) x, z = jnp.split(x_and_z, 2, axis=-1) # Both (B, L, d_inner) # 2. 1D Causal Convolution # nnx.Conv expects (B, C, L), so we transpose x = self.conv1d(x.transpose(0, 2, 1)).transpose(0, 2, 1) # 3. Activation x = nnx.silu(x) # 4. Selective SSM y = self.ssm(x) # 5. D skip connection y = y + x * self.D.value # 6. Gating (SiLU) on the residual connection return y * nnx.silu(z) # --- 2. The Mamba Block --- class MambaBlock(nnx.Module): """A single block of the Mamba architecture.""" def __init__( self, d_model: int, d_state: int, d_conv: int, expand_factor: int, *, rngs: Rngs, ): self.norm = nnx.LayerNorm(num_features=d_model, rngs=rngs) self.ssm = MambaSSM( d_model=d_model, d_state=d_state, d_conv=d_conv, expand_factor=expand_factor, dt_rank=math.ceil(d_model / 16), rngs=rngs, ) self.out_proj = nnx.Linear( in_features=d_model * expand_factor, out_features=d_model, rngs=rngs ) def __call__(self, x: jax.Array) -> jax.Array: # x shape: (batch, seq_len, d_model) residual = x x = self.norm(x) x = self.ssm(x) x = self.out_proj(x) return x + residual # --- 3. The Full Mamba Model --- class MambaModel(nnx.Module): """A complete Mamba model for language modeling.""" def __init__( self, d_model: int, n_layers: int, vocab_size: int, d_state: int = 16, d_conv: int = 4, expand_factor: int = 2, *, rngs: Rngs, ): self.embedding = nnx.Embed(num_embeddings=vocab_size, features=d_model, rngs=rngs) self.layers = [ MambaBlock(d_model, d_state, d_conv, expand_factor, rngs=rngs) for _ in range(n_layers) ] self.norm_f = nnx.LayerNorm(num_features=d_model, rngs=rngs) self.lm_head = nnx.Linear(in_features=d_model, out_features=vocab_size, rngs=rngs) def __call__(self, input_ids: jax.Array) -> jax.Array: # input_ids shape: (batch, seq_len) x = self.embedding(input_ids) for layer in self.layers: x = layer(x) x = self.norm_f(x) logits = self.lm_head(x) return logits # --- 4. Example Usage and Training Step --- if __name__ == "__main__": # --- Model Configuration --- key = jax.random.PRNGKey(0) d_model = 128 n_layers = 4 vocab_size = 1000 batch_size = 8 seq_len = 64 # --- Initialization --- print("Initializing Mamba model...") # NNX uses a context for managing RNGs during initialization model = MambaModel( d_model=d_model, n_layers=n_layers, vocab_size=vocab_size, rngs=nnx.Rngs(params=key) ) # Create some dummy data dummy_input_ids = jax.random.randint(key, (batch_size, seq_len), 0, vocab_size) dummy_labels = jax.random.randint(key, (batch_size, seq_len), 0, vocab_size) # --- Forward Pass --- print("\nPerforming a forward pass...") logits = model(dummy_input_ids) print(f"Input shape: {dummy_input_ids.shape}") print(f"Output (logits) shape: {logits.shape}") assert logits.shape == (batch_size, seq_len, vocab_size) print("Forward pass successful!") # --- Training Step --- print("\nSetting up a training step...") # 1. Define the loss function def calculate_loss(model: MambaModel, inputs: jax.Array, labels: jax.Array): logits = model(inputs) # Flatten for cross-entropy loss = optax.softmax_cross_entropy_with_integer_labels( logits.reshape(-1, vocab_size), labels.reshape(-1) ).mean() return loss # 2. Get value and gradients using nnx.value_and_grad loss_grad_fn = nnx.value_and_grad(calculate_loss) # 3. Create an optimizer # The nnx.Optimizer wraps an optax optimizer and manages model state updates tx = optax.adamw(learning_rate=1e-3) optimizer = nnx.Optimizer(model, tx) # 4. Perform one training step print("Performing one training step...") loss_val, grads = loss_grad_fn(model, dummy_input_ids, dummy_labels) # The optimizer updates the model's parameters in-place optimizer.update(grads) print(f"Initial Loss: {loss_val:.4f}") # 5. Verify parameters have changed new_loss_val = calculate_loss(model, dummy_input_ids, dummy_labels) print(f"Loss after one step: {new_loss_val:.4f}") assert new_loss_val < loss_val print("Training step successful, loss decreased.") # --- Jitting the training step --- print("\nJitting the training step for performance...") @nnx.jit def train_step(optimizer: nnx.Optimizer, batch_inputs: jax.Array, batch_labels: jax.Array): # nnx.jit automatically handles the model state contained within the optimizer loss, grads = loss_grad_fn(optimizer.target, batch_inputs, batch_labels) optimizer.update(grads) return loss, optimizer # Run the jitted step loss, optimizer = train_step(optimizer, dummy_input_ids, dummy_labels) print(f"Loss after jitted step: {loss:.4f}") final_loss_val = calculate_loss(optimizer.target, dummy_input_ids, dummy_labels) print(f"Final loss: {final_loss_val:.4f}") print("Jitted training step successful!")Key Concepts in this
nnxImplementationnnx.Module: All our components (MambaSSM,MambaBlock,MambaModel) inherit fromnnx.Module.- In-place Initialization: Parameters (
nnx.Param) and submodules (nnx.Linear, etc.) are defined directly in the__init__method, just like in PyTorch. There is no separatesetup()method. nnx.RngsContext: Random keys are passed via a context object (nnx.Rngs). This simplifies the key-splitting boilerplate common in traditional Flax.- State is the Object: The model object is the state. You pass the
modelinstance directly to your loss function and gradient transformations. nnx.Optimizer: This is a convenient wrapper that holds both the model (the "target") and theoptaxoptimizer state. Callingoptimizer.update(grads)applies the gradients to the model's parameters in-place, simplifying the update step.nnx.jit: The JIT compiler for NNX. It's designed to seamlessly handle the mutable state ofnnx.Moduleobjects. Notice how thetrain_stepfunction takes theoptimizeras an argument and returns the updatedoptimizer.nnx.jittraces the function and correctly handles the state updates.vmapfor Batched Scan: The core SSM recurrence is defined for a single sequence. We usejax.vmapto efficiently apply this logic across all elements in a batch, which is a standard and powerful pattern in JAX.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.