Write me a small self-contained JAX training script showing best practices.
- User
Write me a small self-contained JAX training script showing best practices.
Assistant
I'll provide a concise JAX training script demonstrating best practices for machine learning model training. This example will use a simple MLP on a synthetic regression task:
import jax import jax.numpy as jnp import optax from flax import linen as nn from flax.training import train_state import numpy as np class MLP(nn.Module): """Simple Multi-Layer Perceptron""" features: list[int] @nn.compact def __call__(self, x): for feat in self.features[:-1]: x = nn.relu(nn.Dense(feat)(x)) return nn.Dense(self.features[-1])(x) def create_train_state(rng, learning_rate): """Initialize training state with model and optimizer""" model = MLP(features=[64, 32, 1]) params = model.init(rng, jnp.ones((1, 10)))['params'] tx = optax.adam(learning_rate) return train_state.TrainState.create(apply_fn=model.apply, params=params, tx=tx) @jax.jit def train_step(state, batch): """Perform a single training step""" def loss_fn(params): x, y = batch pred = state.apply_fn({'params': params}, x) return jnp.mean((pred - y) ** 2) grad_fn = jax.value_and_grad(loss_fn) loss, grads = grad_fn(state.params) return state.apply_gradients(grads=grads), loss def generate_data(num_samples=1000): """Generate synthetic regression dataset""" key = jax.random.PRNGKey(0) x = jax.random.normal(key, (num_samples, 10)) y = jnp.sum(x, axis=1, keepdims=True) + jax.random.normal(key, (num_samples, 1)) * 0.1 return x, y def main(): # Hyperparameters learning_rate = 1e-3 num_epochs = 100 batch_size = 32 # Data preparation x, y = generate_data() # Random key management key = jax.random.PRNGKey(42) key, init_key = jax.random.split(key) # Initialize training state state = create_train_state(init_key, learning_rate) # Training loop for epoch in range(num_epochs): # Shuffle data key, shuffle_key = jax.random.split(key) perm = jax.random.permutation(shuffle_key, len(x)) x, y = x[perm], y[perm] # Batch training for i in range(0, len(x), batch_size): batch_x = x[i:i+batch_size] batch_y = y[i:i+batch_size] state, loss = train_step(state, (batch_x, batch_y)) # Optional logging if epoch % 10 == 0: print(f"Epoch {epoch}, Loss: {loss}") if __name__ == '__main__': main()Key best practices demonstrated:
-
Modular Design:
- Separate model definition
- Clear function responsibilities
- Use of Flax for neural network layers
-
JAX Best Practices:
- Use of
jax.jitfor compilation - Proper random key management
- Functional programming style
- Gradient computation with
jax.value_and_grad
- Use of
-
Training Loop:
- Batch processing
- Data shuffling
- Epoch-based training
- Optional logging
-
Optimizer:
- Using Optax for optimizer (Adam)
- Clean optimizer initialization
-
Randomness:
- Explicit random key splitting
- Reproducible randomness
This script provides a clean, concise example of JAX training with modern best practices. It's a template that can be easily adapted to more complex models and datasets.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.