Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-5-haiku-20241022· AnthropicHugging Face LMArena Human Preference 140K

Write me a small self-contained JAX training script showing best practices.

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-flash-preview-04-17’s answer.
  1. User

    Write me a small self-contained JAX training script showing best practices.

  2. Claude Assistant

    I'll provide a concise JAX training script demonstrating best practices for machine learning model training. This example will use a simple MLP on a synthetic regression task:

    import jax
    import jax.numpy as jnp
    import optax
    from flax import linen as nn
    from flax.training import train_state
    import numpy as np
    
    class MLP(nn.Module):
        """Simple Multi-Layer Perceptron"""
        features: list[int]
        
        @nn.compact
        def __call__(self, x):
            for feat in self.features[:-1]:
                x = nn.relu(nn.Dense(feat)(x))
            return nn.Dense(self.features[-1])(x)
    
    def create_train_state(rng, learning_rate):
        """Initialize training state with model and optimizer"""
        model = MLP(features=[64, 32, 1])
        params = model.init(rng, jnp.ones((1, 10)))['params']
        tx = optax.adam(learning_rate)
        return train_state.TrainState.create(apply_fn=model.apply, params=params, tx=tx)
    
    @jax.jit
    def train_step(state, batch):
        """Perform a single training step"""
        def loss_fn(params):
            x, y = batch
            pred = state.apply_fn({'params': params}, x)
            return jnp.mean((pred - y) ** 2)
        
        grad_fn = jax.value_and_grad(loss_fn)
        loss, grads = grad_fn(state.params)
        return state.apply_gradients(grads=grads), loss
    
    def generate_data(num_samples=1000):
        """Generate synthetic regression dataset"""
        key = jax.random.PRNGKey(0)
        x = jax.random.normal(key, (num_samples, 10))
        y = jnp.sum(x, axis=1, keepdims=True) + jax.random.normal(key, (num_samples, 1)) * 0.1
        return x, y
    
    def main():
        # Hyperparameters
        learning_rate = 1e-3
        num_epochs = 100
        batch_size = 32
    
        # Data preparation
        x, y = generate_data()
        
        # Random key management
        key = jax.random.PRNGKey(42)
        key, init_key = jax.random.split(key)
    
        # Initialize training state
        state = create_train_state(init_key, learning_rate)
    
        # Training loop
        for epoch in range(num_epochs):
            # Shuffle data
            key, shuffle_key = jax.random.split(key)
            perm = jax.random.permutation(shuffle_key, len(x))
            x, y = x[perm], y[perm]
    
            # Batch training
            for i in range(0, len(x), batch_size):
                batch_x = x[i:i+batch_size]
                batch_y = y[i:i+batch_size]
                state, loss = train_step(state, (batch_x, batch_y))
    
            # Optional logging
            if epoch % 10 == 0:
                print(f"Epoch {epoch}, Loss: {loss}")
    
    if __name__ == '__main__':
        main()
    

    Key best practices demonstrated:

    1. Modular Design:

      • Separate model definition
      • Clear function responsibilities
      • Use of Flax for neural network layers
    2. JAX Best Practices:

      • Use of jax.jit for compilation
      • Proper random key management
      • Functional programming style
      • Gradient computation with jax.value_and_grad
    3. Training Loop:

      • Batch processing
      • Data shuffling
      • Epoch-based training
      • Optional logging
    4. Optimizer:

      • Using Optax for optimizer (Adam)
      • Clean optimizer initialization
    5. Randomness:

      • Explicit random key splitting
      • Reproducible randomness

    This script provides a clean, concise example of JAX training with modern best practices. It's a template that can be easily adapted to more complex models and datasets.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.