Skip to content
TheChatArchiveThe Chat Archive
Mistralmistral-medium-2505· Mistral AIHugging Face LMArena Human Preference 140K

Write an LLM.

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwq-32b’s answer.
  1. User

    Write an LLM.

  2. Mistral Assistant

    Building a Simple Language Model (LLM) from Scratch

    Creating a full-scale large language model (LLM) like those from OpenAI or Google requires massive resources and expertise, but I can outline a simplified version that demonstrates the core concepts. Here's a basic implementation of a character-level language model using Python and PyTorch.

    Basic LLM Implementation

    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    import numpy as np
    import random
    
    # Hyperparameters
    batch_size = 32
    block_size = 8
    max_iters = 5000
    eval_interval = 500
    learning_rate = 3e-4
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    eval_iters = 200
    n_embd = 32
    n_head = 4
    n_layer = 3
    dropout = 0.2
    
    # Load and prepare text data
    with open('input.txt', 'r', encoding='utf-8') as f:
        text = f.read()
    
    # Create vocabulary
    chars = sorted(list(set(text)))
    vocab_size = len(chars)
    print(f"Vocabulary size: {vocab_size}")
    
    # Create character to index and index to character mappings
    stoi = { ch:i for i,ch in enumerate(chars) }
    itos = { i:ch for i,ch in enumerate(chars) }
    encode = lambda s: [stoi[c] for c in s]
    decode = lambda l: ''.join([itos[i] for i in l])
    
    # Encode the entire text
    data = torch.tensor(encode(text), dtype=torch.long)
    
    # Split into train and validation sets
    n = int(0.9*len(data))
    train_data = data[:n]
    val_data = data[n:]
    
    # Data loading
    def get_batch(split):
        data = train_data if split == 'train' else val_data
        ix = torch.randint(len(data) - block_size, (batch_size,))
        x = torch.stack([data[i:i+block_size] for i in ix])
        y = torch.stack([data[i+1:i+block_size+1] for i in ix])
        x, y = x.to(device), y.to(device)
        return x, y
    
    # Model architecture
    class Head(nn.Module):
        def __init__(self, head_size):
            super().__init__()
            self.key = nn.Linear(n_embd, head_size, bias=False)
            self.query = nn.Linear(n_embd, head_size, bias=False)
            self.value = nn.Linear(n_embd, head_size, bias=False)
            self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))
            self.dropout = nn.Dropout(dropout)
    
        def forward(self, x):
            B,T,C = x.shape
            k = self.key(x)   # (B,T,C)
            q = self.query(x) # (B,T,C)
            # Compute attention scores
            wei = q @ k.transpose(-2,-1) * C**-0.5 # (B, T, T)
            wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf')) # (B, T, T)
            wei = F.softmax(wei, dim=-1) # (B, T, T)
            wei = self.dropout(wei)
            # Perform weighted aggregation
            v = self.value(x) # (B,T,C)
            out = wei @ v # (B, T, C)
            return out
    
    class MultiHeadAttention(nn.Module):
        def __init__(self, num_heads, head_size):
            super().__init__()
            self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])
            self.proj = nn.Linear(n_embd, n_embd)
            self.dropout = nn.Dropout(dropout)
    
        def forward(self, x):
            out = torch.cat([h(x) for h in self.heads], dim=-1)
            out = self.dropout(self.proj(out))
            return out
    
    class FeedForward(nn.Module):
        def __init__(self, n_embd):
            super().__init__()
            self.net = nn.Sequential(
                nn.Linear(n_embd, 4 * n_embd),
                nn.ReLU(),
                nn.Linear(4 * n_embd, n_embd),
                nn.Dropout(dropout),
            )
    
        def forward(self, x):
            return self.net(x)
    
    class Block(nn.Module):
        def __init__(self, n_embd, n_head):
            super().__init__()
            head_size = n_embd // n_head
            self.sa = MultiHeadAttention(n_head, head_size)
            self.ffwd = FeedForward(n_embd)
            self.ln1 = nn.LayerNorm(n_embd)
            self.ln2 = nn.LayerNorm(n_embd)
    
        def forward(self, x):
            x = x + self.sa(self.ln1(x))
            x = x + self.ffwd(self.ln2(x))
            return x
    
    class LanguageModel(nn.Module):
        def __init__(self):
            super().__init__()
            self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
            self.position_embedding_table = nn.Embedding(block_size, n_embd)
            self.blocks = nn.Sequential(*[Block(n_embd, n_head=n_head) for _ in range(n_layer)])
            self.ln_f = nn.LayerNorm(n_embd)
            self.lm_head = nn.Linear(n_embd, vocab_size)
    
        def forward(self, idx, targets=None):
            B, T = idx.shape
    
            # Get token embeddings
            tok_emb = self.token_embedding_table(idx) # (B,T,C)
    
            # Get position embeddings
            pos_emb = self.position_embedding_table(torch.arange(T, device=device)) # (T,C)
    
            # Combine embeddings
            x = tok_emb + pos_emb # (B,T,C)
    
            # Pass through transformer blocks
            x = self.blocks(x) # (B,T,C)
    
            # Final layer norm
            x = self.ln_f(x) # (B,T,C)
    
            # Language modeling head
            logits = self.lm_head(x) # (B,T,vocab_size)
    
            if targets is None:
                loss = None
            else:
                B, T, C = logits.shape
                logits = logits.view(B*T, C)
                targets = targets.view(B*T)
                loss = F.cross_entropy(logits, targets)
    
            return logits, loss
    
        def generate(self, idx, max_new_tokens):
            for _ in range(max_new_tokens):
                # Crop idx to the last block_size tokens
                idx_cond = idx[:, -block_size:]
                # Get predictions
                logits, loss = self(idx_cond)
                # Focus only on the last time step
                logits = logits[:, -1, :] # (B, C)
                # Apply softmax to get probabilities
                probs = F.softmax(logits, dim=-1) # (B, C)
                # Sample from the distribution
                idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)
                # Append sampled index to the running sequence
                idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)
            return idx
    
    # Initialize model
    model = LanguageModel()
    m = model.to(device)
    
    # Training loop
    optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
    
    for iter in range(max_iters):
        if iter % eval_interval == 0:
            losses = []
            for _ in range(eval_iters):
                xb, yb = get_batch('val')
                logits, loss = model(xb, yb)
                losses.append(loss.item())
            print(f"Step {iter}, val loss: {np.mean(losses):.4f}")
    
        # Sample a batch of data
        xb, yb = get_batch('train')
    
        # Evaluate the loss
        logits, loss = model(xb, yb)
        optimizer.zero_grad(set_to_none=True)
        loss.backward()
        optimizer.step()
    
    # Generate some text
    context = torch.zeros((1, 1), dtype=torch.long, device=device)
    print(decode(m.generate(context, max_new_tokens=500)[0].tolist()))
    

    Key Components Explained

    1. Tokenization: The model works at the character level, converting text to numerical tokens.

    2. Transformer Architecture:

      • Multi-head attention: Allows the model to focus on different parts of the input sequence simultaneously
      • Positional embeddings: Helps the model understand the order of tokens
      • Feed-forward networks: Process the information after attention
    3. Training Process:

      • The model learns to predict the next character in a sequence
      • Uses cross-entropy loss to measure prediction accuracy
      • Implements the AdamW optimizer for training
    4. Text Generation:

      • After training, the model can generate new text by sampling from its predicted probabilities

    Limitations of This Implementation

    1. This is a character-level model, not a word or subword model like most modern LLMs
    2. The model size is very small compared to production LLMs
    3. It lacks many optimizations and features found in state-of-the-art models
    4. Training data is limited to a single text file

    For a production-ready LLM, you would need:

    • Much larger model architecture
    • Distributed training across multiple GPUs/TPUs
    • Sophisticated tokenization (like Byte Pair Encoding)
    • Massive amounts of high-quality training data
    • Advanced optimization techniques
    • Careful hyperparameter tuning

    Would you like me to explain any specific part of this implementation in more detail?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt