Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

Hi, code gpt2 in pytorch

  1. User

    Hi, code gpt2 in pytorch

  2. Assistant

    Sure! Below is an implementation of the GPT-2 model in PyTorch. This implementation includes the key components of the GPT-2 architecture:

    • Multi-head self-attention mechanism
    • Transformer blocks with layer normalization and residual connections
    • Token and positional embeddings
    • Output linear layer for language modeling

    Please note that training this model from scratch would require a considerable amount of data and computational resources. However, this code can serve as a starting point for educational purposes or small-scale experiments.

    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    
    class MultiHeadSelfAttention(nn.Module):
        def __init__(self, embed_size, num_heads):
            super(MultiHeadSelfAttention, self).__init__()
    
            assert embed_size % num_heads == 0, "Embedding size must be divisible by num_heads"
    
            self.embed_size = embed_size
            self.num_heads = num_heads
            self.head_dim = embed_size // num_heads
    
            # Linear layers for queries, keys, and values
            self.query = nn.Linear(embed_size, embed_size)
            self.key   = nn.Linear(embed_size, embed_size)
            self.value = nn.Linear(embed_size, embed_size)
            # Final linear layer after concatenation
            self.fc_out = nn.Linear(embed_size, embed_size)
    
        def forward(self, x, mask=None):
            N, seq_length, embed_size = x.size()
    
            # Linear projections
            queries = self.query(x)   # (N, seq_length, embed_size)
            keys    = self.key(x)
            values  = self.value(x)
    
            # Split the embedding dimension into num_heads
            queries = queries.view(N, seq_length, self.num_heads, self.head_dim)
            keys    = keys.view(N, seq_length, self.num_heads, self.head_dim)
            values  = values.view(N, seq_length, self.num_heads, self.head_dim)
    
            # Rearrange dimensions to (N, num_heads, seq_length, head_dim)
            queries = queries.transpose(1, 2)
            keys    = keys.transpose(1, 2)
            values  = values.transpose(1, 2)
    
            # Calculate scores
            energy = torch.matmul(queries, keys.transpose(-1, -2))  # (N, num_heads, seq_length, seq_length)
            scaling_factor = self.head_dim ** 0.5
            energy = energy / scaling_factor
    
            # Apply causal mask (optional)
            if mask is not None:
                energy = energy.masked_fill(mask == 0, float('-inf'))
    
            # Apply softmax to get attention weights
            attention = torch.softmax(energy, dim=-1)  # (N, num_heads, seq_length, seq_length)
    
            # Multiply attention weights with values
            out = torch.matmul(attention, values)  # (N, num_heads, seq_length, head_dim)
    
            # Concatenate heads
            out = out.transpose(1, 2).contiguous()  # (N, seq_length, num_heads, head_dim)
            out = out.view(N, seq_length, self.embed_size)  # (N, seq_length, embed_size)
    
            # Final linear layer
            out = self.fc_out(out)  # (N, seq_length, embed_size)
    
            return out
    
    class FeedForward(nn.Module):
        def __init__(self, embed_size, ff_hidden_size, dropout=0.1):
            super(FeedForward, self).__init__()
            self.fc1 = nn.Linear(embed_size, ff_hidden_size)
            self.fc2 = nn.Linear(ff_hidden_size, embed_size)
            self.activation = nn.GELU()
            self.dropout = nn.Dropout(dropout)
    
        def forward(self, x):
            x = self.activation(self.fc1(x))  # Apply GELU activation
            x = self.dropout(x)               # Apply dropout
            x = self.fc2(x)
            return x
    
    class TransformerBlock(nn.Module):
        def __init__(self, embed_size, num_heads, ff_hidden_size, dropout=0.1):
            super(TransformerBlock, self).__init__()
            self.attention = MultiHeadSelfAttention(embed_size, num_heads)
            self.norm1 = nn.LayerNorm(embed_size)
            self.feed_forward = FeedForward(embed_size, ff_hidden_size, dropout)
            self.norm2 = nn.LayerNorm(embed_size)
            self.dropout = nn.Dropout(dropout)
    
        def forward(self, x, mask=None):
            # Self-attention with residual connection and layer normalization
            attn_output = self.attention(x, mask)
            x = self.norm1(x + self.dropout(attn_output))
    
            # Feed-forward network with residual connection and layer normalization
            ff_output = self.feed_forward(x)
            x = self.norm2(x + self.dropout(ff_output))
    
            return x
    
    class GPT2(nn.Module):
        def __init__(
            self,
            vocab_size,
            max_length,
            embed_size,
            num_heads,
            num_layers,
            ff_hidden_size,
            dropout=0.1
        ):
            super(GPT2, self).__init__()
    
            self.embed_size = embed_size
            self.vocab_size = vocab_size
            self.max_length = max_length
    
            # Token and positional embeddings
            self.token_embedding = nn.Embedding(vocab_size, embed_size)
            self.position_embedding = nn.Embedding(max_length, embed_size)
    
            # Transformer blocks
            self.layers = nn.ModuleList(
                [
                    TransformerBlock(embed_size, num_heads, ff_hidden_size, dropout)
                    for _ in range(num_layers)
                ]
            )
    
            # Final layer normalization
            self.ln_f = nn.LayerNorm(embed_size)
    
            # Output linear layer for language modeling
            self.fc_out = nn.Linear(embed_size, vocab_size)
    
            self.dropout = nn.Dropout(dropout)
    
        def forward(self, x, mask=None):
            N, seq_length = x.size()
    
            # Ensure position indices do not exceed maximum length
            positions = torch.arange(0, seq_length, device=x.device).unsqueeze(0)
            if seq_length > self.max_length:
                positions = positions[:, -self.max_length:]
                x = x[:, -self.max_length:]
                seq_length = self.max_length
    
            positions = positions.expand(N, seq_length)  # (N, seq_length)
    
            # Embed tokens and positions
            tok_emb = self.token_embedding(x)           # (N, seq_length, embed_size)
            pos_emb = self.position_embedding(positions)  # (N, seq_length, embed_size)
            x = self.dropout(tok_emb + pos_emb)         # (N, seq_length, embed_size)
    
            # Apply Transformer blocks
            for layer in self.layers:
                x = layer(x, mask)
    
            # Final layer normalization
            x = self.ln_f(x)
    
            # Output logits for each vocabulary token
            logits = self.fc_out(x)  # (N, seq_length, vocab_size)
    
            return logits
    
    # Example usage:
    
    # Define model parameters
    vocab_size = 50257       # GPT-2's vocabulary size
    max_length = 1024        # Maximum sequence length
    embed_size = 768         # Embedding size for GPT-2 Small
    num_heads = 12           # Number of attention heads
    num_layers = 12          # Number of Transformer blocks
    ff_hidden_size = 3072    # Feed-forward network hidden size
    
    # Instantiate the model
    model = GPT2(
        vocab_size=vocab_size,
        max_length=max_length,
        embed_size=embed_size,
        num_heads=num_heads,
        num_layers=num_layers,
        ff_hidden_size=ff_hidden_size,
        dropout=0.1
    )
    
    # Move model to GPU if available
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.to(device)
    
    # Sample input (batch_size=2, seq_length=10)
    input_ids = torch.randint(0, vocab_size, (2, 10)).to(device)  # Random integers as token IDs
    
    # Generate causal mask for decoding (optional)
    # This ensures that the model cannot "see" future tokens when predicting the next token
    mask = torch.tril(torch.ones((1, 1, 10, 10), device=device)).bool()
    
    # Forward pass
    outputs = model(input_ids, mask=mask)
    
    print(outputs.shape)  # Should output: torch.Size([2, 10, 50257])
    

    Explanation of the Code:

    1. MultiHeadSelfAttention:

      • Splits the embeddings into multiple heads.
      • Performs scaled dot-product attention for each head.
      • Concatenates the heads and applies a final linear layer.
    2. FeedForward:

      • Applies two linear transformations with a GELU activation in between.
      • Incorporates dropout for regularization.
    3. TransformerBlock:

      • Applies self-attention and feed-forward networks with residual connections.
      • Uses layer normalization before each sub-layer (pre-norm style).
    4. GPT2 Model:

      • Contains token and positional embeddings.
      • Stacks multiple Transformer blocks.
      • Applies a final layer normalization.
      • Outputs logits for each position in the input sequence.
    5. Causal Masking:

      • An optional mask is created to prevent the model from attending to future positions.
      • This is important for autoregressive language modeling tasks, where the model predicts the next token based on past tokens.
    6. Usage Example:

      • Defines the model parameters based on GPT-2 Small.
      • Instantiates the model and moves it to the appropriate device.
      • Creates a sample input and performs a forward pass.

    Notes:

    • Training: To train the model, you would need a dataset of tokenized text. The loss function commonly used is nn.CrossEntropyLoss, comparing the model's predictions to the target tokens shifted by one position.
    • Efficiency: For large models and sequences, consider optimizing the code with techniques like mixed-precision training or model parallelism.
    • Pre-trained Models: In practice, it's recommended to use pre-trained models from libraries like Hugging Face's transformers, which provide optimized implementations and pre-trained weights.

    Dependencies:

    Make sure you have PyTorch installed:

    pip install torch
    

    Disclaimer:

    This implementation is a simplified version of the GPT-2 model and may not include all optimizations and features present in the official model. For production use or extensive research, consider using established libraries that offer robust implementations.