Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-1· AnthropicHugging Face Arena Human Preference 55K

Write me a script in Python that implements a multi head attention that can attend to…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-3.5-turbo-0613’s answer.
  1. User

    Write me a script in Python that implements a multi head attention that can attend to very long context without increasing the hidden dim. (Something like Longformer)

  2. Claude Assistant

    Here is a Python implementation of multi-head attention that can attend to long context:

    import torch
    import torch.nn as nn
    
    class MultiHeadAttention(nn.Module):
        def __init__(self, hidden_dim, num_heads, dropout=0.1):
            super().__init__()
            self.hidden_dim = hidden_dim
            self.num_heads = num_heads
            
            self.q_linear = nn.Linear(hidden_dim, hidden_dim)
            self.k_linear = nn.Linear(hidden_dim, hidden_dim)
            self.v_linear = nn.Linear(hidden_dim, hidden_dim)
            
            self.dropout = nn.Dropout(dropout)
            self.out = nn.Linear(hidden_dim, hidden_dim)
            
        def forward(self, q, k, v, mask=None):
            bsz, q_len, k_len = q.size(0), q.size(1), k.size(1)
            
            # project to multi-head q, k, v
            q = self.q_linear(q).view(bsz, q_len, self.num_heads, self.hidden_dim//self.num_heads)
            k = self.k_linear(k).view(bsz, k_len, self.num_heads, self.hidden_dim//self.num_heads)
            v = self.v_linear(v).view(bsz, k_len, self.num_heads, self.hidden_dim//self.num_heads)
            
            # calculate attention and out
            attn = (q @ k.transpose(-2, -1)) / self.hidden_dim**0.5
            if mask is not None:
                attn = attn.masked_fill(mask == 0, float('-inf'))
            attn = torch.softmax(attn, dim=-1)
            attn = self.dropout(attn)
            
            out = (attn @ v).transpose(1, 2).contiguous().view(bsz, q_len, self.hidden_dim)
            out = self.out(out)
            return out
    

    This implements multi-head attention with the following key points:

    • It projects the queries, keys and values into multiple heads with a smaller dimension
    • It calculates the attention scores as a dot product between queries and keys
    • It applies a mask to the attention scores if a mask is passed
    • It uses dropout for regularization
    • It concatenates the multiple heads and projects back to the original hidden dimension

    This allows the model to jointly attend to information from different representation subspaces at different positions, allowing it to attend to long context without increasing the hidden dim.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.