Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0613· OpenAIHugging Face Arena Human Preference 55K

Write me a script in Python that implements a multi head attention that can attend to…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-1’s answer.
  1. User

    Write me a script in Python that implements a multi head attention that can attend to very long context without increasing the hidden dim. (Something like Longformer)

  2. Assistant

    Sure! Here's an example script that implements a multi-head attention mechanism similar to Longformer in Python:

    import torch
    import torch.nn as nn
    
    class LongformerAttention(nn.Module):
        def __init__(self, input_dim, num_heads, max_context_length):
            super(LongformerAttention, self).__init__()
            
            self.input_dim = input_dim
            self.num_heads = num_heads
            self.max_context_length = max_context_length
            
            self.query_projection = nn.Linear(input_dim, input_dim)
            self.key_projection = nn.Linear(input_dim, input_dim)
            self.value_projection = nn.Linear(input_dim, input_dim)
            
            self.dropout = nn.Dropout(0.1)
            
        def forward(self, input_tensor):
            batch_size, seq_length, _ = input_tensor.size()
            
            # Pad the input tensor if its length is smaller than max_context_length
            if seq_length < self.max_context_length:
                padding = torch.zeros(batch_size, self.max_context_length - seq_length, self.input_dim)
                input_tensor = torch.cat([input_tensor, padding], dim=1)
            
            # Project the input tensor to obtain query, key, and value tensors
            query = self.query_projection(input_tensor)
            key = self.key_projection(input_tensor)
            value = self.value_projection(input_tensor)
            
            # Reshape query, key, and value tensors to split them into multiple heads
            query = query.view(batch_size, -1, self.num_heads, self.input_dim // self.num_heads)
            key = key.view(batch_size, -1, self.num_heads, self.input_dim // self.num_heads)
            value = value.view(batch_size, -1, self.num_heads, self.input_dim // self.num_heads)
            
            # Transpose the dimensions to perform attention across the sequence length
            query = query.transpose(1, 2)
            key = key.transpose(1, 2)
            value = value.transpose(1, 2)
            
            # Compute the attention scores
            attention_scores = torch.matmul(query, key.transpose(-2, -1))
            attention_scores = attention_scores / (self.input_dim ** 0.5)
            
            # Apply mask to prevent attending to padding tokens
            attention_mask = torch.tril(torch.ones(seq_length, seq_length))
            attention_mask = attention_mask.unsqueeze(0).unsqueeze(0)
            attention_mask = (1 - attention_mask) * -1e9
            attention_scores = attention_scores + attention_mask
            
            # Apply softmax to obtain attention probabilities
            attention_probs = nn.Softmax(dim=-1)(attention_scores)
            attention_probs = self.dropout(attention_probs)
            
            # Attend to the value tensor using the attention probabilities
            context = torch.matmul(attention_probs, value)
            
            # Reshape and transpose the context tensor back to its original shape
            context = context.transpose(1, 2)
            context = context.contiguous().view(batch_size, -1, self.input_dim)
            
            # Trim the context tensor to remove padding tokens
            context = context[:, :seq_length, :]
            
            return context
    

    In this script, we define a LongformerAttention class that takes as input the input_dim (dimension of the input tensor), num_heads (number of attention heads), and max_context_length (maximum length of the context). The forward method performs the multi-head attention mechanism similar to Longformer.

    The input tensor is first projected using linear layers to obtain query, key, and value tensors. These tensors are then reshaped and transposed to split them into multiple heads and perform attention across the sequence length. Attention scores are computed by taking the dot product of query and key tensors, and then softmax is applied to obtain attention probabilities. The value tensor is then attended using the attention probabilities, and the resulting context tensor is reshaped and transposed back to its original shape. Finally, the context tensor is trimmed to remove any padding tokens.

    You can use this LongformerAttention class in your code to implement a multi-head attention mechanism that can attend to very long context without increasing the hidden dimension.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.