Write me a script in Python that implements a multi head attention that can attend to…
- User
Write me a script in Python that implements a multi head attention that can attend to very long context without increasing the hidden dim. (Something like Longformer)
- Assistant
Sure! Here's an example script that implements a multi-head attention mechanism similar to Longformer in Python:
import torch import torch.nn as nn class LongformerAttention(nn.Module): def __init__(self, input_dim, num_heads, max_context_length): super(LongformerAttention, self).__init__() self.input_dim = input_dim self.num_heads = num_heads self.max_context_length = max_context_length self.query_projection = nn.Linear(input_dim, input_dim) self.key_projection = nn.Linear(input_dim, input_dim) self.value_projection = nn.Linear(input_dim, input_dim) self.dropout = nn.Dropout(0.1) def forward(self, input_tensor): batch_size, seq_length, _ = input_tensor.size() # Pad the input tensor if its length is smaller than max_context_length if seq_length < self.max_context_length: padding = torch.zeros(batch_size, self.max_context_length - seq_length, self.input_dim) input_tensor = torch.cat([input_tensor, padding], dim=1) # Project the input tensor to obtain query, key, and value tensors query = self.query_projection(input_tensor) key = self.key_projection(input_tensor) value = self.value_projection(input_tensor) # Reshape query, key, and value tensors to split them into multiple heads query = query.view(batch_size, -1, self.num_heads, self.input_dim // self.num_heads) key = key.view(batch_size, -1, self.num_heads, self.input_dim // self.num_heads) value = value.view(batch_size, -1, self.num_heads, self.input_dim // self.num_heads) # Transpose the dimensions to perform attention across the sequence length query = query.transpose(1, 2) key = key.transpose(1, 2) value = value.transpose(1, 2) # Compute the attention scores attention_scores = torch.matmul(query, key.transpose(-2, -1)) attention_scores = attention_scores / (self.input_dim ** 0.5) # Apply mask to prevent attending to padding tokens attention_mask = torch.tril(torch.ones(seq_length, seq_length)) attention_mask = attention_mask.unsqueeze(0).unsqueeze(0) attention_mask = (1 - attention_mask) * -1e9 attention_scores = attention_scores + attention_mask # Apply softmax to obtain attention probabilities attention_probs = nn.Softmax(dim=-1)(attention_scores) attention_probs = self.dropout(attention_probs) # Attend to the value tensor using the attention probabilities context = torch.matmul(attention_probs, value) # Reshape and transpose the context tensor back to its original shape context = context.transpose(1, 2) context = context.contiguous().view(batch_size, -1, self.input_dim) # Trim the context tensor to remove padding tokens context = context[:, :seq_length, :] return contextIn this script, we define a
LongformerAttentionclass that takes as input theinput_dim(dimension of the input tensor),num_heads(number of attention heads), andmax_context_length(maximum length of the context). Theforwardmethod performs the multi-head attention mechanism similar to Longformer.The input tensor is first projected using linear layers to obtain query, key, and value tensors. These tensors are then reshaped and transposed to split them into multiple heads and perform attention across the sequence length. Attention scores are computed by taking the dot product of query and key tensors, and then softmax is applied to obtain attention probabilities. The value tensor is then attended using the attention probabilities, and the resulting context tensor is reshaped and transposed back to its original shape. Finally, the context tensor is trimmed to remove any padding tokens.
You can use this
LongformerAttentionclass in your code to implement a multi-head attention mechanism that can attend to very long context without increasing the hidden dimension.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.