Skip to content
TheChatArchiveThe Chat Archive
Human-written demonstration (OpenAssistant volunteers)· OpenAssistant (human-written)Hugging Face OpenAssistant Conversations (oasst2)

In Transformer machine learning models: How is multi-head attention different from…

In OpenAssistant data, “assistant” turns were written by human volunteers as demonstrations. They are not model outputs.
  1. User

    In Transformer machine learning models: How is multi-head attention different from normal attention? First describe the intuition and then the mathematical perspective.

  2. Assistant

    Intuition:

    In Transformer machine learning models, attention mechanisms allow the model to weigh the importance of different words or tokens in a sequence when generating an output. The primary goal is to enable the model to focus on relevant information and ignore irrelevant parts.

    Normal (single-head) attention computes these weights based on a single context or perspective. In contrast, multi-head attention divides this attention mechanism into multiple heads, allowing the model to capture different relationships and context between words in parallel. This means that each head can learn to focus on different aspects or patterns in the data, leading to a richer representation of the input sequence.

    Mathematical perspective:

    In both normal attention and multi-head attention, the attention mechanism involves computing a score or weight for each word/token in the input sequence based on their relevance to the current output position. These scores are then used to create a weighted sum of input features, resulting in a context-aware output.

    For a normal attention mechanism, the score is calculated using query (Q), key (K), and value (V) matrices, which are derived from the input sequence. The attention is computed as:

    Calculate the dot product between the query and key matrices: Q * K^T Scale the result by dividing it by the square root of the key dimension (d_k): (Q * K^T) / sqrt(d_k) Apply a softmax function to normalize the scores: softmax((Q * K^T) / sqrt(d_k)) Compute the weighted sum of the value matrix using the normalized scores: softmax((Q * K^T) / sqrt(d_k)) * V In multi-head attention, the process is similar, but it is performed independently for each head. To do this, the query, key, and value matrices are first linearly projected into h (number of heads) different subspaces using learnable weight matrices W^Q_i, W^K_i, and W^V_i (for i=1 to h):

    Q_i = Q * W^Q_i K_i = K * W^K_i V_i = V * W^V_i Next, the attention mechanism is applied for each head:

    Attention_i = softmax((Q_i * K_i^T) / sqrt(d_k)) * V_i After computing the attention for each head, the outputs are concatenated and linearly transformed using another learnable weight matrix W^O:

    MultiHeadAttention = Concat(Attention_1, Attention_2, ..., Attention_h) * W^O The multi-head attention allows the model to capture different relationships and context between words in parallel, leading to a richer representation of the input sequence.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.