Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-2.1· AnthropicHugging Face Arena Human Preference 55K

Do attention heads effectively split the embedding space into a direct sum of lower…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-1’s answer.
  1. User

    Do attention heads effectively split the embedding space into a direct sum of lower dimensional spaces?

  2. Claude Assistant

    No, attention heads do not strictly split the embedding space into a direct sum of lower dimensional subspaces. Here are a few key points about how attention heads work:

    • Attention heads project the embeddings into lower dimensional "key", "query", and "value" spaces. So each head gets its own transformed view of the original embedding space.

    • The projections are learned linear transformations, not strict splits of the space. The different heads may learn overlapping projections.

    • The outputs of the attention heads are then concatenated back into the original embedding dimensionality. This is not a direct sum, as there can be interactions between the head outputs in this fuller dimensionality.

    • There are also residual connections around each attention layer, further allowing interactions between the different head outputs as data flows through the network.

    So in summary, while attention heads give different views of the input, they do not strictly decompose the space into non-interacting lower dimensional subspaces. The head outputs are combined and can influence each other in the subsequent network processing. The overall goal is to enable the model to learn useful transformations of the embedding space tailored to different attention queries/keys/values.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.