Do attention heads effectively split the embedding space into a direct sum of lower…
- User
Do attention heads effectively split the embedding space into a direct sum of lower dimensional spaces?
Assistant
No, attention heads do not strictly split the embedding space into a direct sum of lower dimensional subspaces. Here are a few key points about how attention heads work:
-
Attention heads project the embeddings into lower dimensional "key", "query", and "value" spaces. So each head gets its own transformed view of the original embedding space.
-
The projections are learned linear transformations, not strict splits of the space. The different heads may learn overlapping projections.
-
The outputs of the attention heads are then concatenated back into the original embedding dimensionality. This is not a direct sum, as there can be interactions between the head outputs in this fuller dimensionality.
-
There are also residual connections around each attention layer, further allowing interactions between the different head outputs as data flows through the network.
So in summary, while attention heads give different views of the input, they do not strictly decompose the space into non-interacting lower dimensional subspaces. The head outputs are combined and can influence each other in the subsequent network processing. The overall goal is to enable the model to learn useful transformations of the embedding space tailored to different attention queries/keys/values.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.