explain the size of weight matrices in transformer architecture?
- User
explain the size of weight matrices in transformer architecture?
- Assistant
Certainly! In the Transformer architecture, weight matrices are central to its operations, especially in the attention mechanism and feed-forward layers. Here’s a breakdown of the main weight matrices and their sizes, assuming:
- d_model: the model’s hidden size (e.g., 512 or 768)
- d_ff: the feed-forward layer size (often 4×d_model)
- n_heads: number of attention heads (e.g., 8 or 12)
- d_k: dimension per head (d_model / n_heads)
1. Self-Attention Layer
For each attention head, we have three main projections:
- Query (Q)
- Key (K)
- Value (V)
Each of these is a linear transformation from the input embedding space to the head dimension.
Weight matrices:
- W_Q: (d_model, d_k)
- W_K: (d_model, d_k)
- W_V: (d_model, d_k)
For multi-head attention (n_heads heads), these are often implemented as a single large matrix for efficiency:
- W_Q: (d_model, d_model) (since n_heads × d_k = d_model)
- W_K: (d_model, d_model)
- W_V: (d_model, d_model)
After attention, the outputs from all heads are concatenated and projected back to d_model:
- W_O: (d_model, d_model)
2. Feed-Forward Layer
This is a two-layer MLP applied to each position separately:
- First layer: (d_model, d_ff)
- Second layer: (d_ff, d_model)
3. Summary Table
Layer Weight Matrix Size Self-Attention (Q) W_Q (d_model, d_model) Self-Attention (K) W_K (d_model, d_model) Self-Attention (V) W_V (d_model, d_model) Output projection W_O (d_model, d_model) Feed-Forward (1st) W_1 (d_model, d_ff) Feed-Forward (2nd) W_2 (d_ff, d_model) 4. Other Weights
- Embedding layer: (vocab_size, d_model)
- Positional encoding: (max_seq_length, d_model) (not always a learned weight)
In summary:
The main weight matrices in a Transformer block are of size (d_model, d_model) for attention projections, and (d_model, d_ff) and (d_ff, d_model) for the feed-forward layers. The exact sizes depend on the model configuration (d_model, d_ff, n_heads).
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.