Skip to content
TheChatArchiveThe Chat Archive
gpt-4.1-2025-04-14· OpenAIHugging Face LMArena Human Preference 140K

explain the size of weight matrices in transformer architecture?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219-thinking-32k’s answer.
  1. User

    explain the size of weight matrices in transformer architecture?

  2. Assistant

    Certainly! In the Transformer architecture, weight matrices are central to its operations, especially in the attention mechanism and feed-forward layers. Here’s a breakdown of the main weight matrices and their sizes, assuming:

    • d_model: the model’s hidden size (e.g., 512 or 768)
    • d_ff: the feed-forward layer size (often 4×d_model)
    • n_heads: number of attention heads (e.g., 8 or 12)
    • d_k: dimension per head (d_model / n_heads)

    1. Self-Attention Layer

    For each attention head, we have three main projections:

    • Query (Q)
    • Key (K)
    • Value (V)

    Each of these is a linear transformation from the input embedding space to the head dimension.

    Weight matrices:

    • W_Q: (d_model, d_k)
    • W_K: (d_model, d_k)
    • W_V: (d_model, d_k)

    For multi-head attention (n_heads heads), these are often implemented as a single large matrix for efficiency:

    • W_Q: (d_model, d_model) (since n_heads × d_k = d_model)
    • W_K: (d_model, d_model)
    • W_V: (d_model, d_model)

    After attention, the outputs from all heads are concatenated and projected back to d_model:

    • W_O: (d_model, d_model)

    2. Feed-Forward Layer

    This is a two-layer MLP applied to each position separately:

    • First layer: (d_model, d_ff)
    • Second layer: (d_ff, d_model)

    3. Summary Table

    LayerWeight MatrixSize
    Self-Attention (Q)W_Q(d_model, d_model)
    Self-Attention (K)W_K(d_model, d_model)
    Self-Attention (V)W_V(d_model, d_model)
    Output projectionW_O(d_model, d_model)
    Feed-Forward (1st)W_1(d_model, d_ff)
    Feed-Forward (2nd)W_2(d_ff, d_model)

    4. Other Weights

    • Embedding layer: (vocab_size, d_model)
    • Positional encoding: (max_seq_length, d_model) (not always a learned weight)

    In summary:
    The main weight matrices in a Transformer block are of size (d_model, d_model) for attention projections, and (d_model, d_ff) and (d_ff, d_model) for the feed-forward layers. The exact sizes depend on the model configuration (d_model, d_ff, n_heads).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.