Skip to content
TheChatArchiveThe Chat Archive
GPT 5.1 Codex Max· openaiHugging Face Frontier Reasoning Traces

Derive scaled dot-product attention. Explain the scaling factor and why large dot…

  1. User

    Derive scaled dot-product attention. Explain the scaling factor and why large dot products push softmax into vanishing gradient regions without scaling.

  2. Assistant

    Deriving scaled dot-product attention

    Scaled dot-product attention computes weights as softmax of the dot products between queries and keys divided by √d_k, where d_k is key dimension; this scaling normalizes the dot product variance from d_k to 1, preventing large magnitude inputs that would saturate softmax and cause vanishing gradients during training. A modern “dot–product” attention layer takes three sets of vectors

    • a set of queries (q_i\in\mathbb{R}^{d_k}),
    • a set of keys (k_j\in\mathbb{R}^{d_k}), and
    • a set of values (v_j\in\mathbb{R}^{d_v}),

    and uses a similarity score between (q_i) and (k_j) to mix the values. The simplest choice of score is the Euclidean inner product, [ s_{ij} ;=; q_i^\top k_j, ] and the corresponding attention weights are obtained by normalizing these scores with a softmax, [ \alpha_{ij} ;=; \frac{\exp(s_{ij})}{\sum_{j'} \exp(s_{ij'})}. ] The output for query (i) is then the weighted sum of values, [ \operatorname{Att}(q_i,K,V) ;=; \sum_j \alpha_{ij},v_j. ]

    Stacking all queries as rows of a matrix (Q\in\mathbb{R}^{n_q\times d_k}), keys as rows of (K\in\mathbb{R}^{n_k\times d_k}) and values as rows of (V\in\mathbb{R}^{n_k\times d_v}), this can be written compactly as [ \operatorname{Att}(Q,K,V) ;=; \operatorname{softmax}(QK^\top),V,, ] where the softmax is applied row‑wise.

    In practice, the raw dot products (q^\top k) grow in magnitude with the dimensionality. If the components of (q) and (k) are independent with zero mean and unit variance, then (\mathbb{E}[q^\top k]=0) and (\operatorname{Var}(q^\top k)=d_k): a typical dot product is on the order of (\sqrt{d_k}). Feeding numbers of that scale into the exponential of a softmax produces very sharp distributions; one or two keys receive almost all of the mass. The derivative of the softmax with respect to its inputs is [ \frac{\partial \alpha_{ij}}{\partial s_{i\ell}} = \alpha_{ij}(\delta_{j\ell} - \alpha_{i\ell}), ] which tends to zero when (\alpha_{ij}) is close to 0 or 1. Thus, unscaled dot–product attention in high dimensions quickly drives the network into a regime with near one‑hot attention weights and vanishing gradients.

    The “scaled” dot–product attention used in transformers mitigates this by normalizing the scores, [ \operatorname{Att}(Q,K,V) ;=; \operatorname{softmax}!\left(\frac{QK^\top}{\sqrt{d_k}}\right),V,. ] Dividing by (\sqrt{d_k}) ensures that, under the same unit‑variance assumption on (q) and (k), (\operatorname{Var}(q^\top k/\sqrt{d_k})\approx 1), keeping the softmax inputs in a regime where the outputs are not saturated and gradients remain well behaved.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.