Derive scaled dot-product attention. Explain the scaling factor and why large dot…
- User
Derive scaled dot-product attention. Explain the scaling factor and why large dot products push softmax into vanishing gradient regions without scaling.
- Assistant
Deriving scaled dot-product attention
Scaled dot-product attention computes weights as softmax of the dot products between queries and keys divided by √d_k, where d_k is key dimension; this scaling normalizes the dot product variance from d_k to 1, preventing large magnitude inputs that would saturate softmax and cause vanishing gradients during training. A modern “dot–product” attention layer takes three sets of vectors
- a set of queries (q_i\in\mathbb{R}^{d_k}),
- a set of keys (k_j\in\mathbb{R}^{d_k}), and
- a set of values (v_j\in\mathbb{R}^{d_v}),
and uses a similarity score between (q_i) and (k_j) to mix the values. The simplest choice of score is the Euclidean inner product, [ s_{ij} ;=; q_i^\top k_j, ] and the corresponding attention weights are obtained by normalizing these scores with a softmax, [ \alpha_{ij} ;=; \frac{\exp(s_{ij})}{\sum_{j'} \exp(s_{ij'})}. ] The output for query (i) is then the weighted sum of values, [ \operatorname{Att}(q_i,K,V) ;=; \sum_j \alpha_{ij},v_j. ]
Stacking all queries as rows of a matrix (Q\in\mathbb{R}^{n_q\times d_k}), keys as rows of (K\in\mathbb{R}^{n_k\times d_k}) and values as rows of (V\in\mathbb{R}^{n_k\times d_v}), this can be written compactly as [ \operatorname{Att}(Q,K,V) ;=; \operatorname{softmax}(QK^\top),V,, ] where the softmax is applied row‑wise.
In practice, the raw dot products (q^\top k) grow in magnitude with the dimensionality. If the components of (q) and (k) are independent with zero mean and unit variance, then (\mathbb{E}[q^\top k]=0) and (\operatorname{Var}(q^\top k)=d_k): a typical dot product is on the order of (\sqrt{d_k}). Feeding numbers of that scale into the exponential of a softmax produces very sharp distributions; one or two keys receive almost all of the mass. The derivative of the softmax with respect to its inputs is [ \frac{\partial \alpha_{ij}}{\partial s_{i\ell}} = \alpha_{ij}(\delta_{j\ell} - \alpha_{i\ell}), ] which tends to zero when (\alpha_{ij}) is close to 0 or 1. Thus, unscaled dot–product attention in high dimensions quickly drives the network into a regime with near one‑hot attention weights and vanishing gradients.
The “scaled” dot–product attention used in transformers mitigates this by normalizing the scores, [ \operatorname{Att}(Q,K,V) ;=; \operatorname{softmax}!\left(\frac{QK^\top}{\sqrt{d_k}}\right),V,. ] Dividing by (\sqrt{d_k}) ensures that, under the same unit‑variance assumption on (q) and (k), (\operatorname{Var}(q^\top k/\sqrt{d_k})\approx 1), keeping the softmax inputs in a regime where the outputs are not saturated and gradients remain well behaved.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.