Skip to content
The Chat Archive
gpt-4o-2024-08-06· OpenAI WildChat-4.8M (AI2)

KL ( p θ ( w ∣ w 1 : i − 1 ) ∥ p θ ′ ( w ∣ w 1 : i − 1 ) ) = log ( 1 + 1 Z ) So, if Z is…

  1. User

    KL ( p θ ( w ∣ w 1 : i − 1 ) ∥ p θ ′ ( w ∣ w 1 : i − 1 ) )

    log ( 1 + 1 Z )

    So, if Z is small relative to 1 , we get large KL-divergence between the pre- and post-expansion LMs.

    Why would this happen? If all h ⊤ i − 1 e i are large and negative. Because the softmax function is invariant to adding any scalar to all inputs, my best guess is that this happens sort of by accident of optimization. In general, this doesn’t happen for all (or even most) LMs, but when it does, zero-initialization is bad.

    feel confused about the softmax function is invariant to adding any scalar to all inputs and why it is mentioned here. Please explain in simple words and extended logic with examples if needed.

  2. Assistant

    To understand why the softmax function is invariant to adding a scalar to all inputs and why it's relevant in this context, let's break it down step by step.

    What is the Softmax Function?

    The softmax function is used to convert a vector of raw scores (logits) into probabilities. Given a vector ( \mathbf{z} = (z_1, z_2, ..., z_n) ), the softmax function is defined as:

    [ \text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}} ]

    This transforms the raw scores ( z_i ) into a probability distribution where each element is between 0 and 1, and they all sum up to 1.

    Softmax Invariance to Adding a Scalar

    The softmax function is invariant to adding a constant scalar to all the inputs. This means if you add the same number ( c ) to each component of ( \mathbf{z} ), the output probabilities remain unchanged. Mathematically:

    [ \text{softmax}(z_i + c) = \frac{e^{z_i + c}}{\sum_{j=1}^{n} e^{z_j + c}} = \frac{e^{z_i} \cdot e^{c}}{\sum_{j=1}^{n} e^{z_j} \cdot e^{c}} = \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}} = \text{softmax}(z_i) ]

    This happens because the common factor ( e^c ) cancels out in the numerator and the denominator.

    Why is this Relevant to KL-Divergence and Optimization?

    When discussing the KL-divergence between two language models, ( p_\theta ) and ( p_{\theta'} ), we are interested in how much one probability distribution diverges from another. In our case, ( Z ) represents a measure of this divergence.

    Large Values of ( h_i^\top e_i ):

    When ( h_i^\top e_i ) returns large and negative values, it implies that the raw scores (logits before the softmax) are significantly lower for the corresponding words. This can lead to low probabilities after applying softmax, as exponential functions grow more rapidly with positive values and shrink with negative ones.

    Zero Initialization and Optimization "Accidents":

    In neural network training, beginning with zero-initialized weights can sometimes lead to poor convergence. If the optimization process inadvertently causes weights to adjust such that many negative large scores occur consistently across models (causing ( h_i^\top e_i ) to be large and negative), it implies scores are uniformly low, potentially skewing distributions in a certain way.

    Summary

    The invariant property of the softmax means that any absolute scaling of scores does not change the resulting distributions unless interactions among scores themselves change (e.g., one score growing relative to others). This property ensures that focusing on relative changes is key when considering differences between language models' distributions pre- and post-optimization. Here, introducing zero-bias conditions potentially masks such significant relative shifts, making certain optimizations "unlucky," leading them to a poor divergence measure ( Z ) and thus large KL-divergence.