Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen3-235b-a22b-no-thinking· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

Please explain Xavier initialization of parameters in neural network. Why the following…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-sonnet-4-20250514’s answer.
  1. User

    Please explain Xavier initialization of parameters in neural network. Why the following approach works. Don't simplify explanation, because i know math. But try to explain every detail.without omitting obvious steps

  2. Qwen Assistant

    Certainly! Let's dive into Xavier initialization (also known as Glorot initialization) in detail, with full mathematical rigor and reasoning. This is one of the most important techniques in deep learning for initializing neural network weights to prevent exploding or vanishing gradients during training.


    🧠 Goal of Xavier Initialization

    The main idea behind Xavier initialization is to keep the variance of the activations and gradients roughly the same across all layers, both during forward propagation and backpropagation.

    If we don't initialize weights carefully:

    • Gradients may vanish: signals become too small → no learning.
    • Gradients may explode: signals become too large → unstable learning.

    So, we want an initialization that maintains a stable distribution of activations and gradients throughout the network.


    🔢 Mathematical Assumptions

    To derive Xavier initialization, we make some simplifying assumptions:

    1. Linear activations: For theoretical derivation, assume linear activation functions (i.e., identity). The results are heuristic but still work well for nonlinearities like tanh.
    2. Zero-mean inputs: Input features are centered around zero (e.g., normalized).
    3. Weights initialized independently: Each weight Wij∼N(0,σ2)W_{ij} \sim \mathcal{N}(0, \sigma^2) or uniformly distributed.
    4. No bias terms considered in this derivation (for simplicity; they can be initialized to zero).

    📐 Forward Propagation Variance Derivation

    Let’s consider a single neuron in a fully connected layer:

    yj=∑i=1ninwjixiy_j = \sum_{i=1}^{n_{\text{in}}} w_{ji} x_i

    Where:

    • ninn_{\text{in}}: number of inputs to the neuron (fan-in)
    • wjiw_{ji}: weight connecting input ii to neuron jj
    • xix_i: input to neuron jj

    We want to compute the variance of yjy_j:

    Var(yj)=Var(∑i=1ninwjixi)\text{Var}(y_j) = \text{Var}\left( \sum_{i=1}^{n_{\text{in}}} w_{ji} x_i \right)

    Assuming all wjiw_{ji} and xix_i are independent and identically distributed (i.i.d.), and have zero mean:

    Var(yj)=∑i=1ninVar(wjixi)\text{Var}(y_j) = \sum_{i=1}^{n_{\text{in}}} \text{Var}(w_{ji} x_i)

    Using the property Var(ab)=Var(a)Var(b)\text{Var}(ab) = \text{Var}(a)\text{Var}(b) if aa and bb are independent and zero-mean:

    Var(wjixi)=Var(wji)⋅Var(xi)\text{Var}(w_{ji} x_i) = \text{Var}(w_{ji}) \cdot \text{Var}(x_i)

    Since all wjiw_{ji} and xix_i are identically distributed:

    Var(yj)=nin⋅Var(w)⋅Var(x)\text{Var}(y_j) = n_{\text{in}} \cdot \text{Var}(w) \cdot \text{Var}(x)

    This tells us how the variance scales from input to output through a layer.

    Goal:

    We want the variance to remain constant across layers:

    Var(y)=Var(x)⇒nin⋅Var(w)=1⇒Var(w)=1nin\text{Var}(y) = \text{Var}(x) \Rightarrow n_{\text{in}} \cdot \text{Var}(w) = 1 \Rightarrow \text{Var}(w) = \frac{1}{n_{\text{in}}}

    That gives us a condition on the variance of the weight initialization depending on the fan-in.


    ⬅️ Backward Propagation Variance Derivation

    Now consider gradient flow backwards through the network.

    For a loss function LL, the gradient with respect to the weights is:

    ∂L∂wji=∂L∂yj⋅xi\frac{\partial L}{\partial w_{ji}} = \frac{\partial L}{\partial y_j} \cdot x_i

    So the variance of the gradient depends on both ∂L∂yj\frac{\partial L}{\partial y_j} and xix_i. To maintain stable gradients, we also need:

    Var(∂L∂w)=const⇒Var(∂L∂y)⋅Var(x)=const\text{Var}\left(\frac{\partial L}{\partial w}\right) = \text{const} \Rightarrow \text{Var}\left(\frac{\partial L}{\partial y}\right) \cdot \text{Var}(x) = \text{const}

    A similar analysis shows that to keep the variance of gradients stable, we should use:

    Var(w)=1nout\text{Var}(w) = \frac{1}{n_{\text{out}}}

    Where noutn_{\text{out}} is the number of neurons in the current layer (fan-out), because each gradient is summed over outputs.


    ✨ Xavier Initialization Strategy

    To satisfy both forward and backward propagation stability, Glorot & Bengio (2010) proposed using the harmonic mean of fan-in and fan-out:

    Var(w)=2nin+nout\text{Var}(w) = \frac{2}{n_{\text{in}} + n_{\text{out}}}

    This balances the variance scaling in both directions.

    Two Common Implementations:

    1. Normal Distribution Initialization

    W∼N(0,2nin+nout)W \sim \mathcal{N}\left(0, \sqrt{\frac{2}{n_{\text{in}} + n_{\text{out}}}}\right)

    2. Uniform Distribution Initialization

    W∼U(−a,a),where a=6nin+noutW \sim \mathcal{U}\left(-a, a\right), \quad \text{where } a = \sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}

    This ensures that the variance of the uniform distribution matches the desired normal distribution variance:

    Var(W)=(2a)212=a23=2nin+nout⇒a=6nin+nout\text{Var}(W) = \frac{(2a)^2}{12} = \frac{a^2}{3} = \frac{2}{n_{\text{in}} + n_{\text{out}}} \Rightarrow a = \sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}

    🔍 Why This Works — Summary

    1. Forward pass: Ensures activations do not blow up or shrink exponentially.
    2. Backward pass: Ensures gradients do not vanish or explode.
    3. Balance between fan-in and fan-out: Xavier initialization accounts for both the number of inputs and outputs per layer.
    4. Works empirically well with symmetric nonlinearities like tanh, which preserve zero-mean and bounded variance better than ReLU-type activations.

    🚫 When Not to Use Xavier Initialization

    • With ReLU and its variants (Leaky ReLU, etc.), the nonlinearity introduces asymmetry and causes half the units to be inactive. In such cases, He initialization (which uses only fan-in) is preferred:
    Var(w)=2nin\text{Var}(w) = \frac{2}{n_{\text{in}}}

    Because ReLU doesn’t propagate negative values, it changes the effective variance behavior.


    📌 Final Notes

    • Xavier initialization assumes linearity and symmetric activation functions. It works well for tanh and sigmoid, less so for ReLU.
    • The key insight is that the variance of the weights must scale inversely with the number of connections to avoid signal distortion.
    • Proper initialization helps stabilize training, especially in deep networks, by preserving the dynamic range of signals.

    Let me know if you'd like a code example or comparison with He initialization!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.