Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro· GoogleHugging Face LMArena Human Preference 140K

Explain step-by-step instruction to perform backpropagation in paper

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-sonnet-4-20250514’s answer.
  1. User

    Explain step-by-step instruction to perform backpropagation in paper

  2. Gemini Assistant

    Of course! Here is a detailed, step-by-step guide to performing backpropagation on paper. We'll use a very simple neural network and a concrete example so you can follow the math.

    The Goal of Backpropagation

    The goal is simple: to adjust the network's weights and biases to make the network's output closer to the target (correct) output. We do this by figuring out how much "blame" each weight and bias shares for the total error, and then we nudge them in the direction that reduces that error.

    This "blame" is mathematically represented by the gradient (the partial derivative) of the total error with respect to each weight and bias.


    Setup: Our Sample Neural Network

    Let's define a simple network we can work with on paper.

    1. Architecture: 2 input neurons, 1 hidden layer with 2 neurons, and 1 output layer with 2 neurons.

      [image: Simple Neural Network Diagram] open

    2. Initial Weights and Biases: We'll assign some initial (usually random) values.

      • w1 = 0.15, w2 = 0.20, w3 = 0.25, w4 = 0.30
      • w5 = 0.40, w6 = 0.45, w7 = 0.50, w8 = 0.55
      • b1 = 0.35, b2 = 0.35 (Biases for the hidden and output layers)
    3. Training Sample:

      • Inputs: i1 = 0.05, i2 = 0.10
      • Target Outputs: target_o1 = 0.01, target_o2 = 0.99
    4. Formulas We Need:

      • Activation Function: Sigmoid function. It squashes values between 0 and 1.
        • σ(x)=11+e−xσ(x) = \frac{1}{1 + e^{-x}}
      • Derivative of Sigmoid: We need this for the backward pass. The derivative is simple:
        • σ′(x)=σ(x)∗(1−σ(x))σ'(x) = σ(x) * (1 - σ(x))
      • Loss Function: Mean Squared Error (MSE). We use the 1/2 version to simplify the derivative.
        • Etotal=12(target1−output1)2+12(target2−output2)2E_{total} = \frac{1}{2}(target_1 - output_1)^2 + \frac{1}{2}(target_2 - output_2)^2
      • Weight Update Rule:
        • wnew=wold−η∗∂Etotal∂woldw_{new} = w_{old} - η * \frac{∂E_{total}}{∂w_{old}}
        • η (eta) is the learning rate. We'll set η = 0.5.

    The Step-by-Step Calculation

    The process has three main parts:

    1. The Forward Pass: Calculate the network's output.
    2. The Backward Pass: Calculate the gradients (the "blame").
    3. The Update Step: Update the weights and biases.

    Part 1: The Forward Pass

    We feed the inputs forward through the network to get the final output.

    Step 1.1: Calculate Hidden Layer Outputs

    • Net input for h1: net_h1 = w1*i1 + w2*i2 + b1 net_h1 = (0.15 * 0.05) + (0.20 * 0.10) + 0.35 = 0.0075 + 0.02 + 0.35 = 0.3775

    • Output of h1: out_h1 = σ(net_h1) = 1 / (1 + e^-0.3775) = 0.59326

    • Net input for h2: net_h2 = w3*i1 + w4*i2 + b1 net_h2 = (0.25 * 0.05) + (0.30 * 0.10) + 0.35 = 0.0125 + 0.03 + 0.35 = 0.3925

    • Output of h2: out_h2 = σ(net_h2) = 1 / (1 + e^-0.3925) = 0.59688

    Step 1.2: Calculate Output Layer Outputs

    • Net input for o1: net_o1 = w5*out_h1 + w6*out_h2 + b2 net_o1 = (0.40 * 0.59326) + (0.45 * 0.59688) + 0.35 = 0.2373 + 0.2686 + 0.35 = 0.8559

    • Output of o1: out_o1 = σ(net_o1) = 1 / (1 + e^-0.8559) = 0.7018

    • Net input for o2: net_o2 = w7*out_h1 + w8*out_h2 + b2 net_o2 = (0.50 * 0.59326) + (0.55 * 0.59688) + 0.35 = 0.2966 + 0.3283 + 0.35 = 0.9749

    • Output of o2: out_o2 = σ(net_o2) = 1 / (1 + e^-0.9749) = 0.7261

    Step 1.3: Calculate the Total Error

    Now we compare our outputs (0.7018, 0.7261) with the targets (0.01, 0.99).

    • E1 = 1/2 * (target_o1 - out_o1)^2 = 1/2 * (0.01 - 0.7018)^2 = 0.2393
    • E2 = 1/2 * (target_o2 - out_o2)^2 = 1/2 * (0.99 - 0.7261)^2 = 0.0348
    • E_total = E1 + E2 = 0.2393 + 0.0348 = 0.2741

    Our network is quite wrong! Now let's go backward and find out why.


    Part 2: The Backward Pass (The Core of Backprop)

    We start from the end and work backward, calculating the gradient for each weight. This relies on the Chain Rule from calculus.

    The Chain Rule Logic: To find how w5 affects E_total, we chain the derivatives together: ∂E_total / ∂w5 = (∂E_total / ∂out_o1) * (∂out_o1 / ∂net_o1) * (∂net_o1 / ∂w5)

    Step 2.1: Gradients for the Output Layer Weights (w5, w6, w7, w8)

    Let's calculate the gradient for w5.

    1. ∂E_total / ∂out_o1: How much does the total error change with respect to the output of o1?

      • E_total = E1 + E2. Since out_o1 only affects E1, this is ∂E1 / ∂out_o1.
      • ∂E1 / ∂out_o1 = 2 * 1/2 * (target_o1 - out_o1)^(2-1) * -1 = -(target_o1 - out_o1)
      • = -(0.01 - 0.7018) = 0.6918
    2. ∂out_o1 / ∂net_o1: How much does the output of o1 change with respect to its net input? (This is the derivative of the sigmoid).

      • = out_o1 * (1 - out_o1) = 0.7018 * (1 - 0.7018) = 0.2092
    3. ∂net_o1 / ∂w5: How much does the net input of o1 change with respect to w5?

      • net_o1 = w5*out_h1 + w6*out_h2 + b2. The derivative with respect to w5 is just out_h1.
      • = out_h1 = 0.59326

    Now, multiply them all:

    • ∂E_total / ∂w5 = 0.6918 * 0.2092 * 0.59326 = 0.0858

    We can repeat this for w6, w7, and w8:

    • ∂E_total / ∂w6 = (∂E_total / ∂out_o1) * (∂out_o1 / ∂net_o1) * (∂net_o1 / ∂w6) = 0.6918 * 0.2092 * out_h2 = 0.0863
    • ∂E_total / ∂w7 = (∂E_total / ∂out_o2) * (∂out_o2 / ∂net_o2) * (∂net_o2 / ∂w7) = -(0.99 - 0.7261) * (0.7261 * (1-0.7261)) * out_h1 = (-0.2639) * (0.2004) * 0.59326 = -0.0314
    • ∂E_total / ∂w8 = (-0.2639) * (0.2004) * out_h2 = -0.0316

    Step 2.2: Gradients for the Hidden Layer Weights (w1, w2, w3, w4)

    This is the trickiest part. The output of a hidden neuron (out_h1) affects both out_o1 and out_o2. So, we have to sum the error contributions from both output neurons.

    Let's calculate the gradient for w1. ∂E_total / ∂w1 = (∂E_total / ∂out_h1) * (∂out_h1 / ∂net_h1) * (∂net_h1 / ∂w1)

    1. ∂E_total / ∂out_h1: This is the sum of the "blame" out_h1 gets from o1 and o2.

      • = (∂E_total / ∂net_o1) * (∂net_o1 / ∂out_h1) + (∂E_total / ∂net_o2) * (∂net_o2 / ∂out_h1)
      • The term (∂E_total / ∂net_o1) is just the first two parts of our w5 calculation: (∂E_total / ∂out_o1) * (∂out_o1 / ∂net_o1) = 0.6918 * 0.2092 = 0.1447. Let's call this the error signal δ_o1.
      • The term (∂E_total / ∂net_o2) is the first two parts of our w7 calculation: (-0.2639) * (0.2004) = -0.0529. Let's call this error signal δ_o2.
      • ∂net_o1 / ∂out_h1 is simply w5 = 0.40.
      • ∂net_o2 / ∂out_h1 is simply w7 = 0.50.
      • ∂E_total / ∂out_h1 = (δ_o1 * w5) + (δ_o2 * w7) = (0.1447 * 0.40) + (-0.0529 * 0.50) = 0.05788 - 0.02645 = 0.03143
    2. ∂out_h1 / ∂net_h1: Derivative of sigmoid for h1.

      • = out_h1 * (1 - out_h1) = 0.59326 * (1 - 0.59326) = 0.2413
    3. ∂net_h1 / ∂w1: Derivative of net_h1 with respect to w1.

      • = i1 = 0.05

    Now, multiply them all:

    • ∂E_total / ∂w1 = 0.03143 * 0.2413 * 0.05 = 0.000379

    You would repeat this entire process for w2, w3, and w4.


    Part 3: The Update Step

    Now we use our calculated gradients and the learning rate (η = 0.5) to update all the weights.

    Formula: w_new = w_old - η * gradient

    Update Output Layer Weights:

    • w5_new = w5 - η * (∂E_total / ∂w5) = 0.40 - 0.5 * 0.0858 = 0.3571
    • w6_new = w6 - η * (∂E_total / ∂w6) = 0.45 - 0.5 * 0.0863 = 0.40685
    • w7_new = w7 - η * (∂E_total / ∂w7) = 0.50 - 0.5 * (-0.0314) = 0.5157
    • w8_new = w8 - η * (∂E_total / ∂w8) = 0.55 - 0.5 * (-0.0316) = 0.5658

    Update Hidden Layer Weights:

    • w1_new = w1 - η * (∂E_total / ∂w1) = 0.15 - 0.5 * 0.000379 = 0.1498
    • (And so on for w2, w3, w4...)

    Updating Biases: Biases are updated in the exact same way as weights. The derivative of the net input with respect to the bias is always 1. For example, to update b2:

    • ∂E_total / ∂b2 = (δ_o1 * 1) + (δ_o2 * 1) = 0.1447 - 0.0529 = 0.0918
    • b2_new = b2 - η * (∂E_total / ∂b2) = 0.35 - 0.5 * 0.0918 = 0.3041

    Summary of the Process

    1. Forward Pass: Calculate the network's output and then the total error.
    2. Backward Pass:
      • For the output layer, calculate the error gradient (∂E_total/∂w) by multiplying three terms: (Error Derivative) * (Activation Derivative) * (Previous Layer's Output).
      • For hidden layers, calculate the error gradient by multiplying three terms: (Sum of Back-Propagated Error Signals) * (Activation Derivative) * (Previous Layer's Output).
    3. Update Weights: Use the calculated gradients and the learning rate to nudge every weight and bias in the direction that minimizes the error.

    You have now completed one full cycle of backpropagation! You would repeat this entire process thousands of times with many different training examples to fully train the network.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.