Explain step-by-step instruction to perform backpropagation in paper
- User
Explain step-by-step instruction to perform backpropagation in paper
Assistant
Of course! Here is a detailed, step-by-step guide to performing backpropagation on paper. We'll use a very simple neural network and a concrete example so you can follow the math.
The Goal of Backpropagation
The goal is simple: to adjust the network's weights and biases to make the network's output closer to the target (correct) output. We do this by figuring out how much "blame" each weight and bias shares for the total error, and then we nudge them in the direction that reduces that error.
This "blame" is mathematically represented by the gradient (the partial derivative) of the total error with respect to each weight and bias.
Setup: Our Sample Neural Network
Let's define a simple network we can work with on paper.
-
Architecture: 2 input neurons, 1 hidden layer with 2 neurons, and 1 output layer with 2 neurons.
[image: Simple Neural Network Diagram] open
-
Initial Weights and Biases: We'll assign some initial (usually random) values.
w1 = 0.15,w2 = 0.20,w3 = 0.25,w4 = 0.30w5 = 0.40,w6 = 0.45,w7 = 0.50,w8 = 0.55b1 = 0.35,b2 = 0.35(Biases for the hidden and output layers)
-
Training Sample:
- Inputs:
i1 = 0.05,i2 = 0.10 - Target Outputs:
target_o1 = 0.01,target_o2 = 0.99
- Inputs:
-
Formulas We Need:
- Activation Function: Sigmoid function. It squashes values between 0 and 1.
- Derivative of Sigmoid: We need this for the backward pass. The derivative is simple:
- Loss Function: Mean Squared Error (MSE). We use the
1/2version to simplify the derivative. - Weight Update Rule:
η(eta) is the learning rate. We'll setη = 0.5.
- Activation Function: Sigmoid function. It squashes values between 0 and 1.
The Step-by-Step Calculation
The process has three main parts:
- The Forward Pass: Calculate the network's output.
- The Backward Pass: Calculate the gradients (the "blame").
- The Update Step: Update the weights and biases.
Part 1: The Forward Pass
We feed the inputs forward through the network to get the final output.
Step 1.1: Calculate Hidden Layer Outputs
-
Net input for h1:
net_h1 = w1*i1 + w2*i2 + b1net_h1 = (0.15 * 0.05) + (0.20 * 0.10) + 0.35 = 0.0075 + 0.02 + 0.35 = 0.3775 -
Output of h1:
out_h1 = σ(net_h1) = 1 / (1 + e^-0.3775) = 0.59326 -
Net input for h2:
net_h2 = w3*i1 + w4*i2 + b1net_h2 = (0.25 * 0.05) + (0.30 * 0.10) + 0.35 = 0.0125 + 0.03 + 0.35 = 0.3925 -
Output of h2:
out_h2 = σ(net_h2) = 1 / (1 + e^-0.3925) = 0.59688
Step 1.2: Calculate Output Layer Outputs
-
Net input for o1:
net_o1 = w5*out_h1 + w6*out_h2 + b2net_o1 = (0.40 * 0.59326) + (0.45 * 0.59688) + 0.35 = 0.2373 + 0.2686 + 0.35 = 0.8559 -
Output of o1:
out_o1 = σ(net_o1) = 1 / (1 + e^-0.8559) = 0.7018 -
Net input for o2:
net_o2 = w7*out_h1 + w8*out_h2 + b2net_o2 = (0.50 * 0.59326) + (0.55 * 0.59688) + 0.35 = 0.2966 + 0.3283 + 0.35 = 0.9749 -
Output of o2:
out_o2 = σ(net_o2) = 1 / (1 + e^-0.9749) = 0.7261
Step 1.3: Calculate the Total Error
Now we compare our outputs (
0.7018,0.7261) with the targets (0.01,0.99).E1 = 1/2 * (target_o1 - out_o1)^2 = 1/2 * (0.01 - 0.7018)^2 = 0.2393E2 = 1/2 * (target_o2 - out_o2)^2 = 1/2 * (0.99 - 0.7261)^2 = 0.0348E_total = E1 + E2 = 0.2393 + 0.0348 = 0.2741
Our network is quite wrong! Now let's go backward and find out why.
Part 2: The Backward Pass (The Core of Backprop)
We start from the end and work backward, calculating the gradient for each weight. This relies on the Chain Rule from calculus.
The Chain Rule Logic: To find how
w5affectsE_total, we chain the derivatives together:∂E_total / ∂w5 = (∂E_total / ∂out_o1) * (∂out_o1 / ∂net_o1) * (∂net_o1 / ∂w5)Step 2.1: Gradients for the Output Layer Weights (w5, w6, w7, w8)
Let's calculate the gradient for
w5.-
∂E_total / ∂out_o1: How much does the total error change with respect to the output ofo1?E_total = E1 + E2. Sinceout_o1only affectsE1, this is∂E1 / ∂out_o1.∂E1 / ∂out_o1 = 2 * 1/2 * (target_o1 - out_o1)^(2-1) * -1 = -(target_o1 - out_o1)= -(0.01 - 0.7018) = 0.6918
-
∂out_o1 / ∂net_o1: How much does the output ofo1change with respect to its net input? (This is the derivative of the sigmoid).= out_o1 * (1 - out_o1) = 0.7018 * (1 - 0.7018) = 0.2092
-
∂net_o1 / ∂w5: How much does the net input ofo1change with respect tow5?net_o1 = w5*out_h1 + w6*out_h2 + b2. The derivative with respect tow5is justout_h1.= out_h1 = 0.59326
Now, multiply them all:
∂E_total / ∂w5=0.6918 * 0.2092 * 0.59326 = 0.0858
We can repeat this for
w6,w7, andw8:∂E_total / ∂w6=(∂E_total / ∂out_o1) * (∂out_o1 / ∂net_o1) * (∂net_o1 / ∂w6)=0.6918 * 0.2092 * out_h2=0.0863∂E_total / ∂w7=(∂E_total / ∂out_o2) * (∂out_o2 / ∂net_o2) * (∂net_o2 / ∂w7)=-(0.99 - 0.7261) * (0.7261 * (1-0.7261)) * out_h1=(-0.2639) * (0.2004) * 0.59326=-0.0314∂E_total / ∂w8=(-0.2639) * (0.2004) * out_h2=-0.0316
Step 2.2: Gradients for the Hidden Layer Weights (w1, w2, w3, w4)
This is the trickiest part. The output of a hidden neuron (
out_h1) affects bothout_o1andout_o2. So, we have to sum the error contributions from both output neurons.Let's calculate the gradient for
w1.∂E_total / ∂w1 = (∂E_total / ∂out_h1) * (∂out_h1 / ∂net_h1) * (∂net_h1 / ∂w1)-
∂E_total / ∂out_h1: This is the sum of the "blame"out_h1gets fromo1ando2.= (∂E_total / ∂net_o1) * (∂net_o1 / ∂out_h1) + (∂E_total / ∂net_o2) * (∂net_o2 / ∂out_h1)- The term
(∂E_total / ∂net_o1)is just the first two parts of ourw5calculation:(∂E_total / ∂out_o1) * (∂out_o1 / ∂net_o1) = 0.6918 * 0.2092 = 0.1447. Let's call this the error signal δ_o1. - The term
(∂E_total / ∂net_o2)is the first two parts of ourw7calculation:(-0.2639) * (0.2004) = -0.0529. Let's call this error signal δ_o2. ∂net_o1 / ∂out_h1is simplyw5 = 0.40.∂net_o2 / ∂out_h1is simplyw7 = 0.50.∂E_total / ∂out_h1 = (δ_o1 * w5) + (δ_o2 * w7) = (0.1447 * 0.40) + (-0.0529 * 0.50) = 0.05788 - 0.02645 = 0.03143
-
∂out_h1 / ∂net_h1: Derivative of sigmoid forh1.= out_h1 * (1 - out_h1) = 0.59326 * (1 - 0.59326) = 0.2413
-
∂net_h1 / ∂w1: Derivative ofnet_h1with respect tow1.= i1 = 0.05
Now, multiply them all:
∂E_total / ∂w1=0.03143 * 0.2413 * 0.05 = 0.000379
You would repeat this entire process for
w2,w3, andw4.
Part 3: The Update Step
Now we use our calculated gradients and the learning rate (
η = 0.5) to update all the weights.Formula:
w_new = w_old - η * gradientUpdate Output Layer Weights:
w5_new = w5 - η * (∂E_total / ∂w5) = 0.40 - 0.5 * 0.0858 = 0.3571w6_new = w6 - η * (∂E_total / ∂w6) = 0.45 - 0.5 * 0.0863 = 0.40685w7_new = w7 - η * (∂E_total / ∂w7) = 0.50 - 0.5 * (-0.0314) = 0.5157w8_new = w8 - η * (∂E_total / ∂w8) = 0.55 - 0.5 * (-0.0316) = 0.5658
Update Hidden Layer Weights:
w1_new = w1 - η * (∂E_total / ∂w1) = 0.15 - 0.5 * 0.000379 = 0.1498- (And so on for w2, w3, w4...)
Updating Biases: Biases are updated in the exact same way as weights. The derivative of the net input with respect to the bias is always 1. For example, to update
b2:∂E_total / ∂b2 = (δ_o1 * 1) + (δ_o2 * 1) = 0.1447 - 0.0529 = 0.0918b2_new = b2 - η * (∂E_total / ∂b2) = 0.35 - 0.5 * 0.0918 = 0.3041
Summary of the Process
- Forward Pass: Calculate the network's output and then the total error.
- Backward Pass:
- For the output layer, calculate the error gradient (
∂E_total/∂w) by multiplying three terms:(Error Derivative) * (Activation Derivative) * (Previous Layer's Output). - For hidden layers, calculate the error gradient by multiplying three terms:
(Sum of Back-Propagated Error Signals) * (Activation Derivative) * (Previous Layer's Output).
- For the output layer, calculate the error gradient (
- Update Weights: Use the calculated gradients and the learning rate to nudge every weight and bias in the direction that minimizes the error.
You have now completed one full cycle of backpropagation! You would repeat this entire process thousands of times with many different training examples to fully train the network.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.