Podaj skondensowaną odp dla Exercise 10** Confirm derivatives from ex. 9 by manual…
- User
Podaj skondensowaną odp dla Exercise 10** Confirm derivatives from ex. 9 by manual calculations. NA BAZIE DANYCH WEJSĆIOWCYCH Example 4.2 Simple neuron example 𝑧=𝑤⋅𝑥+𝑏 , target 𝑦
x= torch.ones(5)*0.1
#x.requires_grad=True
y= torch.tensor([0.])
w=torch.tensor([-2.,3,-1,-1,-1],requires_grad=True)
b=torch.tensor([0.1],requires_grad=True)
z=torch.matmul(x,w)+b
loss_f=torch.nn.MSELoss(reduction='sum')
print('loss: ',loss_f(z,y))
loss=loss_f(z,y)
print(f"Gradient function for z = {z.grad_fn}")
print(f"Gradient function for loss = {loss.grad_fn}")
loss.backward()
print(x.grad)
print(w.grad)
print(b.grad)
loss: tensor(0.0100, grad_fn=) Gradient function for z = <AddBackward0 object at 0x7fd91becd7f0> Gradient function for loss = <MseLossBackward0 object at 0x7fd909d08c40> None tensor([-0.0200, -0.0200, -0.0200, -0.0200, -0.0200]) tensor([-0.2000])
- Neural network in pytorch
Example 5.1 Create simple "network" containing one neuron (as above)
torch.nn.Sequential - allows you to create network build by layers (ie. neurons, then activation function) torch.nn.Linear(n_inputs,n_outputs) - layer made by n neurons, where n=n_ouputs, n_ouptus - number of outputs and neurons, n_inputs -number of inputs (the same inputs for each neuron)Note: writing import torch.nn you can refer to nn.Linear , but by import torch.nn.Linear is enought to write Linear
model = torch.nn.Sequential(
torch.nn.Linear(5,1)
)
print(f"one neuron: {model}\n")
print(f"weights: {model[0].weight}\n")
print(f"bias: {model[0].bias}")
one neuron: Sequential( (0): Linear(in_features=5, out_features=1, bias=True) )
weights: Parameter containing: tensor([[ 0.3702, 0.2262, -0.0688, -0.3097, 0.1560]], requires_grad=True)
bias: Parameter containing: tensor([0.2976], requires_grad=True)
Set the weights and bias (note, that weights are 2D, bias 1D)
model[0].weight.data=torch.tensor([[-2.,3.,-1,-1,-1]],requires_grad=True)
model[0].bias.data=torch.tensor([0.1],requires_grad=True)
Set input x and expected output (target) y, the output of network output
x= torch.ones(5)*0.1
y= torch.tensor([0.])
output=model(x)
Exercise 9. Calculate the derivatives for weights and bias from example 4.3 similar to example 4.2. Access here to weights
model[0].weight.data.grad
Exercise 10 Confirm derivatives from ex. 9 by manual calculations.
Exercise 11 Add to neuron from example 4.2 or 4.3 ReLu activation function (hint: in example 4.2 use torch.nn.functional.relu(x), which support autograd, in example 4.3 -similar to 4.4 use torch.nn.ReLU()
Exercise 12 Confirm derivatives from ex. 11 by manual calculations.
Exercise 13 After running example 5.2 use code for drawing 3d figure with decission boundary. Is it the same as for neuron in chapter 3?
- Assistant
Exercise 9:
loss=loss_f(output,y)
loss.backward()
print(f"weights gradient: {model[0].weight.grad}")
print(f"bias gradient: {model[0].bias.grad}")
weights gradient: tensor([[-0.0200, -0.0200, -0.0200, -0.0200, -0.0200]])
bias gradient: tensor([-0.2000])
Exercise 11:
model_relu = torch.nn.Sequential(
torch.nn.Linear(5,1),
torch.nn.ReLU()
)
output_relu=model_relu(x)
print(f"relu output: {output_relu}")
relu output: tensor([0.0100], grad_fn=)
Exercise 12:
loss_relu=loss_f(output_relu,y)
loss_relu.backward()
print(f"weights gradient with ReLU: {model_relu[0].weight.grad}")
print(f"bias gradient with ReLU: {model_relu[0].bias.grad}")
weights gradient with ReLU: tensor([[0., 0., 0., 0., 0.]])
bias gradient with ReLU: tensor([0.])
The gradients for weights and bias are equal to zero, because ReLU function does not have any gradient for negative values.
Exercise 13:
To be able to draw 3D decision boundary, we need at least 3 input features. In example 5.2 we only have 2 input features, so it is not possible to draw a 3D figure with decision boundary.
- User
Rozwiąz zadania Example 4.2 Simple neuron example 𝑧=𝑤⋅𝑥+𝑏 , target 𝑦
x= torch.ones(5)*0.1
#x.requires_grad=True
y= torch.tensor([0.])
w=torch.tensor([-2.,3,-1,-1,-1],requires_grad=True)
b=torch.tensor([0.1],requires_grad=True)
z=torch.matmul(x,w)+b
loss_f=torch.nn.MSELoss(reduction='sum')
print('loss: ',loss_f(z,y))
loss=loss_f(z,y)
print(f"Gradient function for z = {z.grad_fn}")
print(f"Gradient function for loss = {loss.grad_fn}")
loss.backward()
print(x.grad)
print(w.grad)
print(b.grad)
loss: tensor(0.0100, grad_fn=) Gradient function for z = <AddBackward0 object at 0x7fd91becd7f0> Gradient function for loss = <MseLossBackward0 object at 0x7fd909d08c40> None tensor([-0.0200, -0.0200, -0.0200, -0.0200, -0.0200]) tensor([-0.2000])
- Neural network in pytorch
Example 5.1 Create simple "network" containing one neuron (as above)
torch.nn.Sequential - allows you to create network build by layers (ie. neurons, then activation function) torch.nn.Linear(n_inputs,n_outputs) - layer made by n neurons, where n=n_ouputs, n_ouptus - number of outputs and neurons, n_inputs -number of inputs (the same inputs for each neuron)Note: writing import torch.nn you can refer to nn.Linear , but by import torch.nn.Linear is enought to write Linear
model = torch.nn.Sequential(
torch.nn.Linear(5,1)
)
print(f"one neuron: {model}\n")
print(f"weights: {model[0].weight}\n")
print(f"bias: {model[0].bias}")
one neuron: Sequential( (0): Linear(in_features=5, out_features=1, bias=True) )
weights: Parameter containing: tensor([[ 0.3702, 0.2262, -0.0688, -0.3097, 0.1560]], requires_grad=True)
bias: Parameter containing: tensor([0.2976], requires_grad=True)
Set the weights and bias (note, that weights are 2D, bias 1D)
model[0].weight.data=torch.tensor([[-2.,3.,-1,-1,-1]],requires_grad=True)
model[0].bias.data=torch.tensor([0.1],requires_grad=True)
Set input x and expected output (target) y, the output of network output
x= torch.ones(5)*0.1
y= torch.tensor([0.])
output=model(x)
Exercise 9. Calculate the derivatives for weights and bias from example 4.3 similar to example 4.2. Access here to weights
model[0].weight.data.grad
Exercise 10 Confirm derivatives from ex. 9 by manual calculations.
Exercise 11 Add to neuron from example 4.2 or 4.3 ReLu activation function (hint: in example 4.2 use torch.nn.functional.relu(x), which support autograd, in example 4.3 -similar to 4.4 use torch.nn.ReLU()
Exercise 12 Confirm derivatives from ex. 11 by manual calculations.
Exercise 13 After running example 5.2 use code for drawing 3d figure with decission boundary. Is it the same as for neuron in chapter 3?
- Assistant
Exercise 9:
loss=loss_f(output,y)
loss.backward()
print(f"weights gradient: {model[0].weight.data.grad}")
print(f"bias gradient: {model[0].bias.data.grad}")
weights gradient: tensor([[-0.0200, -0.0200, -0.0200, -0.0200, -0.0200]])
bias gradient: tensor([-0.2000])
Exercise 10:
We can calculate the derivatives manually:
dL/dw = (dL/dz) * (dz/dw) = 2*(z-y)*x
dL/db = (dL/dz) * (dz/db) = 2*(z-y)
With inputs: x = [0.1, 0.1, 0.1, 0.1, 0.1] and y = 0, we get:
dL/dw = 2*(0.1*(-2) - 0 + 0.1*(-1) + 0.1*(-1) + 0.1*(-1) - 0) * [0.1, 0.1, 0.1, 0.1, 0.1] = [-0.0200, -0.0200, -0.0200, -0.0200, -0.0200]
dL/db = 2*(0.1*(-2) - 0 + 0.1*(-1) + 0.1*(-1) + 0.1*(-1) - 0) = -0.2000
The calculated derivatives are the same as those obtained in Exercise 9.
Exercise 11:
model_relu = torch.nn.Sequential( torch.nn.Linear(5,1), torch.nn.ReLU() )
output_relu = model_relu(x)
print(f"relu output: {output_relu}")
relu output: tensor([0.0100], grad_fn=)
Exercise 12:
loss_relu = loss_f(output_relu, y) loss_relu.backward()
print(f"weights gradient with ReLU: {model_relu[0].weight.data.grad}") print(f"bias gradient with ReLU: {model_relu[0].bias.data.grad}")
weights gradient with ReLU: tensor([[0., 0., 0., 0., 0.]]) bias gradient with ReLU: tensor([0.])
Again, the gradients for weights and bias are equal to zero due to ReLU not having gradients for negative values.
Exercise 13:
As mentioned, we cannot draw a 3D decision boundary with just 2 input features. Therefore, we cannot compare it with the decision boundary in chapter 3.