Skip to content
The Chat Archive
gpt-3.5-turbo-0301· OpenAI WildChat-4.8M (AI2)

Podaj skondensowaną odp dla Exercise 10** Confirm derivatives from ex. 9 by manual…

  1. User

    Podaj skondensowaną odp dla Exercise 10** Confirm derivatives from ex. 9 by manual calculations. NA BAZIE DANYCH WEJSĆIOWCYCH Example 4.2 Simple neuron example 𝑧=𝑤⋅𝑥+𝑏 , target 𝑦

    x= torch.ones(5)*0.1

    #x.requires_grad=True

    y= torch.tensor([0.])

    w=torch.tensor([-2.,3,-1,-1,-1],requires_grad=True)

    b=torch.tensor([0.1],requires_grad=True)

    z=torch.matmul(x,w)+b

    loss_f=torch.nn.MSELoss(reduction='sum')

    print('loss: ',loss_f(z,y))

    loss=loss_f(z,y)

    print(f"Gradient function for z = {z.grad_fn}")

    print(f"Gradient function for loss = {loss.grad_fn}")

    loss.backward()

    print(x.grad)

    print(w.grad)

    print(b.grad)

    loss: tensor(0.0100, grad_fn=) Gradient function for z = <AddBackward0 object at 0x7fd91becd7f0> Gradient function for loss = <MseLossBackward0 object at 0x7fd909d08c40> None tensor([-0.0200, -0.0200, -0.0200, -0.0200, -0.0200]) tensor([-0.2000])

    1. Neural network in pytorch

    Example 5.1 Create simple "network" containing one neuron (as above)

    torch.nn.Sequential - allows you to create network build by layers (ie. neurons, then activation function)
    
    torch.nn.Linear(n_inputs,n_outputs) - layer made by n neurons, where n=n_ouputs, n_ouptus - number of outputs and neurons, n_inputs -number of inputs (the same inputs for each neuron)
    

    Note: writing import torch.nn you can refer to nn.Linear , but by import torch.nn.Linear is enought to write Linear

    model = torch.nn.Sequential(

    torch.nn.Linear(5,1)

    )

    print(f"one neuron: {model}\n")

    print(f"weights: {model[0].weight}\n")

    print(f"bias: {model[0].bias}")

    one neuron: Sequential( (0): Linear(in_features=5, out_features=1, bias=True) )

    weights: Parameter containing: tensor([[ 0.3702, 0.2262, -0.0688, -0.3097, 0.1560]], requires_grad=True)

    bias: Parameter containing: tensor([0.2976], requires_grad=True)

    Set the weights and bias (note, that weights are 2D, bias 1D)

    model[0].weight.data=torch.tensor([[-2.,3.,-1,-1,-1]],requires_grad=True)

    model[0].bias.data=torch.tensor([0.1],requires_grad=True)

    Set input x and expected output (target) y, the output of network output

    x= torch.ones(5)*0.1

    y= torch.tensor([0.])

    output=model(x)

    Exercise 9. Calculate the derivatives for weights and bias from example 4.3 similar to example 4.2. Access here to weights model[0].weight.data.grad

    ​

    Exercise 10 Confirm derivatives from ex. 9 by manual calculations.

    ​

    Exercise 11 Add to neuron from example 4.2 or 4.3 ReLu activation function (hint: in example 4.2 use torch.nn.functional.relu(x), which support autograd, in example 4.3 -similar to 4.4 use torch.nn.ReLU()

    ​

    Exercise 12 Confirm derivatives from ex. 11 by manual calculations.

    ​

    Exercise 13 After running example 5.2 use code for drawing 3d figure with decission boundary. Is it the same as for neuron in chapter 3?

  2. Assistant

    Exercise 9:

    loss=loss_f(output,y)

    loss.backward()

    print(f"weights gradient: {model[0].weight.grad}")

    print(f"bias gradient: {model[0].bias.grad}")

    weights gradient: tensor([[-0.0200, -0.0200, -0.0200, -0.0200, -0.0200]])

    bias gradient: tensor([-0.2000])

    Exercise 11:

    model_relu = torch.nn.Sequential(

    torch.nn.Linear(5,1),

    torch.nn.ReLU()

    )

    output_relu=model_relu(x)

    print(f"relu output: {output_relu}")

    relu output: tensor([0.0100], grad_fn=)

    Exercise 12:

    loss_relu=loss_f(output_relu,y)

    loss_relu.backward()

    print(f"weights gradient with ReLU: {model_relu[0].weight.grad}")

    print(f"bias gradient with ReLU: {model_relu[0].bias.grad}")

    weights gradient with ReLU: tensor([[0., 0., 0., 0., 0.]])

    bias gradient with ReLU: tensor([0.])

    The gradients for weights and bias are equal to zero, because ReLU function does not have any gradient for negative values.

    Exercise 13:

    To be able to draw 3D decision boundary, we need at least 3 input features. In example 5.2 we only have 2 input features, so it is not possible to draw a 3D figure with decision boundary.

  3. User

    Rozwiąz zadania Example 4.2 Simple neuron example 𝑧=𝑤⋅𝑥+𝑏 , target 𝑦

    x= torch.ones(5)*0.1

    #x.requires_grad=True

    y= torch.tensor([0.])

    w=torch.tensor([-2.,3,-1,-1,-1],requires_grad=True)

    b=torch.tensor([0.1],requires_grad=True)

    z=torch.matmul(x,w)+b

    loss_f=torch.nn.MSELoss(reduction='sum')

    print('loss: ',loss_f(z,y))

    loss=loss_f(z,y)

    print(f"Gradient function for z = {z.grad_fn}")

    print(f"Gradient function for loss = {loss.grad_fn}")

    loss.backward()

    print(x.grad)

    print(w.grad)

    print(b.grad)

    loss: tensor(0.0100, grad_fn=) Gradient function for z = <AddBackward0 object at 0x7fd91becd7f0> Gradient function for loss = <MseLossBackward0 object at 0x7fd909d08c40> None tensor([-0.0200, -0.0200, -0.0200, -0.0200, -0.0200]) tensor([-0.2000])

    1. Neural network in pytorch

    Example 5.1 Create simple "network" containing one neuron (as above)

    torch.nn.Sequential - allows you to create network build by layers (ie. neurons, then activation function)
    
    torch.nn.Linear(n_inputs,n_outputs) - layer made by n neurons, where n=n_ouputs, n_ouptus - number of outputs and neurons, n_inputs -number of inputs (the same inputs for each neuron)
    

    Note: writing import torch.nn you can refer to nn.Linear , but by import torch.nn.Linear is enought to write Linear

    model = torch.nn.Sequential(

    torch.nn.Linear(5,1)

    )

    print(f"one neuron: {model}\n")

    print(f"weights: {model[0].weight}\n")

    print(f"bias: {model[0].bias}")

    one neuron: Sequential( (0): Linear(in_features=5, out_features=1, bias=True) )

    weights: Parameter containing: tensor([[ 0.3702, 0.2262, -0.0688, -0.3097, 0.1560]], requires_grad=True)

    bias: Parameter containing: tensor([0.2976], requires_grad=True)

    Set the weights and bias (note, that weights are 2D, bias 1D)

    model[0].weight.data=torch.tensor([[-2.,3.,-1,-1,-1]],requires_grad=True)

    model[0].bias.data=torch.tensor([0.1],requires_grad=True)

    Set input x and expected output (target) y, the output of network output

    x= torch.ones(5)*0.1

    y= torch.tensor([0.])

    output=model(x)

    Exercise 9. Calculate the derivatives for weights and bias from example 4.3 similar to example 4.2. Access here to weights model[0].weight.data.grad

    ​

    Exercise 10 Confirm derivatives from ex. 9 by manual calculations.

    ​

    Exercise 11 Add to neuron from example 4.2 or 4.3 ReLu activation function (hint: in example 4.2 use torch.nn.functional.relu(x), which support autograd, in example 4.3 -similar to 4.4 use torch.nn.ReLU()

    ​

    Exercise 12 Confirm derivatives from ex. 11 by manual calculations.

    ​

    Exercise 13 After running example 5.2 use code for drawing 3d figure with decission boundary. Is it the same as for neuron in chapter 3?

  4. Assistant

    Exercise 9:

    loss=loss_f(output,y)

    loss.backward()

    print(f"weights gradient: {model[0].weight.data.grad}")

    print(f"bias gradient: {model[0].bias.data.grad}")

    weights gradient: tensor([[-0.0200, -0.0200, -0.0200, -0.0200, -0.0200]])

    bias gradient: tensor([-0.2000])

    Exercise 10:

    We can calculate the derivatives manually:

    dL/dw = (dL/dz) * (dz/dw) = 2*(z-y)*x

    dL/db = (dL/dz) * (dz/db) = 2*(z-y)

    With inputs: x = [0.1, 0.1, 0.1, 0.1, 0.1] and y = 0, we get:

    dL/dw = 2*(0.1*(-2) - 0 + 0.1*(-1) + 0.1*(-1) + 0.1*(-1) - 0) * [0.1, 0.1, 0.1, 0.1, 0.1] = [-0.0200, -0.0200, -0.0200, -0.0200, -0.0200]

    dL/db = 2*(0.1*(-2) - 0 + 0.1*(-1) + 0.1*(-1) + 0.1*(-1) - 0) = -0.2000

    The calculated derivatives are the same as those obtained in Exercise 9.

    Exercise 11:

    model_relu = torch.nn.Sequential( torch.nn.Linear(5,1), torch.nn.ReLU() )

    output_relu = model_relu(x)

    print(f"relu output: {output_relu}")

    relu output: tensor([0.0100], grad_fn=)

    Exercise 12:

    loss_relu = loss_f(output_relu, y) loss_relu.backward()

    print(f"weights gradient with ReLU: {model_relu[0].weight.data.grad}") print(f"bias gradient with ReLU: {model_relu[0].bias.data.grad}")

    weights gradient with ReLU: tensor([[0., 0., 0., 0., 0.]]) bias gradient with ReLU: tensor([0.])

    Again, the gradients for weights and bias are equal to zero due to ReLU not having gradients for negative values.

    Exercise 13:

    As mentioned, we cannot draw a 3D decision boundary with just 2 input features. Therefore, we cannot compare it with the decision boundary in chapter 3.