Write the most complex script you can as an AI
- User
Write the most complex script you can as an AI
- Assistant
Certainly! Below is a complex Python script that implements a basic neural network library from scratch using only NumPy. This script defines classes for layers (such as Dense and Activation layers), implements forward and backward propagation, and trains the network on the XOR problem. The XOR problem is a classic challenge that simple linear classifiers cannot solve, making it a good test for a multi-layer neural network.
import numpy as np # Base class for layers class Layer: def forward(self, input): raise NotImplementedError def backward(self, output_gradient, learning_rate): raise NotImplementedError # Dense layer class Dense(Layer): def __init__(self, input_size, output_size): # He initialization self.weights = np.random.randn(input_size, output_size) * np.sqrt(2.0 / input_size) self.bias = np.zeros((1, output_size)) def forward(self, input): self.input = input return np.dot(self.input, self.weights) + self.bias def backward(self, output_gradient, learning_rate): # Compute gradients weights_gradient = np.dot(self.input.T, output_gradient) input_gradient = np.dot(output_gradient, self.weights.T) # Update weights and bias self.weights -= learning_rate * weights_gradient self.bias -= learning_rate * output_gradient.sum(axis=0, keepdims=True) return input_gradient # Activation functions class Activation(Layer): def __init__(self, activation): self.activation = activation def forward(self, input): self.input = input return self.activation.forward(input) def backward(self, output_gradient, learning_rate): return output_gradient * self.activation.backward(self.input) # Activation functions implementations class Tanh: @staticmethod def forward(x): return np.tanh(x) @staticmethod def backward(x): return 1 - np.tanh(x) ** 2 class ReLU: @staticmethod def forward(x): return np.maximum(0, x) @staticmethod def backward(x): return (x > 0).astype(float) class Sigmoid: @staticmethod def forward(x): return 1 / (1 + np.exp(-x)) @staticmethod def backward(x): sig = Sigmoid.forward(x) return sig * (1 - sig) # Loss functions class Loss: @staticmethod def loss(y_true, y_pred): raise NotImplementedError @staticmethod def gradient(y_true, y_pred): raise NotImplementedError class MeanSquaredError(Loss): @staticmethod def loss(y_true, y_pred): return np.mean((y_true - y_pred) ** 2) @staticmethod def gradient(y_true, y_pred): return 2 * (y_pred - y_true) / y_true.size # Neural Network class class NeuralNetwork: def __init__(self): self.layers = [] self.loss = None self.loss_prime = None def add(self, layer): self.layers.append(layer) def use(self, loss): self.loss = loss.loss self.loss_prime = loss.gradient def predict(self, input_data): # Sample dimensions samples = len(input_data) result = [] # Run network over all samples for i in range(samples): # Forward propagation output = input_data[i] for layer in self.layers: output = layer.forward(output) result.append(output) return result def fit(self, x_train, y_train, epochs, learning_rate): # Sample dimensions samples = len(x_train) # Training loop for i in range(epochs): err = 0 for j in range(samples): # Forward propagation output = x_train[j] for layer in self.layers: output = layer.forward(output) # Compute loss err += self.loss(y_train[j], output) # Backward propagation gradient = self.loss_prime(y_train[j], output) for layer in reversed(self.layers): gradient = layer.backward(gradient, learning_rate) # Average error on all samples err /= samples print(f'Epoch {i+1}/{epochs}, Error: {err}') # XOR problem dataset x_train = np.array([ [[0, 0]], [[0, 1]], [[1, 0]], [[1, 1]] ]) y_train = np.array([ [[0]], [[1]], [[1]], [[0]] ]) # Create the neural network net = NeuralNetwork() net.add(Dense(2, 3)) net.add(Activation(Tanh())) net.add(Dense(3, 1)) net.add(Activation(Sigmoid())) # Set loss function net.use(MeanSquaredError()) # Train the network net.fit(x_train, y_train, epochs=1000, learning_rate=0.1) # Test the network out = net.predict(x_train) print("\nPredictions:") for i, prediction in enumerate(out): print(f"Input: {x_train[i][0]}, Predicted Output: {prediction[0]}, True Output: {y_train[i][0]}")Explanation:
-
Layer Classes: The script defines a base
Layerclass and implementsDense(fully connected) layers andActivationlayers. -
Activation Functions: Implements
Tanh,ReLU, andSigmoidactivation functions with their forward and backward methods for computing gradients. -
Loss Function: Implements the mean squared error loss and its derivative for gradient computation.
-
Neural Network Class: The
NeuralNetworkclass manages layers, loss computation, forward and backward propagation, prediction, and training. -
Dataset: Uses the XOR dataset, where the output is
1if the inputs are different and0if they are the same. -
Training: Trains the network for 1000 epochs with a learning rate of 0.1.
-
Prediction: After training, it prints out the predictions of the network compared to the true outputs.
Key Concepts Demonstrated:
-
Object-Oriented Programming: Uses classes and inheritance to structure the neural network components.
-
Forward and Backward Propagation: Manually implements the forward pass and backpropagation algorithm.
-
Activation Functions: Shows how different activation functions can be implemented and used within the network.
-
Loss Computation: Demonstrates how to compute the loss and its gradient for training.
-
Training Loop: Implements the training loop, including gradient descent optimization.
Output:
Upon running the script, the network will train on the XOR dataset, and you will see the error decreasing over epochs. After training, it will output the predicted values for each input in the XOR dataset.
Note: This script is a simplified example of building a neural network from scratch for educational purposes. In practice, using established libraries like TensorFlow or PyTorch is recommended for building neural networks, as they are optimized and offer many advanced features.
-