Skip to content
TheChatArchiveThe Chat Archive
GPT 5.1· openaiHugging Face Frontier Reasoning Traces

Write Python code to implement a neural network from scratch without libraries.

  1. User

    Write Python code to implement a neural network from scratch without libraries.

  2. Assistant

    Clarifying neural network request

    The user wants Python code to create a neural network from scratch without using any libraries. That likely means no machine learning libraries like TensorFlow or PyTorch, but I wonder if I can use numpy. Requests like this often imply using pure Python, which could exclude numpy altogether. Since they haven't specified details like the type of tasks or dataset, I think they might be looking for simple educational code for a fully connected network, focusing on standard library usage and basic operations.Defining neural network constraints

    The user mentioned implementing a neural network from scratch without any libraries, which likely means avoiding even common libraries like numpy, but I think utilities from the standard library, like random and math, should be fine. I need to create a self-contained script featuring utility functions for operations like the dot product, activation functions, and cross-entropy loss. I have to keep explanations minimal and focus mainly on providing the code, possibly with brief comments, while ensuring it handles something simple like the XOR dataset for demonstration.Creating the neural network class

    I think it would be useful to show how the training works, so I’ll create a class called NeuralNetwork. In the __init__ method, I'll randomly create weights and biases based on the provided layer sizes. The forward method will handle classification, using the sigmoid activation for hidden layers and softmax for the output layer. For the backward method, I'll compute gradients using cross-entropy and softmax, followed by updating weights with stochastic gradient descent. I’ll represent weight matrices as lists of lists, run it on single samples, and then code the backpropagation using these calculations.Implementing the backpropagation step

    I’m now focusing on the backpropagation for each layer, starting from the last layer back to the first. For each layer ( L ), I’ll calculate the weight gradients and the biases using the delta values. I need to compute the deltas for the hidden layers as well and store the intermediate values during the forward pass. I might combine the forward and backward processes into a single train_step function. I’ll also need to set up the neural network class with arbitrary layer sizes and a learning rate, randomly initializing weights and biases while keeping the structure clear and systematic.Implementing forward and training methods

    I’m implementing the forward method in my neural network class. This function will calculate the activations and intermediate values for each layer based on input ( x ). I’ll also create a train method that handles multiple epochs, processes data, and updates weights while calculating loss. For each training example, I’ll use a private method _train_single to perform forward propagation, computing losses and gradients in the backward pass, ensuring I handle the shapes and indices correctly throughout. I want to keep it clear and functional.Defining layer indexing and training logic

    I’m setting up a 1-based layer indexing system for the neural network. Layer 0 is the input, layer 1 is the first hidden layer, up to the output layer, which is layer L. I need to ensure that weight indices align with the corresponding layers for forward and backward propagation. For the training method, I must account for cases where there may not be any hidden layers, handling special conditions gracefully. I’m also planning a small script for an XOR training example to demonstrate the neural network's functionality.Writing a simple neural network in Python

    I'm going to add comments at the top of the code to explain that it's a simple fully-connected neural network built from scratch without using external libraries. I need to ensure that built-in functions from math and random are fine, as they’re not considered external libraries. I also want to implement essential methods like softmax, sigmoid, and cross-entropy, ensuring that they handle potential floating-point issues. The training method should track total loss and optionally print progress to keep it user-friendly. Lastly, I'll prepare a clear introduction to the code for users. Here’s a minimal fully-connected neural network (feedforward MLP) implemented from scratch using only the Python standard library (no NumPy, no ML libraries). It trains on the XOR problem as an example.

    #!/usr/bin/env python3
    """
    Simple neural network (MLP) from scratch using only Python standard library.
    - Fully-connected layers
    - Sigmoid activations for hidden layers
    - Softmax + cross-entropy for output
    - Trained with stochastic gradient descent (one sample at a time)
    """
    
    import random
    import math
    
    
    class NeuralNetwork:
        def __init__(self, layer_sizes, learning_rate=0.1):
            """
            layer_sizes: list of ints, e.g. [input_dim, hidden1, hidden2, ..., output_dim]
            learning_rate: SGD step size
            """
            self.layer_sizes = layer_sizes
            self.num_layers = len(layer_sizes) - 1  # number of weight layers
            self.learning_rate = learning_rate
    
            # Initialize weights and biases
            self.weights = []  # list of matrices [out_size x in_size]
            self.biases = []   # list of vectors [out_size]
    
            for i in range(self.num_layers):
                in_size = layer_sizes[i]
                out_size = layer_sizes[i + 1]
                # Small random initialization
                W = [[random.uniform(-0.5, 0.5) * 0.1 for _ in range(in_size)]
                     for _ in range(out_size)]
                b = [0.0 for _ in range(out_size)]
                self.weights.append(W)
                self.biases.append(b)
    
        # ---------- Activation functions ----------
    
        @staticmethod
        def _sigmoid(x):
            # Numerically stable-ish sigmoid
            if x >= 0:
                z = math.exp(-x)
                return 1.0 / (1.0 + z)
            else:
                z = math.exp(x)
                return z / (1.0 + z)
    
        @staticmethod
        def _sigmoid_derivative(x):
            s = NeuralNetwork._sigmoid(x)
            return s * (1.0 - s)
    
        @staticmethod
        def _softmax(z):
            max_z = max(z)
            exps = [math.exp(v - max_z) for v in z]
            sum_exps = sum(exps)
            return [e / sum_exps for e in exps]
    
        @staticmethod
        def _cross_entropy(pred, target):
            """
            pred: list of probabilities (softmax output)
            target: one-hot vector or probability distribution
            """
            eps = 1e-15
            loss = 0.0
            for p, t in zip(pred, target):
                p_clamped = max(min(p, 1.0 - eps), eps)
                loss -= t * math.log(p_clamped)
            return loss
    
        # ---------- Forward & backward passes ----------
    
        def forward(self, x):
            """
            Forward pass through the network.
            x: input vector (list of floats)
            Returns:
                activations: [a^0, a^1, ..., a^L]
                zs: [z^1, z^2, ..., z^L]
            """
            a = x[:]  # copy
            activations = [a]
            zs = []
    
            for layer in range(self.num_layers):
                W = self.weights[layer]
                b = self.biases[layer]
    
                # Compute z = W * a + b
                z = []
                for i in range(len(W)):  # each output neuron
                    s = 0.0
                    for j in range(len(W[0])):  # each input neuron
                        s += W[i][j] * a[j]
                    z.append(s + b[i])
    
                zs.append(z)
    
                # Apply activation
                if layer == self.num_layers - 1:
                    # Output layer: softmax
                    a = self._softmax(z)
                else:
                    # Hidden layers: sigmoid
                    a = [self._sigmoid(v) for v in z]
    
                activations.append(a)
    
            return activations, zs
    
        def predict(self, x):
            """
            Predict output probabilities for a single input.
            """
            activations, _ = self.forward(x)
            return activations[-1]
    
        def predict_class(self, x):
            """
            Return index of the highest-probability output neuron.
            """
            probs = self.predict(x)
            return max(range(len(probs)), key=lambda i: probs[i])
    
        def _train_single(self, x, y):
            """
            Train on a single (x, y) pair using SGD.
            x: input vector
            y: target one-hot vector
            Returns: loss (float)
            """
            # Forward pass
            activations, zs = self.forward(x)
            output = activations[-1]
    
            # Compute loss
            loss = self._cross_entropy(output, y)
    
            # Backward pass (backpropagation)
            # Gradient of loss w.r.t. z^L for softmax + cross-entropy:
            # delta^L = (a^L - y)
            delta = [output[i] - y[i] for i in range(len(y))]
    
            nabla_b = [None] * self.num_layers
            nabla_w = [None] * self.num_layers
    
            # Gradients for last layer
            nabla_b[-1] = delta
            a_prev = activations[-2]  # a^(L-1)
    
            W_grad_last = []
            for i in range(len(delta)):  # output neurons
                row = []
                for j in range(len(a_prev)):  # inputs to this layer
                    row.append(delta[i] * a_prev[j])
                W_grad_last.append(row)
            nabla_w[-1] = W_grad_last
    
            # Backpropagate through hidden layers
            for l in range(self.num_layers - 2, -1, -1):
                z = zs[l]  # z^(l+1)
                sp = [self._sigmoid_derivative(v) for v in z]
    
                # delta_{l} = (W_{l+1}^T * delta_{l+1}) ⊙ sigmoid'(z^{l+1})
                W_next = self.weights[l + 1]
                delta_next = delta
    
                delta_new = []
                # For each neuron j in layer l+1
                for j in range(len(W_next[0])):  # number of neurons in layer l+1
                    s = 0.0
                    # Sum over neurons i in layer l+2
                    for i in range(len(W_next)):
                        s += W_next[i][j] * delta_next[i]
                    delta_new.append(s * sp[j])
    
                delta = delta_new
                nabla_b[l] = delta
    
                a_prev = activations[l]  # a^l
                W_grad = []
                for i in range(len(delta)):  # neurons in layer l+1
                    row = []
                    for j in range(len(a_prev)):  # neurons in layer l
                        row.append(delta[i] * a_prev[j])
                    W_grad.append(row)
                nabla_w[l] = W_grad
    
            # Gradient descent update
            eta = self.learning_rate
            for l in range(self.num_layers):
                # Update weights
                for i in range(len(self.weights[l])):
                    for j in range(len(self.weights[l][0])):
                        self.weights[l][i][j] -= eta * nabla_w[l][i][j]
                # Update biases
                for i in range(len(self.biases[l])):
                    self.biases[l][i] -= eta * nabla_b[l][i]
    
            return loss
    
        def train(self, X, Y, epochs=1000, verbose=True):
            """
            Train the network.
            X: list of input vectors
            Y: list of target one-hot vectors
            """
            n = len(X)
            for epoch in range(epochs):
                total_loss = 0.0
                # Simple SGD over all samples
                for x, y in zip(X, Y):
                    total_loss += self._train_single(x, y)
    
                avg_loss = total_loss / n
                if verbose and (epoch % max(1, epochs // 10) == 0 or epoch == epochs - 1):
                    print(f"Epoch {epoch+1}/{epochs} - avg loss: {avg_loss:.4f}")
    
    
    # -------------------- Example usage (XOR problem) --------------------
    
    if __name__ == "__main__":
        # XOR dataset
        X = [
            [0.0, 0.0],
            [0.0, 1.0],
            [1.0, 0.0],
            [1.0, 1.0],
        ]
    
        # Targets as one-hot vectors: class 0 for output 0, class 1 for output 1
        Y = [
            [1.0, 0.0],  # 0 XOR 0 = 0
            [0.0, 1.0],  # 0 XOR 1 = 1
            [0.0, 1.0],  # 1 XOR 0 = 1
            [1.0, 0.0],  # 1 XOR 1 = 0
        ]
    
        # 2 inputs -> 2 hidden neurons -> 2 outputs
        nn = NeuralNetwork(layer_sizes=[2, 2, 2], learning_rate=0.5)
    
        nn.train(X, Y, epochs=5000, verbose=True)
    
        # Test predictions
        print("\nPredictions after training:")
        for x, y in zip(X, Y):
            probs = nn.predict(x)
            predicted_class = nn.predict_class(x)
            target_class = y.index(1.0)
            print(f"Input: {x} | "
                  f"Predicted probs: {[round(p, 3) for p in probs]} | "
                  f"Predicted class: {predicted_class}, Target: {target_class}")
    

    You can adapt layer_sizes, activation functions, and training loop logic to build deeper or different networks.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.