Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

create a text neural network ( basically an AI model ) in rust without using tch put…

  1. User

    create a text neural network ( basically an AI model ) in rust without using tch put everything in main.rs

    make sure its a thorough implementation.

  2. Assistant

    Certainly! Below is a thorough implementation of a simple text classification neural network in Rust without using the tch crate. The code will be contained entirely within main.rs. This implementation includes basic tokenization, a simple vocabulary, an embedding layer, and a single-layer neural network trained using stochastic gradient descent.

    // main.rs
    
    use std::collections::HashMap;
    use rand::prelude::*;
    
    fn main() {
        // Sample dataset: Sentences labeled as positive (1) or negative (0)
        let sentences = vec![
            ("I love this movie", 1),
            ("This film was terrible", 0),
            ("What an amazing experience", 1),
            ("I do not like this film", 0),
            ("Fantastic acting and great storyline", 1),
            ("The plot was dull and boring", 0),
        ];
    
        // Build the vocabulary from the dataset
        let vocab = build_vocab(&sentences);
        let vocab_size = vocab.len();
    
        // Convert sentences to sequences of token IDs
        let data: Vec<(Vec<usize>, usize)> = sentences
            .iter()
            .map(|(sentence, label)| {
                let token_ids = tokenize(sentence)
                    .iter()
                    .map(|token| *vocab.get(token).unwrap())
                    .collect();
                (token_ids, *label)
            })
            .collect();
    
        // Initialize the neural network
        let embedding_dim = 8;
        let nn = NeuralNetwork::new(vocab_size, embedding_dim);
    
        // Train the neural network
        let epochs = 1000;
        let learning_rate = 0.1;
        let trained_nn = train(nn, &data, epochs, learning_rate);
    
        // Test the trained model on new sentences
        let test_sentences = vec![
            ("I really enjoyed this", 1),
            ("This was a waste of time", 0),
        ];
    
        for (sentence, label) in test_sentences {
            let token_ids: Vec<usize> = tokenize(sentence)
                .iter()
                .map(|token| *vocab.get(token).unwrap_or(&0)) // Use 0 if the word is unseen
                .collect();
            let prediction = trained_nn.predict(&token_ids);
            println!(
                "Sentence: '{}', Actual Label: {}, Predicted Label: {}",
                sentence,
                label,
                if prediction > 0.5 { 1 } else { 0 }
            );
        }
    }
    
    // Tokenize a sentence into words
    fn tokenize(sentence: &str) -> Vec<String> {
        sentence
            .to_lowercase()
            .split_whitespace()
            .map(|s| s.to_string())
            .collect()
    }
    
    // Build a vocabulary mapping from words to unique IDs
    fn build_vocab(sentences: &Vec<(&str, usize)>) -> HashMap<String, usize> {
        let mut vocab = HashMap::new();
        let mut index = 1; // Start indexing from 1
    
        for (sentence, _) in sentences {
            for token in tokenize(sentence) {
                if !vocab.contains_key(&token) {
                    vocab.insert(token, index);
                    index += 1;
                }
            }
        }
    
        vocab
    }
    
    // Neural Network struct
    struct NeuralNetwork {
        vocab_size: usize,
        embedding_dim: usize,
        embeddings: Vec<Vec<f32>>,
        weights: Vec<f32>,
        bias: f32,
    }
    
    impl NeuralNetwork {
        // Initialize the neural network with random weights
        fn new(vocab_size: usize, embedding_dim: usize) -> Self {
            let mut rng = thread_rng();
    
            let embeddings = (0..vocab_size + 1)
                .map(|_| {
                    (0..embedding_dim)
                        .map(|_| rng.gen_range(-0.5..0.5))
                        .collect()
                })
                .collect();
    
            let weights: Vec<f32> = (0..embedding_dim).map(|_| rng.gen_range(-0.5..0.5)).collect();
            let bias = rng.gen_range(-0.5..0.5);
    
            NeuralNetwork {
                vocab_size,
                embedding_dim,
                embeddings,
                weights,
                bias,
            }
        }
    
        // Forward pass: Compute the output probability for a given input
        fn forward(&self, inputs: &Vec<usize>) -> f32 {
            // Get embeddings for each token ID
            let mut embedded = vec![0.0; self.embedding_dim];
            for &token_id in inputs {
                let token_embedding = &self.embeddings[token_id];
                for i in 0..self.embedding_dim {
                    embedded[i] += token_embedding[i];
                }
            }
    
            // Average the embeddings
            let len = inputs.len() as f32;
            for i in 0..self.embedding_dim {
                embedded[i] /= len;
            }
    
            // Compute the dot product with weights
            let mut output = 0.0;
            for i in 0..self.embedding_dim {
                output += embedded[i] * self.weights[i];
            }
    
            output += self.bias;
    
            // Apply sigmoid activation
            sigmoid(output)
        }
    
        // Backward pass: Update weights and embeddings based on the error
        fn backward(&mut self, inputs: &Vec<usize>, target: f32, lr: f32) {
            // Forward pass to get the prediction
            let pred = self.forward(inputs);
    
            // Compute the error derivative
            let error = pred - target;
    
            // Get embeddings for each token ID
            let mut embedded = vec![0.0; self.embedding_dim];
            for &token_id in inputs {
                let token_embedding = &self.embeddings[token_id];
                for i in 0..self.embedding_dim {
                    embedded[i] += token_embedding[i];
                }
            }
    
            // Average the embeddings
            let len = inputs.len() as f32;
            for i in 0..self.embedding_dim {
                embedded[i] /= len;
            }
    
            // Update weights and bias
            for i in 0..self.embedding_dim {
                self.weights[i] -= lr * error * embedded[i];
            }
            self.bias -= lr * error;
    
            // Update embeddings
            for &token_id in inputs {
                let token_embedding = &mut self.embeddings[token_id];
                for i in 0..self.embedding_dim {
                    let gradient = self.weights[i] * error / len;
                    token_embedding[i] -= lr * gradient;
                }
            }
        }
    
        // Predict the probability for a given input
        fn predict(&self, inputs: &Vec<usize>) -> f32 {
            self.forward(inputs)
        }
    }
    
    // Train the neural network using the dataset
    fn train(
        mut nn: NeuralNetwork,
        data: &Vec<(Vec<usize>, usize)>,
        epochs: usize,
        learning_rate: f32,
    ) -> NeuralNetwork {
        for epoch in 0..epochs {
            let mut total_loss = 0.0;
            for (inputs, label) in data {
                let target = *label as f32;
                let pred = nn.forward(inputs);
                let loss = binary_cross_entropy(pred, target);
                total_loss += loss;
                nn.backward(inputs, target, learning_rate);
            }
    
            if epoch % 100 == 0 {
                println!("Epoch {}, Loss: {:.4}", epoch, total_loss / data.len() as f32);
            }
        }
    
        nn
    }
    
    // Sigmoid activation function
    fn sigmoid(x: f32) -> f32 {
        1.0 / (1.0 + (-x).exp())
    }
    
    // Binary cross-entropy loss function
    fn binary_cross_entropy(pred: f32, target: f32) -> f32 {
        -(target * pred.ln() + (1.0 - target) * (1.0 - pred).ln())
    }
    

    Explanation

    Data Preparation

    • Tokenization: The tokenize function splits sentences into lowercase words.
    • Vocabulary Building: The build_vocab function creates a mapping from words to unique integer IDs starting from 1 (we reserve 0 for unknown words).
    • Data Encoding: Sentences are converted into sequences of token IDs using the vocabulary.

    Neural Network Structure

    • Embedding Layer: Maps each word ID to a vector of real numbers (embeddings). This helps in representing words in a continuous vector space.
    • Average Pooling: Averages the embeddings of all words in the sentence to create a fixed-size input vector for the neural network.
    • Output Layer: A single neuron (weights and bias) computes a weighted sum of the averaged embeddings and applies the sigmoid activation function to output a probability.

    Training Process

    • Forward Pass: The forward method computes the output probability for a given input.
    • Backward Pass: The backward method updates the weights, bias, and embeddings based on the error derivative computed from the loss function.
    • Loss Function: Binary cross-entropy loss measures the difference between the predicted probability and the actual label.
    • Optimization: Stochastic gradient descent updates the model parameters to minimize the loss.

    Testing

    • The trained model is tested on new sentences to predict their sentiment.
    • Words not seen during training default to an embedding index of 0.

    Notes

    • Random Initialization: We use random values to initialize weights and embeddings to break symmetry and allow the network to learn.
    • Learning Rate: A hyperparameter that needs tuning; too high may cause divergence, too low may slow down learning.
    • Epochs: Number of times the training process iterates over the entire dataset.

    Dependencies

    • std::collections::HashMap: For creating the vocabulary mapping.
    • rand crate: For random number generation (rand = "0.8.5" in Cargo.toml).

    To run this code, ensure you have the rand crate included in your Cargo.toml:

    [dependencies]
    rand = "0.8.5"
    

    Conclusion

    This implementation demonstrates how you can build a simple text neural network from scratch in Rust without relying on external machine learning crates like tch. While this model is simplistic and meant for educational purposes, it provides a foundation on which more complex models (e.g., incorporating more layers, different architectures, or advanced optimization techniques) can be built.


    Feel free to ask if you need further explanations or enhancements to the code!