create a text neural network ( basically an AI model ) in rust without using tch put…
- User
create a text neural network ( basically an AI model ) in rust without using tch put everything in main.rs
make sure its a thorough implementation.
- Assistant
Certainly! Below is a thorough implementation of a simple text classification neural network in Rust without using the
tchcrate. The code will be contained entirely withinmain.rs. This implementation includes basic tokenization, a simple vocabulary, an embedding layer, and a single-layer neural network trained using stochastic gradient descent.// main.rs use std::collections::HashMap; use rand::prelude::*; fn main() { // Sample dataset: Sentences labeled as positive (1) or negative (0) let sentences = vec![ ("I love this movie", 1), ("This film was terrible", 0), ("What an amazing experience", 1), ("I do not like this film", 0), ("Fantastic acting and great storyline", 1), ("The plot was dull and boring", 0), ]; // Build the vocabulary from the dataset let vocab = build_vocab(&sentences); let vocab_size = vocab.len(); // Convert sentences to sequences of token IDs let data: Vec<(Vec<usize>, usize)> = sentences .iter() .map(|(sentence, label)| { let token_ids = tokenize(sentence) .iter() .map(|token| *vocab.get(token).unwrap()) .collect(); (token_ids, *label) }) .collect(); // Initialize the neural network let embedding_dim = 8; let nn = NeuralNetwork::new(vocab_size, embedding_dim); // Train the neural network let epochs = 1000; let learning_rate = 0.1; let trained_nn = train(nn, &data, epochs, learning_rate); // Test the trained model on new sentences let test_sentences = vec![ ("I really enjoyed this", 1), ("This was a waste of time", 0), ]; for (sentence, label) in test_sentences { let token_ids: Vec<usize> = tokenize(sentence) .iter() .map(|token| *vocab.get(token).unwrap_or(&0)) // Use 0 if the word is unseen .collect(); let prediction = trained_nn.predict(&token_ids); println!( "Sentence: '{}', Actual Label: {}, Predicted Label: {}", sentence, label, if prediction > 0.5 { 1 } else { 0 } ); } } // Tokenize a sentence into words fn tokenize(sentence: &str) -> Vec<String> { sentence .to_lowercase() .split_whitespace() .map(|s| s.to_string()) .collect() } // Build a vocabulary mapping from words to unique IDs fn build_vocab(sentences: &Vec<(&str, usize)>) -> HashMap<String, usize> { let mut vocab = HashMap::new(); let mut index = 1; // Start indexing from 1 for (sentence, _) in sentences { for token in tokenize(sentence) { if !vocab.contains_key(&token) { vocab.insert(token, index); index += 1; } } } vocab } // Neural Network struct struct NeuralNetwork { vocab_size: usize, embedding_dim: usize, embeddings: Vec<Vec<f32>>, weights: Vec<f32>, bias: f32, } impl NeuralNetwork { // Initialize the neural network with random weights fn new(vocab_size: usize, embedding_dim: usize) -> Self { let mut rng = thread_rng(); let embeddings = (0..vocab_size + 1) .map(|_| { (0..embedding_dim) .map(|_| rng.gen_range(-0.5..0.5)) .collect() }) .collect(); let weights: Vec<f32> = (0..embedding_dim).map(|_| rng.gen_range(-0.5..0.5)).collect(); let bias = rng.gen_range(-0.5..0.5); NeuralNetwork { vocab_size, embedding_dim, embeddings, weights, bias, } } // Forward pass: Compute the output probability for a given input fn forward(&self, inputs: &Vec<usize>) -> f32 { // Get embeddings for each token ID let mut embedded = vec![0.0; self.embedding_dim]; for &token_id in inputs { let token_embedding = &self.embeddings[token_id]; for i in 0..self.embedding_dim { embedded[i] += token_embedding[i]; } } // Average the embeddings let len = inputs.len() as f32; for i in 0..self.embedding_dim { embedded[i] /= len; } // Compute the dot product with weights let mut output = 0.0; for i in 0..self.embedding_dim { output += embedded[i] * self.weights[i]; } output += self.bias; // Apply sigmoid activation sigmoid(output) } // Backward pass: Update weights and embeddings based on the error fn backward(&mut self, inputs: &Vec<usize>, target: f32, lr: f32) { // Forward pass to get the prediction let pred = self.forward(inputs); // Compute the error derivative let error = pred - target; // Get embeddings for each token ID let mut embedded = vec![0.0; self.embedding_dim]; for &token_id in inputs { let token_embedding = &self.embeddings[token_id]; for i in 0..self.embedding_dim { embedded[i] += token_embedding[i]; } } // Average the embeddings let len = inputs.len() as f32; for i in 0..self.embedding_dim { embedded[i] /= len; } // Update weights and bias for i in 0..self.embedding_dim { self.weights[i] -= lr * error * embedded[i]; } self.bias -= lr * error; // Update embeddings for &token_id in inputs { let token_embedding = &mut self.embeddings[token_id]; for i in 0..self.embedding_dim { let gradient = self.weights[i] * error / len; token_embedding[i] -= lr * gradient; } } } // Predict the probability for a given input fn predict(&self, inputs: &Vec<usize>) -> f32 { self.forward(inputs) } } // Train the neural network using the dataset fn train( mut nn: NeuralNetwork, data: &Vec<(Vec<usize>, usize)>, epochs: usize, learning_rate: f32, ) -> NeuralNetwork { for epoch in 0..epochs { let mut total_loss = 0.0; for (inputs, label) in data { let target = *label as f32; let pred = nn.forward(inputs); let loss = binary_cross_entropy(pred, target); total_loss += loss; nn.backward(inputs, target, learning_rate); } if epoch % 100 == 0 { println!("Epoch {}, Loss: {:.4}", epoch, total_loss / data.len() as f32); } } nn } // Sigmoid activation function fn sigmoid(x: f32) -> f32 { 1.0 / (1.0 + (-x).exp()) } // Binary cross-entropy loss function fn binary_cross_entropy(pred: f32, target: f32) -> f32 { -(target * pred.ln() + (1.0 - target) * (1.0 - pred).ln()) }Explanation
Data Preparation
- Tokenization: The
tokenizefunction splits sentences into lowercase words. - Vocabulary Building: The
build_vocabfunction creates a mapping from words to unique integer IDs starting from 1 (we reserve 0 for unknown words). - Data Encoding: Sentences are converted into sequences of token IDs using the vocabulary.
Neural Network Structure
- Embedding Layer: Maps each word ID to a vector of real numbers (
embeddings). This helps in representing words in a continuous vector space. - Average Pooling: Averages the embeddings of all words in the sentence to create a fixed-size input vector for the neural network.
- Output Layer: A single neuron (
weightsandbias) computes a weighted sum of the averaged embeddings and applies the sigmoid activation function to output a probability.
Training Process
- Forward Pass: The
forwardmethod computes the output probability for a given input. - Backward Pass: The
backwardmethod updates the weights, bias, and embeddings based on the error derivative computed from the loss function. - Loss Function: Binary cross-entropy loss measures the difference between the predicted probability and the actual label.
- Optimization: Stochastic gradient descent updates the model parameters to minimize the loss.
Testing
- The trained model is tested on new sentences to predict their sentiment.
- Words not seen during training default to an embedding index of 0.
Notes
- Random Initialization: We use random values to initialize weights and embeddings to break symmetry and allow the network to learn.
- Learning Rate: A hyperparameter that needs tuning; too high may cause divergence, too low may slow down learning.
- Epochs: Number of times the training process iterates over the entire dataset.
Dependencies
std::collections::HashMap: For creating the vocabulary mapping.randcrate: For random number generation (rand = "0.8.5"inCargo.toml).
To run this code, ensure you have the
randcrate included in yourCargo.toml:[dependencies] rand = "0.8.5"Conclusion
This implementation demonstrates how you can build a simple text neural network from scratch in Rust without relying on external machine learning crates like
tch. While this model is simplistic and meant for educational purposes, it provides a foundation on which more complex models (e.g., incorporating more layers, different architectures, or advanced optimization techniques) can be built.
Feel free to ask if you need further explanations or enhancements to the code!
- Tokenization: The