Skip to content
The Chat Archive
o1-preview-2024-09-12· OpenAI WildChat-4.8M (AI2)

Make this more advanced. Don't use TCH or seperate this into multiple files. Only…

  1. User

    Make this more advanced. Don't use TCH or seperate this into multiple files. Only main.rs

    use rand::prelude::*; use std::collections::HashMap;

    // Entry point of the program fn main() { // Load and preprocess the dataset let (sentences, labels) = load_dataset();

    // Build the vocabulary from the dataset
    let vocab = build_vocab(&sentences);
    let vocab_size = vocab.len();
    
    // Convert sentences to sequences of token IDs
    let data: Vec<(Vec<usize>, usize)> = sentences
        .iter()
        .zip(labels.iter())
        .map(|(sentence, label)| {
            let token_ids = tokenize(sentence)
                .iter()
                .map(|token| *vocab.get(token).unwrap_or(&0)) // Use 0 for unknown words
                .collect();
            (token_ids, *label)
        })
        .collect();
    
    // Split the data into training and testing sets
    let split_idx = (data.len() as f32 * 0.8) as usize;
    let (train_data, test_data) = data.split_at(split_idx);
    
    // Initialize the neural network
    let embedding_dim = 32;
    let hidden_dim = 64;
    let output_dim = 2; // For binary classification
    let mut nn = NeuralNetwork::new(vocab_size + 1, embedding_dim, hidden_dim, output_dim);
    
    // Train the neural network
    let epochs = 10;
    let learning_rate = 0.01;
    train(&mut nn, train_data, epochs, learning_rate);
    
    // Evaluate the model on the test data
    evaluate(&nn, test_data);
    

    }

    // Load and preprocess a dataset (dummy implementation) fn load_dataset() -> (Vec, Vec) { // For illustration, using a small dataset let sentences = vec![ "I love this movie".to_string(), "This film was terrible".to_string(), "What an amazing experience".to_string(), "I do not like this film".to_string(), "Fantastic acting and great storyline".to_string(), "The plot was dull and boring".to_string(), "Absolutely loved it".to_string(), "Not my cup of tea".to_string(), "Brilliant from start to finish".to_string(), "Waste of time".to_string(), ];

    let labels = vec![1, 0, 1, 0, 1, 0, 1, 0, 1, 0]; // 1: Positive, 0: Negative
    
    (sentences, labels)
    

    }

    // Tokenize a sentence into words fn tokenize(sentence: &str) -> Vec { sentence .to_lowercase() .split_whitespace() .map(|s| s.trim_matches(|c: char| !c.is_alphanumeric()).to_string()) .collect() }

    // Build a vocabulary mapping from words to unique IDs fn build_vocab(sentences: &[String]) -> HashMap<String, usize> { let mut vocab = HashMap::new(); let mut index = 1; // Start indexing from 1

    for sentence in sentences {
        for token in tokenize(sentence) {
            if !vocab.contains_key(&token) {
                vocab.insert(token, index);
                index += 1;
            }
        }
    }
    
    vocab
    

    }

    // Recurrent Neural Network struct struct NeuralNetwork { vocab_size: usize, embedding_dim: usize, hidden_dim: usize, output_dim: usize, embeddings: Vec<Vec>, wxh: Vec<Vec>, // Input to hidden weights whh: Vec<Vec>, // Hidden to hidden weights why: Vec<Vec>, // Hidden to output weights bh: Vec, // Hidden bias by: Vec, // Output bias }

    impl NeuralNetwork { // Initialize the neural network with random weights fn new(vocab_size: usize, embedding_dim: usize, hidden_dim: usize, output_dim: usize) -> Self { let mut rng = thread_rng();

        let embeddings = (0..vocab_size)
            .map(|_| {
                (0..embedding_dim)
                    .map(|_| rng.gen_range(-0.1..0.1))
                    .collect()
            })
            .collect();
    
        let wxh = (0..embedding_dim)
            .map(|_| (0..hidden_dim).map(|_| rng.gen_range(-0.1..0.1)).collect())
            .collect();
    
        let whh = (0..hidden_dim)
            .map(|_| (0..hidden_dim).map(|_| rng.gen_range(-0.1..0.1)).collect())
            .collect();
    
        let why = (0..hidden_dim)
            .map(|_| (0..output_dim).map(|_| rng.gen_range(-0.1..0.1)).collect())
            .collect();
    
        let bh = vec![0.0; hidden_dim];
        let by = vec![0.0; output_dim];
    
        NeuralNetwork {
            vocab_size,
            embedding_dim,
            hidden_dim,
            output_dim,
            embeddings,
            wxh,
            whh,
            why,
            bh,
            by,
        }
    }
    
    // Forward pass: Compute the output probabilities for a given input sequence
    fn forward(
        &self,
        inputs: &[usize],
    ) -> (
        Vec<Vec<f32>>, // Hidden states
        Vec<Vec<f32>>, // Outputs
    ) {
        let mut hs: Vec<Vec<f32>> = vec![vec![0.0; self.hidden_dim]; inputs.len() + 1];
        let mut ys: Vec<Vec<f32>> = vec![vec![0.0; self.output_dim]; inputs.len()];
    
        for t in 0..inputs.len() {
            // Input Embedding
            let x_t = &self.embeddings[inputs[t]];
    
            // Hidden State Calculation: h_t = tanh(Wxh * x_t + Whh * h_{t-1} + bh)
            let wxh_xt = mat_vec_mul(&self.wxh, x_t);
            let whh_ht_1 = mat_vec_mul(&self.whh, &hs[t]);
            let mut h_t = vec_add(&wxh_xt, &whh_ht_1);
            h_t = vec_add(&h_t, &self.bh);
            h_t = h_t.iter().map(|&v| v.tanh()).collect();
    
            hs[t + 1] = h_t;
    
            // Output Calculation: y_t = Why * h_t + by
            let y_t = vec_add(&mat_vec_mul(&self.why, &hs[t + 1]), &self.by);
            ys[t] = y_t;
        }
    
        (hs, ys)
    }
    
    // Compute the loss and gradients using Backpropagation Through Time (BPTT)
    fn backward(
        &mut self,
        inputs: &[usize],
        targets: &[usize],
        hs: &[Vec<f32>],
        ys: &[Vec<f32>],
    ) -> (
        Vec<Vec<f32>>, // dWxh
        Vec<Vec<f32>>, // dWhh
        Vec<Vec<f32>>, // dWhy
        Vec<f32>,      // dbh
        Vec<f32>,      // dby
        Vec<Vec<f32>>, // dEmbeddings
    ) {
        let mut dwhy = vec![vec![0.0; self.output_dim]; self.hidden_dim];
        let mut dwhh = vec![vec![0.0; self.hidden_dim]; self.hidden_dim];
        let mut dwxh = vec![vec![0.0; self.hidden_dim]; self.embedding_dim];
        let mut dbh = vec![0.0; self.hidden_dim];
        let mut dby = vec![0.0; self.output_dim];
        let mut dh_next = vec![0.0; self.hidden_dim];
    
        let mut dembeddings = vec![vec![0.0; self.embedding_dim]; self.vocab_size];
    
        for t in (0..inputs.len()).rev() {
            // Output Error
            let mut dy = ys[t].clone();
            let target = targets[0] as f32;
            dy[0] -= target; // For binary classification
    
            // Gradients for Why and by
            let dh = mat_vec_mul_transpose(&self.why, &dy);
            let dh_total = vec_add(&dh, &dh_next);
    
            dwhy = mat_add(&dwhy, &outer_product(&hs[t + 1], &dy));
            dby = vec_add(&dby, &dy);
    
            // Backpropagate through tanh nonlinearity
            let dh_raw: Vec<f32> = (0..self.hidden_dim)
                .map(|i| dh_total[i] * (1.0 - hs[t + 1][i].powi(2)))
                .collect();
    
            // Gradients for Whh and bh
            dwhh = mat_add(&dwhh, &outer_product(&hs[t], &dh_raw));
            dbh = vec_add(&dbh, &dh_raw);
    
            // Gradients for Wxh and embeddings
            dwxh = mat_add(&dwxh, &outer_product(&self.embeddings[inputs[t]], &dh_raw));
            let dx = mat_vec_mul_transpose(&self.wxh, &dh_raw);
    
            // Update embedding gradients
            for i in 0..self.embedding_dim {
                dembeddings[inputs[t]][i] += dx[i];
            }
    
            dh_next = mat_vec_mul_transpose(&self.whh, &dh_raw);
        }
    
        (dwxh, dwhh, dwhy, dbh, dby, dembeddings)
    }
    
    // Update the weights and biases using the gradients
    fn update_parameters(
        &mut self,
        dwxh: &[Vec<f32>],
        dwhh: &[Vec<f32>],
        dwhy: &[Vec<f32>],
        dbh: &[f32],
        dby: &[f32],
        dembeddings: &[Vec<f32>],
        lr: f32,
    ) {
        // Update weights and biases
        for i in 0..self.embedding_dim {
            for j in 0..self.hidden_dim {
                self.wxh[i][j] -= lr * dwxh[i][j];
            }
        }
    
        for i in 0..self.hidden_dim {
            for j in 0..self.hidden_dim {
                self.whh[i][j] -= lr * dwhh[i][j];
            }
        }
    
        for i in 0..self.hidden_dim {
            for j in 0..self.output_dim {
                self.why[i][j] -= lr * dwhy[i][j];
            }
        }
    
        for i in 0..self.hidden_dim {
            self.bh[i] -= lr * dbh[i];
        }
    
        for i in 0..self.output_dim {
            self.by[i] -= lr * dby[i];
        }
    
        // Update embeddings
        for i in 0..self.vocab_size {
            for j in 0..self.embedding_dim {
                self.embeddings[i][j] -= lr * dembeddings[i][j];
            }
        }
    }
    
    // Predict the class for a given input sequence
    fn predict(&self, inputs: &[usize]) -> usize {
        let (_hs, ys) = self.forward(inputs);
    
        // For classification, we take the output at the last time step
        let last_output = ys.last().unwrap();
        let prediction = argmax(last_output);
    
        prediction
    }
    

    }

    // Training the neural network fn train(nn: &mut NeuralNetwork, data: &[(Vec, usize)], epochs: usize, learning_rate: f32) { for epoch in 0..epochs { let mut total_loss = 0.0; let mut correct = 0; for (inputs, label) in data { // Forward pass let (hs, ys) = nn.forward(inputs);

            // Compute loss (cross-entropy)
            let logits = ys.last().unwrap();
            let probs = softmax(logits);
            let loss = -probs[*label].ln();
            total_loss += loss;
    
            // Backward pass
            let targets = &[*label];
            let (dwxh, dwhh, dwhy, dbh, dby, dembeddings) = nn.backward(inputs, targets, &hs, &ys);
    
            // Update parameters
            nn.update_parameters(&dwxh, &dwhh, &dwhy, &dbh, &dby, &dembeddings, learning_rate);
    
            // Check prediction
            let pred = nn.predict(inputs);
            if pred == *label {
                correct += 1;
            }
        }
    
        let accuracy = correct as f32 / data.len() as f32 * 100.0;
        println!(
            "Epoch {}/{} - Loss: {:.4} - Accuracy: {:.2}%",
            epoch + 1,
            epochs,
            total_loss / data.len() as f32,
            accuracy
        );
    }
    

    }

    // Evaluate the neural network on test data fn evaluate(nn: &NeuralNetwork, data: &[(Vec, usize)]) { let mut correct = 0; for (inputs, label) in data { let pred = nn.predict(inputs); if pred == *label { correct += 1; } }

    let accuracy = correct as f32 / data.len() as f32 * 100.0;
    println!("Test Accuracy: {:.2}%", accuracy);
    

    }

    // Utility functions

    // Matrix-vector multiplication fn mat_vec_mul(mat: &[Vec], vec: &[f32]) -> Vec { let mut result = vec![0.0; mat[0].len()]; for (i, row) in mat.iter().enumerate() { for (j, &val) in row.iter().enumerate() { result[j] += vec[i] * val; } } result }

    // Matrix-vector multiplication with transpose fn mat_vec_mul_transpose(mat: &[Vec], vec: &[f32]) -> Vec { let mut result = vec![0.0; mat.len()]; for (i, row) in mat.iter().enumerate() { for (j, &val) in row.iter().enumerate() { result[i] += vec[j] * val; } } result }

    // Vector addition fn vec_add(a: &[f32], b: &[f32]) -> Vec { a.iter().zip(b.iter()).map(|(x, y)| x + y).collect() }

    // Matrix addition fn mat_add(a: &[Vec], b: &[Vec]) -> Vec<Vec> { a.iter() .zip(b.iter()) .map(|(row_a, row_b)| vec_add(row_a, row_b)) .collect() }

    // Outer product of two vectors fn outer_product(a: &[f32], b: &[f32]) -> Vec<Vec> { let mut result = vec![vec![0.0; b.len()]; a.len()]; for i in 0..a.len() { for j in 0..b.len() { result[i][j] = a[i] * b[j]; } } result }

    // Softmax function fn softmax(logits: &[f32]) -> Vec { let max = logits.iter().cloned().fold(f32::NEG_INFINITY, f32::max); let exps: Vec = logits.iter().map(|&x| (x - max).exp()).collect(); let sum: f32 = exps.iter().sum(); exps.iter().map(|&x| x / sum).collect() }

    // Argmax function fn argmax(values: &[f32]) -> usize { let mut max_index = 0; let mut max_value = f32::MIN; for (i, &value) in values.iter().enumerate() { if value > max_value { max_value = value; max_index = i; } } max_index }

  2. Assistant

    Certainly! Below is a more advanced version of your Rust code that implements an LSTM (Long Short-Term Memory) network instead of a simple RNN. This change enhances the model's capability to capture long-term dependencies in the sequence data, which is crucial for tasks like sentiment analysis.

    As per your request, the code remains in a single main.rs file and does not use the tch crate or separate into multiple files. It uses only standard libraries and rand for random number generation.

    // main.rs
    use rand::prelude::*;
    use std::collections::HashMap;
    
    // Entry point of the program
    fn main() {
        // Load and preprocess the dataset
        let (sentences, labels) = load_dataset();
    
        // Build the vocabulary from the dataset
        let vocab = build_vocab(&sentences);
        let vocab_size = vocab.len();
    
        // Convert sentences to sequences of token IDs
        let data: Vec<(Vec<usize>, usize)> = sentences
            .iter()
            .zip(labels.iter())
            .map(|(sentence, label)| {
                let token_ids = tokenize(sentence)
                    .iter()
                    .map(|token| *vocab.get(token).unwrap_or(&0)) // Use 0 for unknown words
                    .collect();
                (token_ids, *label)
            })
            .collect();
    
        // Split the data into training and testing sets
        let split_idx = (data.len() as f32 * 0.8) as usize;
        let (train_data, test_data) = data.split_at(split_idx);
    
        // Initialize the neural network
        let embedding_dim = 32;
        let hidden_dim = 64;
        let output_dim = 2; // For binary classification
        let mut nn = NeuralNetwork::new(vocab_size + 1, embedding_dim, hidden_dim, output_dim);
    
        // Train the neural network
        let epochs = 10;
        let learning_rate = 0.01;
        train(&mut nn, train_data, epochs, learning_rate);
    
        // Evaluate the model on the test data
        evaluate(&nn, test_data);
    }
    
    // Load and preprocess a dataset
    fn load_dataset() -> (Vec<String>, Vec<usize>) {
        // For illustration, using a small dataset
        let sentences = vec![
            "I love this movie".to_string(),
            "This film was terrible".to_string(),
            "What an amazing experience".to_string(),
            "I do not like this film".to_string(),
            "Fantastic acting and great storyline".to_string(),
            "The plot was dull and boring".to_string(),
            "Absolutely loved it".to_string(),
            "Not my cup of tea".to_string(),
            "Brilliant from start to finish".to_string(),
            "Waste of time".to_string(),
        ];
    
        let labels = vec![1, 0, 1, 0, 1, 0, 1, 0, 1, 0]; // 1: Positive, 0: Negative
    
        (sentences, labels)
    }
    
    // Tokenize a sentence into words
    fn tokenize(sentence: &str) -> Vec<String> {
        sentence
            .to_lowercase()
            .split_whitespace()
            .map(|s| s.trim_matches(|c: char| !c.is_alphanumeric()).to_string())
            .collect()
    }
    
    // Build a vocabulary mapping from words to unique IDs
    fn build_vocab(sentences: &[String]) -> HashMap<String, usize> {
        let mut vocab = HashMap::new();
        let mut index = 1; // Start indexing from 1
    
        for sentence in sentences {
            for token in tokenize(sentence) {
                if !vocab.contains_key(&token) {
                    vocab.insert(token, index);
                    index += 1;
                }
            }
        }
    
        vocab
    }
    
    // LSTM Neural Network struct
    struct NeuralNetwork {
        vocab_size: usize,
        embedding_dim: usize,
        hidden_dim: usize,
        output_dim: usize,
        embeddings: Vec<Vec<f32>>,
        wx: Vec<Vec<f32>>,  // Input weights for gates
        wh: Vec<Vec<f32>>,  // Hidden weights for gates
        b: Vec<f32>,        // Biases for gates
        wy: Vec<Vec<f32>>,  // Hidden to output weights
        by: Vec<f32>,       // Output bias
    }
    
    impl NeuralNetwork {
        // Initialize the neural network with random weights
        fn new(vocab_size: usize, embedding_dim: usize, hidden_dim: usize, output_dim: usize) -> Self {
            let mut rng = thread_rng();
    
            let embeddings = (0..vocab_size)
                .map(|_| {
                    (0..embedding_dim)
                        .map(|_| rng.gen_range(-0.1..0.1))
                        .collect()
                })
                .collect();
    
            let gate_num = 4; // Input gate, Forget gate, Cell gate, Output gate
    
            let wx = (0..embedding_dim)
                .map(|_| {
                    (0..hidden_dim * gate_num)
                        .map(|_| rng.gen_range(-0.1..0.1))
                        .collect()
                })
                .collect();
    
            let wh = (0..hidden_dim)
                .map(|_| {
                    (0..hidden_dim * gate_num)
                        .map(|_| rng.gen_range(-0.1..0.1))
                        .collect()
                })
                .collect();
    
            let b = vec![0.0; hidden_dim * gate_num];
    
            let wy = (0..hidden_dim)
                .map(|_| (0..output_dim).map(|_| rng.gen_range(-0.1..0.1)).collect())
                .collect();
    
            let by = vec![0.0; output_dim];
    
            NeuralNetwork {
                vocab_size,
                embedding_dim,
                hidden_dim,
                output_dim,
                embeddings,
                wx,
                wh,
                b,
                wy,
                by,
            }
        }
    
        // Forward pass: Compute the output probabilities for a given input sequence
        fn forward(
            &self,
            inputs: &[usize],
        ) -> (
            Vec<Vec<f32>>, // Hidden states
            Vec<Vec<f32>>, // Cell states
            Vec<Vec<f32>>, // Outputs
            Vec<Vec<f32>>, // Gates concatenated
        ) {
            let seq_len = inputs.len();
            let mut hs = vec![vec![0.0; self.hidden_dim]; seq_len + 1];
            let mut cs = vec![vec![0.0; self.hidden_dim]; seq_len + 1];
            let mut ys = vec![vec![0.0; self.output_dim]; seq_len];
            let mut gates_cache = Vec::with_capacity(seq_len);
    
            for t in 0..seq_len {
                let x_t = &self.embeddings[inputs[t]];
    
                // Gates computation
                let wx_x = mat_vec_mul(&self.wx, x_t);
                let wh_h = mat_vec_mul(&self.wh, &hs[t]);
                let mut gates = vec_add(&wx_x, &wh_h);
                gates = vec_add(&gates, &self.b);
    
                let (f_t, i_t, g_t, o_t) = self.split_gates(&gates);
    
                // Cell state
                let mut c_t = vec![0.0; self.hidden_dim];
                for h in 0..self.hidden_dim {
                    c_t[h] = f_t[h] * cs[t][h] + i_t[h] * g_t[h];
                }
    
                // Hidden state
                let mut h_t = vec![0.0; self.hidden_dim];
                for h in 0..self.hidden_dim {
                    h_t[h] = o_t[h] * c_t[h].tanh();
                }
    
                hs[t + 1] = h_t;
                cs[t + 1] = c_t;
                gates_cache.push((f_t, i_t, g_t, o_t, c_t.clone()));
    
                // Output
                let y_t = vec_add(&mat_vec_mul(&self.wy, &h_t), &self.by);
                ys[t] = y_t;
            }
    
            (hs, cs, ys, gates_cache)
        }
    
        // Splits concatenated gates into individual gates
        fn split_gates(&self, gates: &[f32]) -> (Vec<f32>, Vec<f32>, Vec<f32>, Vec<f32>) {
            let hidden_dim = self.hidden_dim;
            let f_t = gates[0 * hidden_dim..1 * hidden_dim]
                .iter()
                .map(|&x| sigmoid(x))
                .collect();
            let i_t = gates[1 * hidden_dim..2 * hidden_dim]
                .iter()
                .map(|&x| sigmoid(x))
                .collect();
            let g_t = gates[2 * hidden_dim..3 * hidden_dim]
                .iter()
                .map(|&x| x.tanh())
                .collect();
            let o_t = gates[3 * hidden_dim..4 * hidden_dim]
                .iter()
                .map(|&x| sigmoid(x))
                .collect();
    
            (f_t, i_t, g_t, o_t)
        }
    
        // Compute the loss and gradients using Backpropagation Through Time (BPTT)
        fn backward(
            &self,
            inputs: &[usize],
            targets: &[usize],
            hs: &[Vec<f32>],
            cs: &[Vec<f32>],
            ys: &[Vec<f32>],
            gates_cache: &[(Vec<f32>, Vec<f32>, Vec<f32>, Vec<f32>, Vec<f32>)],
        ) -> (
            Vec<Vec<f32>>, // dWx
            Vec<Vec<f32>>, // dWh
            Vec<f32>,      // db
            Vec<Vec<f32>>, // dWy
            Vec<f32>,      // dby
            Vec<Vec<f32>>, // dEmbeddings
        ) {
            let mut dwy = vec![vec![0.0; self.output_dim]; self.hidden_dim];
            let mut dwh = vec![vec![0.0; self.hidden_dim * 4]; self.hidden_dim];
            let mut dwx = vec![vec![0.0; self.hidden_dim * 4]; self.embedding_dim];
            let mut db = vec![0.0; self.hidden_dim * 4];
            let mut dby = vec![0.0; self.output_dim];
            let mut dh_next = vec![0.0; self.hidden_dim];
            let mut dc_next = vec![0.0; self.hidden_dim];
    
            let mut dembeddings = vec![vec![0.0; self.embedding_dim]; self.vocab_size];
    
            for t_rev in 0..inputs.len() {
                let t = inputs.len() - 1 - t_rev;
    
                // Output error
                let mut dy = ys[t].clone();
                let target = targets[0] as usize;
                dy[target] -= 1.0; // For cross-entropy loss
    
                // Gradients for wy and by
                dwy = mat_add(&dwy, &outer_product(&hs[t + 1], &dy));
                dby = vec_add(&dby, &dy);
    
                // Backprop into h_t
                let dh = vec_add(&mat_vec_mul_transpose(&self.wy, &dy), &dh_next);
    
                // Retrieve cached gates and cell state
                let (f_t, i_t, g_t, o_t, c_t) = &gates_cache[t];
                let c_prev = &cs[t];
                let tanh_c_t = c_t.iter().map(|&x| x.tanh()).collect::<Vec<f32>>();
    
                // Backprop through output gate
                let do_t = elementwise_mul(&dh, &tanh_c_t);
                let do_t = elementwise_mul(&do_t, &sigmoid_derivative_vec(o_t));
    
                // Backprop into cell state
                let dc = vec_add(
                    &elementwise_mul(&dh, &elementwise_mul(o_t, &tanh_derivative_vec(&tanh_c_t))),
                    &dc_next,
                );
    
                let dg_t = elementwise_mul(&dc, i_t);
                let dg_t = elementwise_mul(&dg_t, &tanh_derivative_vec(g_t));
    
                let di_t = elementwise_mul(&dc, g_t);
                let di_t = elementwise_mul(&di_t, &sigmoid_derivative_vec(i_t));
    
                let df_t = elementwise_mul(&dc, c_prev);
                let df_t = elementwise_mul(&df_t, &sigmoid_derivative_vec(f_t));
    
                // Concatenate gate derivatives
                let gates_grad = [
                    df_t.clone(),
                    di_t.clone(),
                    dg_t.clone(),
                    do_t.clone(),
                ]
                .concat();
    
                // Gradients for wx and wh
                dwx = mat_add(
                    &dwx,
                    &outer_product(&self.embeddings[inputs[t]], &gates_grad),
                );
                dwh = mat_add(&dwh, &outer_product(&hs[t], &gates_grad));
                db = vec_add(&db, &gates_grad);
    
                // Gradients w.r.t inputs for t-1
                let dx = mat_vec_mul_transpose(&self.wx, &gates_grad);
                let dh_prev = mat_vec_mul_transpose(&self.wh, &gates_grad);
    
                // Update embedding gradients
                for i in 0..self.embedding_dim {
                    dembeddings[inputs[t]][i] += dx[i];
                }
    
                dh_next = dh_prev;
                dc_next = elementwise_mul(&dc, f_t);
            }
    
            (dwx, dwh, db, dwy, dby, dembeddings)
        }
    
        // Update the weights and biases using the gradients
        fn update_parameters(
            &mut self,
            dwx: &[Vec<f32>],
            dwh: &[Vec<f32>],
            db: &[f32],
            dwy: &[Vec<f32>],
            dby: &[f32],
            dembeddings: &[Vec<f32>],
            lr: f32,
        ) {
            // Update weights and biases
            for i in 0..self.embedding_dim {
                for j in 0..self.hidden_dim * 4 {
                    self.wx[i][j] -= lr * dwx[i][j];
                }
            }
    
            for i in 0..self.hidden_dim {
                for j in 0..self.hidden_dim * 4 {
                    self.wh[i][j] -= lr * dwh[i][j];
                }
            }
    
            for i in 0..self.hidden_dim * 4 {
                self.b[i] -= lr * db[i];
            }
    
            for i in 0..self.hidden_dim {
                for j in 0..self.output_dim {
                    self.wy[i][j] -= lr * dwy[i][j];
                }
            }
    
            for i in 0..self.output_dim {
                self.by[i] -= lr * dby[i];
            }
    
            // Update embeddings
            for i in 0..self.vocab_size {
                for j in 0..self.embedding_dim {
                    self.embeddings[i][j] -= lr * dembeddings[i][j];
                }
            }
        }
    
        // Predict the class for a given input sequence
        fn predict(&self, inputs: &[usize]) -> usize {
            let (_hs, _cs, ys, _gates_cache) = self.forward(inputs);
    
            // For classification, we take the output at the last time step
            let last_output = ys.last().unwrap();
            let prediction = argmax(last_output);
    
            prediction
        }
    }
    
    // Training the neural network
    fn train(
        nn: &mut NeuralNetwork,
        data: &[(Vec<usize>, usize)],
        epochs: usize,
        learning_rate: f32,
    ) {
        for epoch in 0..epochs {
            let mut total_loss = 0.0;
            let mut correct = 0;
            for (inputs, label) in data {
                // Forward pass
                let (hs, cs, ys, gates_cache) = nn.forward(inputs);
    
                // Compute loss (cross-entropy)
                let logits = ys.last().unwrap();
                let probs = softmax(logits);
                let loss = -probs[*label].ln();
                total_loss += loss;
    
                // Backward pass
                let targets = &[*label];
                let (dwx, dwh, db, dwy, dby, dembeddings) =
                    nn.backward(inputs, targets, &hs, &cs, &ys, &gates_cache);
    
                // Update parameters
                nn.update_parameters(&dwx, &dwh, &db, &dwy, &dby, &dembeddings, learning_rate);
    
                // Check prediction
                let pred = nn.predict(inputs);
                if pred == *label {
                    correct += 1;
                }
            }
    
            let accuracy = correct as f32 / data.len() as f32 * 100.0;
            println!(
                "Epoch {}/{} - Loss: {:.4} - Accuracy: {:.2}%",
                epoch + 1,
                epochs,
                total_loss / data.len() as f32,
                accuracy
            );
        }
    }
    
    // Evaluate the neural network on test data
    fn evaluate(nn: &NeuralNetwork, data: &[(Vec<usize>, usize)]) {
        let mut correct = 0;
        for (inputs, label) in data {
            let pred = nn.predict(inputs);
            if pred == *label {
                correct += 1;
            }
        }
    
        let accuracy = correct as f32 / data.len() as f32 * 100.0;
        println!("Test Accuracy: {:.2}%", accuracy);
    }
    
    // Utility functions
    
    // Matrix-vector multiplication
    fn mat_vec_mul(mat: &[Vec<f32>], vec: &[f32]) -> Vec<f32> {
        let mut result = vec![0.0; mat[0].len()];
        for (i, row) in mat.iter().enumerate() {
            for (j, &val) in row.iter().enumerate() {
                result[j] += vec[i] * val;
            }
        }
        result
    }
    
    // Matrix-vector multiplication with transpose
    fn mat_vec_mul_transpose(mat: &[Vec<f32>], vec: &[f32]) -> Vec<f32> {
        let mut result = vec![0.0; mat.len()];
        for (i, row) in mat.iter().enumerate() {
            for (j, &val) in row.iter().enumerate() {
                result[i] += vec[j] * val;
            }
        }
        result
    }
    
    // Vector addition
    fn vec_add(a: &[f32], b: &[f32]) -> Vec<f32> {
        a.iter().zip(b.iter()).map(|(x, y)| x + y).collect()
    }
    
    // Element-wise multiplication
    fn elementwise_mul(a: &[f32], b: &[f32]) -> Vec<f32> {
        a.iter().zip(b.iter()).map(|(x, y)| x * y).collect()
    }
    
    // Sigmoid function
    fn sigmoid(x: f32) -> f32 {
        1.0 / (1.0 + (-x).exp())
    }
    
    // Derivative of sigmoid function
    fn sigmoid_derivative_vec(s: &[f32]) -> Vec<f32> {
        s.iter().map(|&x| x * (1.0 - x)).collect()
    }
    
    // Derivative of tanh function
    fn tanh_derivative_vec(tanh_x: &[f32]) -> Vec<f32> {
        tanh_x.iter().map(|&x| 1.0 - x * x).collect()
    }
    
    // Matrix addition
    fn mat_add(a: &[Vec<f32>], b: &[Vec<f32>]) -> Vec<Vec<f32>> {
        a.iter()
            .zip(b.iter())
            .map(|(row_a, row_b)| vec_add(row_a, row_b))
            .collect()
    }
    
    // Outer product of two vectors
    fn outer_product(a: &[f32], b: &[f32]) -> Vec<Vec<f32>> {
        let mut result = vec![vec![0.0; b.len()]; a.len()];
        for i in 0..a.len() {
            for j in 0..b.len() {
                result[i][j] = a[i] * b[j];
            }
        }
        result
    }
    
    // Softmax function
    fn softmax(logits: &[f32]) -> Vec<f32> {
        let max = logits.iter().cloned().fold(f32::NEG_INFINITY, f32::max);
        let exps: Vec<f32> = logits.iter().map(|&x| (x - max).exp()).collect();
        let sum: f32 = exps.iter().sum();
        exps.iter().map(|&x| x / sum).collect()
    }
    
    // Argmax function
    fn argmax(values: &[f32]) -> usize {
        let mut max_index = 0;
        let mut max_value = f32::MIN;
        for (i, &value) in values.iter().enumerate() {
            if value > max_value {
                max_value = value;
                max_index = i;
            }
        }
        max_index
    }
    

    Explanation of Changes:

    • Replaced Simple RNN with LSTM:

      • Implemented an LSTM network to handle long-term dependencies better than a simple RNN.
      • Modified the NeuralNetwork struct to include weights and biases for the LSTM gates: input gate, forget gate, cell gate, and output gate.
    • Updated Forward Pass (forward function):

      • Computations for LSTM gates are added.
      • The hidden state (h_t) and cell state (c_t) are updated using the LSTM formulas.
      • Gates activations and cell states are cached for backpropagation.
    • Updated Backward Pass (backward function):

      • Implemented the backward pass for LSTM, including backpropagation through time (BPTT).
      • Computed gradients for each gate: input, forget, output, and cell gate.
      • Updated the gradient calculations for weights, biases, and embeddings accordingly.
    • Added Utility Functions:

      • Implemented sigmoid, sigmoid_derivative_vec, tanh_derivative_vec, and elementwise_mul functions to support LSTM computations.
    • Adjusted Training and Evaluation Functions:

      • Updated the train function to work with the modified forward and backward functions.
      • The loss calculation now properly handles multi-class output using softmax and cross-entropy.
    • Kept Code in a Single File:

      • All changes are made within main.rs without using external crates like tch or splitting into multiple files.

    Additional Enhancements:

    • Improved Loss Function:

      • Replaced the binary cross-entropy with softmax and cross-entropy to handle multi-class outputs properly.
      • The code can easily be extended to multi-class classification if needed.
    • Optimizations:

      • Vectorized operations are used wherever possible to improve computational efficiency.
      • Caching of intermediate results is done to prevent redundant computations during backpropagation.
    • Flexibility:

      • The network architecture is more modular, allowing for easy adjustments of hyperparameters like embedding_dim, hidden_dim, and output_dim.

    Note:

    This implementation is a simplified version of an LSTM network for educational purposes. In practice, implementing neural networks from scratch involves careful handling of numerical stability, computational efficiency, and often requires optimizations like gradient clipping and advanced optimization algorithms (e.g., Adam optimizer).

    For real-world applications, it's recommended to use established deep learning libraries that handle these complexities internally, such as tch, tensorflow, or autograd crates in Rust. These libraries provide optimized computational graphs, automatic differentiation, and support for GPU acceleration.


    Please let me know if you need further adjustments or explanations!