Skip to content
The Chat Archive
o1-mini-2024-09-12· OpenAI WildChat-4.8M (AI2)

Usage Installation Grokfast doesn't require additional packages except for PyTorch. The…

  1. User

    Usage Installation Grokfast doesn't require additional packages except for PyTorch. The file requirements.txt is only for reproduction of the experiments in the article, as described in the Reproduction section below.

    Instructions Grokfast can be applied by inserting a single line before the optimizer call.

    Download a single file grokfast.py from our repository. wget https://raw.githubusercontent.com/ironjr/grokfast/main/grokfast.py Import the helper function. from grokfast import gradfilter_ma, gradfilter_ema Insert the following line before the training loop. grads = None Between loss.backward() and optimizer.step(), insert one of the following line. Make sure model is of type nn.Module and grads are initialized properly before the training loop:

    ... in the optimization loop.

    loss.backwards() # Calculate the gradients.

    Option 1: Grokfast (has argument alpha, lamb)

    grads = gradfilter_ema(model, grads=grads, alpha=alpha, lamb=lamb)

    Option 2: Grokfast-MA (has argument window_size, lamb)

    grads = gradfilter_ma(model, grads=grads, window_size=window_size, lamb=lamb)

    optimizer.step() # Call the optimizer.

    ... logging & other codes.

    Done!

    (2-1) ...or, copy and paste the method directly into your code! Arguments Grokfast (gradfilter_ema)

    m: nn.Module: Model that contains every trainable parameters. grads: Optional[Dict[str, torch.Tensor]] = None: Running memory (EMA). Initialize by setting it to None. Feed the output of the method recursively after on. alpha: float = 0.98: Momentum hyperparmeter of the EMA. lamb: float = 2.0: Amplifying factor hyperparameter of the filter. Grokfast-MA (gradfilter_ma)

    m: nn.Module: Model that contains every trainable parameters. grads: Optional[Dict[str, deque]] = None: Running memory (Queue for windowed moving average). Initialize by setting it to None. Feed the output of the method recursively after on. window_size: int = 100: The width of the filter window. Additional memory requirements increases linearly with respect to the windows size. lamb: float = 5.0: Amplifying factor hyperparameter of the filter. filter_type: Literal['mean', 'sum'] = 'mean': Aggregation method for the running queue. warmup: bool = True: If true, filter is not applied until the queue is filled. trigger: bool = False: For ablation study only. If true, the filter is simply not applied. Reproduction We also note the additional computational resources required for each run. Time & memory costs are measured with a single GTX 1080 Ti GPU.

    Installation This will install the additional packages to preprocess each data and to summarize the results.

    conda create -n grok python=3.10 && conda activate grok git clone https://github.com/ironjr/grokfast pip install -r requirements.txt Algorithmic Data (Transformer decoder, Grokfast-MA) Run Iterations to Reach 95% Val. Acc. Wall Clock Time to Reach 95% Val. Acc. (s) VRAM Requirements (MB) Latency Per Iteration (s) Baseline 39890 5984 290 0.15 Grokfast-MA 790 ( × 50.49 ↓ ) 292 ( × 20.49 ↓ ) 458 0.37

    python main.py --label test # Baseline.

    python main.py --label test --filter ma --window_size 100 --lamb 5.0 --weight_decay 0.01 Algorithmic Data (Transformer decoder, Grokfast) Run Iterations to Reach 95% Val. Acc. Wall Clock Time to Reach 95% Val. Acc. (s) VRAM Requirements (MB) Latency Per Iteration (s) Baseline 39890 5984 $290 0.15 Grokfast 910 ( × 43.84 ↓ ) 137 ( × 43.79 ↓ ) 294 0.15

    python main.py --label test # Baseline.

    python main.py --label test --filter ema --alpha 0.98 --lamb 2.0 --weight_decay 0.005 MNIST (MLP) Run Iterations to Reach 95% Val. Acc. Wall Clock Time to Reach 95% Val. Acc. (s) VRAM Requirements (MB) Latency Per Iteration (ms) Baseline 44022 1928 196 43.8 Grokfast 2001 ( × 22.00 ↓ ) 87.8 ( × 21.96 ↓ ) 198 43.9

    python main_mnist.py --label test # Baseline.

    python main_mnist.py --label test --alpha 0.8 --lamb 0.1 --weight_decay 2.0 IMDb (LSTM) Run Best Validation Acc. Minimum Validation Loss VRAM Requirements (MB) Latency Per Iteration (ms) Baseline 0.84 0.517 754 20.4 Grokfast 0.90 0.412 762 21.2 Before training, download the IMDb dataset from Google Drive or Baidu Webdisk (pw: vdp7).

    python main_imdb.py --label test # Baseline.

    python main_imdb.py --label test --alpha 0.98 --lamb 2.0 --weight_decay 10.0 QM9 (G-CNN) Run Minimum Validation Loss VRAM Requirements (MB) Latency Per Iteration (ms) Baseline 0.00659 216 40.2 Grokfast 0.00348 216 41.4

    python main_qm9.py --label test # Baseline.

    python main_qm9.py --label test --alpha 0.9 --lamb 1.0 --weight_decay 0.01 FAQ Choosing the right hyperparameters These recommendations are based on my experiences during the experiments shown in the main manuscript. This may not work perfectly to every other problems, and maybe more intelligent techniques can do better jobs than this procedure. So, please take these as one possible starting guidelines for designing your own filters.

    Cutoff parameters: The work uses MA/EMA filters to implement the filtering techniques. The cutoff frequency is determined by the window size for the MA filter, and the momentum parameter for the EMA filter. Roughly figure out the amount of acceleration you want to achieve. For example, in the main manuscript, the cutoff parameters are determined based on the original grokking report, where experiments shows generalization happening X100 slower than overfitting. Therefore, we want N=100 times faster acceleration. Set the pivotal values for the cutoff parameter search. For MA, I started to set the window size of "w=N=100" and for EMA, I began with the momentum parameter alpha that satisfies "alpha^{N} = alpha^{100} = 0.1" (which is roughly alpha ~ 0.98). Perform hyperparameter search near the pivot values. I swept across hyperparameter values near the values set in (1.b). Weight decay: The weight decay is set in the optimizer constructor as usual (e.g., optimizer = optim.Adam(m.parameters(), weight_decay=wd)). Start from the default weight decay of that task. For example, the value chosen by the most widely used Github repository of that task. Fix the weight decay and try to find the optimal setting for the Grokfast filter parameters (momentum, window size, and amplitude) first. Although weight decay do affect the values of the optimal filter parameters, its effect seems to be insignificant in my experiences. Start increasing the weight decay value. Start from X1 then try (X2, X5, X10). I couldn't get better results with X100 scale of the default value.

    using this documentation, train 2 MLP MNIST models with everything being the same (including epochs) except for the usage of grokfast (only 1 model should use that). The models should be small and trained for 100 epochs. every 10 epochs, print the train loss, val loss, train accuracy, and test accuracy and after that, make a graph with all that for each model to compare the differences in performance.

  2. Assistant

    Sure! Below is a comprehensive guide to training two MLP models on the MNIST dataset using PyTorch: one with the Grokfast gradient filter and one without. Both models are small and trained for 100 epochs. Every 10 epochs, the training will print the training loss, validation loss, training accuracy, and test accuracy. Finally, a graph will be generated to compare the performance of both models.

    Step-by-Step Guide

    1. Setup Environment
    2. Download Grokfast
    3. Import Necessary Libraries
    4. Define the MLP Model
    5. Prepare the MNIST Dataset
    6. Define Training and Evaluation Functions
    7. Train Both Models
    8. Plot the Results

    1. Setup Environment

    First, ensure you have Python 3.10 installed. Then, create a new Conda environment and install the required packages.

    # Create and activate a new Conda environment
    conda create -n grok_mnist python=3.10
    conda activate grok_mnist
    
    # Install PyTorch (adjust the command based on your CUDA version)
    # For CPU only:
    pip install torch torchvision
    
    # Install other required packages
    pip install matplotlib
    

    2. Download Grokfast

    Download the grokfast.py file directly from the repository.

    wget https://raw.githubusercontent.com/ironjr/grokfast/main/grokfast.py
    

    3. Import Necessary Libraries

    import torch
    import torch.nn as nn
    import torch.optim as optim
    from torchvision import datasets, transforms
    from torch.utils.data import DataLoader, random_split
    from grokfast import gradfilter_ema  # Import Grokfast's EMA filter
    import matplotlib.pyplot as plt
    import copy
    import time
    

    4. Define the MLP Model

    class SimpleMLP(nn.Module):
        def __init__(self):
            super(SimpleMLP, self).__init__()
            self.flatten = nn.Flatten()
            self.model = nn.Sequential(
                nn.Linear(28*28, 256),
                nn.ReLU(),
                nn.Linear(256, 128),
                nn.ReLU(),
                nn.Linear(128, 10)
            )
        
        def forward(self, x):
            x = self.flatten(x)
            return self.model(x)
    

    5. Prepare the MNIST Dataset

    # Transformations
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.1307,), (0.3081,))
    ])
    
    # Download MNIST dataset
    train_val_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
    test_dataset = datasets.MNIST(root='./data', train=False, download=True, transform=transform)
    
    # Split into training and validation
    train_size = int(0.8 * len(train_val_dataset))
    val_size = len(train_val_dataset) - train_size
    train_dataset, val_dataset = random_split(train_val_dataset, [train_size, val_size])
    
    # Data loaders
    batch_size = 64
    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)
    test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
    

    6. Define Training and Evaluation Functions

    def train_model(model, optimizer, criterion, train_loader, val_loader, use_grokfast=False, epochs=100, print_every=10, alpha=0.98, lamb=2.0):
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        model.to(device)
        
        grads = None  # Initialize grads for Grokfast
        metrics = {
            'train_loss': [],
            'val_loss': [],
            'train_acc': [],
            'val_acc': []
        }
        
        for epoch in range(1, epochs + 1):
            model.train()
            running_loss = 0.0
            correct = 0
            total = 0
            
            start_time = time.time()
            
            for data, target in train_loader:
                data, target = data.to(device), target.to(device)
                
                optimizer.zero_grad()
                outputs = model(data)
                loss = criterion(outputs, target)
                loss.backward()
                
                if use_grokfast:
                    grads = gradfilter_ema(model, grads=grads, alpha=alpha, lamb=lamb)
                
                optimizer.step()
                
                running_loss += loss.item() * data.size(0)
                _, predicted = torch.max(outputs.data, 1)
                total += target.size(0)
                correct += (predicted == target).sum().item()
            
            train_loss = running_loss / len(train_loader.dataset)
            train_acc = correct / total
            
            # Validation
            model.eval()
            val_loss = 0.0
            correct = 0
            total = 0
            
            with torch.no_grad():
                for data, target in val_loader:
                    data, target = data.to(device), target.to(device)
                    outputs = model(data)
                    loss = criterion(outputs, target)
                    
                    val_loss += loss.item() * data.size(0)
                    _, predicted = torch.max(outputs.data, 1)
                    total += target.size(0)
                    correct += (predicted == target).sum().item()
            
            val_loss /= len(val_loader.dataset)
            val_acc = correct / total
            
            metrics['train_loss'].append(train_loss)
            metrics['val_loss'].append(val_loss)
            metrics['train_acc'].append(train_acc)
            metrics['val_acc'].append(val_acc)
            
            if epoch % print_every == 0:
                epoch_time = time.time() - start_time
                print(f'Epoch [{epoch}/{epochs}] - Time: {epoch_time:.2f}s')
                print(f'Train Loss: {train_loss:.4f} | Train Acc: {train_acc*100:.2f}%')
                print(f'Val Loss: {val_loss:.4f} | Val Acc: {val_acc*100:.2f}%\n')
        
        return metrics
    

    7. Train Both Models

    # Define loss criterion
    criterion = nn.CrossEntropyLoss()
    
    # ======= Model without Grokfast =======
    model_baseline = SimpleMLP()
    optimizer_baseline = optim.Adam(model_baseline.parameters(), lr=0.001, weight_decay=0.01)
    
    print("Training Baseline Model (Without Grokfast)...\n")
    metrics_baseline = train_model(
        model=model_baseline,
        optimizer=optimizer_baseline,
        criterion=criterion,
        train_loader=train_loader,
        val_loader=val_loader,
        use_grokfast=False,
        epochs=100,
        print_every=10
    )
    
    # ======= Model with Grokfast =======
    model_grokfast = SimpleMLP()
    optimizer_grokfast = optim.Adam(model_grokfast.parameters(), lr=0.001, weight_decay=0.01)
    
    print("\nTraining Model with Grokfast...\n")
    metrics_grokfast = train_model(
        model=model_grokfast,
        optimizer=optimizer_grokfast,
        criterion=criterion,
        train_loader=train_loader,
        val_loader=val_loader,
        use_grokfast=True,
        epochs=100,
        print_every=10,
        alpha=0.98,
        lamb=2.0
    )
    

    8. Evaluate on Test Set

    def evaluate_model(model, test_loader):
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        model.to(device)
        model.eval()
        
        correct = 0
        total = 0
        
        with torch.no_grad():
            for data, target in test_loader:
                data, target = data.to(device), target.to(device)
                outputs = model(data)
                _, predicted = torch.max(outputs.data, 1)
                total += target.size(0)
                correct += (predicted == target).sum().item()
        
        test_acc = correct / total
        return test_acc
    
    # Evaluate Baseline Model
    test_acc_baseline = evaluate_model(model_baseline, test_loader)
    print(f'Baseline Model Test Accuracy: {test_acc_baseline*100:.2f}%')
    
    # Evaluate Grokfast Model
    test_acc_grokfast = evaluate_model(model_grokfast, test_loader)
    print(f'Grokfast Model Test Accuracy: {test_acc_grokfast*100:.2f}%')
    

    9. Plot the Results

    epochs = range(10, 101, 10)
    
    plt.figure(figsize=(12, 10))
    
    # Train Loss
    plt.subplot(2, 2, 1)
    plt.plot(epochs, metrics_baseline['train_loss'], label='Baseline')
    plt.plot(epochs, metrics_grokfast['train_loss'], label='Grokfast')
    plt.xlabel('Epoch')
    plt.ylabel('Training Loss')
    plt.title('Training Loss Comparison')
    plt.legend()
    
    # Validation Loss
    plt.subplot(2, 2, 2)
    plt.plot(epochs, metrics_baseline['val_loss'], label='Baseline')
    plt.plot(epochs, metrics_grokfast['val_loss'], label='Grokfast')
    plt.xlabel('Epoch')
    plt.ylabel('Validation Loss')
    plt.title('Validation Loss Comparison')
    plt.legend()
    
    # Train Accuracy
    plt.subplot(2, 2, 3)
    plt.plot(epochs, [acc * 100 for acc in metrics_baseline['train_acc']], label='Baseline')
    plt.plot(epochs, [acc * 100 for acc in metrics_grokfast['train_acc']], label='Grokfast')
    plt.xlabel('Epoch')
    plt.ylabel('Training Accuracy (%)')
    plt.title('Training Accuracy Comparison')
    plt.legend()
    
    # Validation Accuracy
    plt.subplot(2, 2, 4)
    plt.plot(epochs, [acc * 100 for acc in metrics_baseline['val_acc']], label='Baseline')
    plt.plot(epochs, [acc * 100 for acc in metrics_grokfast['val_acc']], label='Grokfast')
    plt.xlabel('Epoch')
    plt.ylabel('Validation Accuracy (%)')
    plt.title('Validation Accuracy Comparison')
    plt.legend()
    
    plt.tight_layout()
    plt.show()
    

    Complete Script

    For convenience, here's the complete script combining all the steps above. Save this as train_mnist_grokfast.py and run it.

    import torch
    import torch.nn as nn
    import torch.optim as optim
    from torchvision import datasets, transforms
    from torch.utils.data import DataLoader, random_split
    from grokfast import gradfilter_ema  # Import Grokfast's EMA filter
    import matplotlib.pyplot as plt
    import copy
    import time
    
    # Define the MLP Model
    class SimpleMLP(nn.Module):
        def __init__(self):
            super(SimpleMLP, self).__init__()
            self.flatten = nn.Flatten()
            self.model = nn.Sequential(
                nn.Linear(28*28, 256),
                nn.ReLU(),
                nn.Linear(256, 128),
                nn.ReLU(),
                nn.Linear(128, 10)
            )
        
        def forward(self, x):
            x = self.flatten(x)
            return self.model(x)
    
    # Prepare the MNIST Dataset
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.1307,), (0.3081,))
    ])
    
    train_val_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
    test_dataset = datasets.MNIST(root='./data', train=False, download=True, transform=transform)
    
    train_size = int(0.8 * len(train_val_dataset))
    val_size = len(train_val_dataset) - train_size
    train_dataset, val_dataset = random_split(train_val_dataset, [train_size, val_size])
    
    batch_size = 64
    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)
    test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
    
    # Training Function
    def train_model(model, optimizer, criterion, train_loader, val_loader, use_grokfast=False, epochs=100, print_every=10, alpha=0.98, lamb=2.0):
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        model.to(device)
        
        grads = None  # Initialize grads for Grokfast
        metrics = {
            'train_loss': [],
            'val_loss': [],
            'train_acc': [],
            'val_acc': []
        }
        
        for epoch in range(1, epochs + 1):
            model.train()
            running_loss = 0.0
            correct = 0
            total = 0
            
            start_time = time.time()
            
            for data, target in train_loader:
                data, target = data.to(device), target.to(device)
                
                optimizer.zero_grad()
                outputs = model(data)
                loss = criterion(outputs, target)
                loss.backward()
                
                if use_grokfast:
                    grads = gradfilter_ema(model, grads=grads, alpha=alpha, lamb=lamb)
                
                optimizer.step()
                
                running_loss += loss.item() * data.size(0)
                _, predicted = torch.max(outputs.data, 1)
                total += target.size(0)
                correct += (predicted == target).sum().item()
            
            train_loss = running_loss / len(train_loader.dataset)
            train_acc = correct / total
            
            # Validation
            model.eval()
            val_loss = 0.0
            correct = 0
            total = 0
            
            with torch.no_grad():
                for data, target in val_loader:
                    data, target = data.to(device), target.to(device)
                    outputs = model(data)
                    loss = criterion(outputs, target)
                    
                    val_loss += loss.item() * data.size(0)
                    _, predicted = torch.max(outputs.data, 1)
                    total += target.size(0)
                    correct += (predicted == target).sum().item()
            
            val_loss /= len(val_loader.dataset)
            val_acc = correct / total
            
            metrics['train_loss'].append(train_loss)
            metrics['val_loss'].append(val_loss)
            metrics['train_acc'].append(train_acc)
            metrics['val_acc'].append(val_acc)
            
            if epoch % print_every == 0:
                epoch_time = time.time() - start_time
                print(f'Epoch [{epoch}/{epochs}] - Time: {epoch_time:.2f}s')
                print(f'Train Loss: {train_loss:.4f} | Train Acc: {train_acc*100:.2f}%')
                print(f'Val Loss: {val_loss:.4f} | Val Acc: {val_acc*100:.2f}%\n')
        
        return metrics
    
    # Evaluation Function
    def evaluate_model(model, test_loader):
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        model.to(device)
        model.eval()
        
        correct = 0
        total = 0
        
        with torch.no_grad():
            for data, target in test_loader:
                data, target = data.to(device), target.to(device)
                outputs = model(data)
                _, predicted = torch.max(outputs.data, 1)
                total += target.size(0)
                correct += (predicted == target).sum().item()
        
        test_acc = correct / total
        return test_acc
    
    # Define loss criterion
    criterion = nn.CrossEntropyLoss()
    
    # ======= Model without Grokfast =======
    model_baseline = SimpleMLP()
    optimizer_baseline = optim.Adam(model_baseline.parameters(), lr=0.001, weight_decay=0.01)
    
    print("Training Baseline Model (Without Grokfast)...\n")
    metrics_baseline = train_model(
        model=model_baseline,
        optimizer=optimizer_baseline,
        criterion=criterion,
        train_loader=train_loader,
        val_loader=val_loader,
        use_grokfast=False,
        epochs=100,
        print_every=10
    )
    
    # ======= Model with Grokfast =======
    model_grokfast = SimpleMLP()
    optimizer_grokfast = optim.Adam(model_grokfast.parameters(), lr=0.001, weight_decay=0.01)
    
    print("\nTraining Model with Grokfast...\n")
    metrics_grokfast = train_model(
        model=model_grokfast,
        optimizer=optimizer_grokfast,
        criterion=criterion,
        train_loader=train_loader,
        val_loader=val_loader,
        use_grokfast=True,
        epochs=100,
        print_every=10,
        alpha=0.98,
        lamb=2.0
    )
    
    # Evaluate Models on Test Set
    test_acc_baseline = evaluate_model(model_baseline, test_loader)
    print(f'\nBaseline Model Test Accuracy: {test_acc_baseline*100:.2f}%')
    
    test_acc_grokfast = evaluate_model(model_grokfast, test_loader)
    print(f'Grokfast Model Test Accuracy: {test_acc_grokfast*100:.2f}%')
    
    # Plotting the Results
    epochs_plot = range(10, 101, 10)
    
    plt.figure(figsize=(12, 10))
    
    # Train Loss
    plt.subplot(2, 2, 1)
    plt.plot(epochs_plot, metrics_baseline['train_loss'], label='Baseline')
    plt.plot(epochs_plot, metrics_grokfast['train_loss'], label='Grokfast')
    plt.xlabel('Epoch')
    plt.ylabel('Training Loss')
    plt.title('Training Loss Comparison')
    plt.legend()
    
    # Validation Loss
    plt.subplot(2, 2, 2)
    plt.plot(epochs_plot, metrics_baseline['val_loss'], label='Baseline')
    plt.plot(epochs_plot, metrics_grokfast['val_loss'], label='Grokfast')
    plt.xlabel('Epoch')
    plt.ylabel('Validation Loss')
    plt.title('Validation Loss Comparison')
    plt.legend()
    
    # Train Accuracy
    plt.subplot(2, 2, 3)
    plt.plot(epochs_plot, [acc * 100 for acc in metrics_baseline['train_acc']], label='Baseline')
    plt.plot(epochs_plot, [acc * 100 for acc in metrics_grokfast['train_acc']], label='Grokfast')
    plt.xlabel('Epoch')
    plt.ylabel('Training Accuracy (%)')
    plt.title('Training Accuracy Comparison')
    plt.legend()
    
    # Validation Accuracy
    plt.subplot(2, 2, 4)
    plt.plot(epochs_plot, [acc * 100 for acc in metrics_baseline['val_acc']], label='Baseline')
    plt.plot(epochs_plot, [acc * 100 for acc in metrics_grokfast['val_acc']], label='Grokfast')
    plt.xlabel('Epoch')
    plt.ylabel('Validation Accuracy (%)')
    plt.title('Validation Accuracy Comparison')
    plt.legend()
    
    plt.tight_layout()
    plt.show()
    

    Running the Script

    1. Ensure all dependencies are installed and the grokfast.py file is in the same directory as your script.
    2. Run the script:
    python train_mnist_grokfast.py
    

    Expected Output

    Every 10 epochs, you will see output similar to:

    Training Baseline Model (Without Grokfast)...
    
    Epoch [10/100] - Time: 5.23s
    Train Loss: 0.3214 | Train Acc: 89.50%
    Val Loss: 0.1023 | Val Acc: 96.30%
    
    ...
    

    After training both models, the test accuracies will be printed, and a graph comparing training and validation losses and accuracies will be displayed.

    Notes

    • Hyperparameters: The alpha and lamb values used for Grokfast are based on the recommendations from the documentation. You may need to adjust these based on your specific requirements.

    • Model Complexity: The MLP model defined here is relatively small. Depending on your computational resources, you can adjust the number of layers and neurons.

    • Performance: Training for 100 epochs might take some time depending on your hardware. You can reduce the number of epochs for quicker experiments.

    Feel free to modify and extend this script to better fit your needs!