Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

write a step by step code to implement the sentiment analysis of imdb dataset using bert…

  1. User

    write a step by step code to implement the sentiment analysis of imdb dataset using bert and roberta models from tokeniztion , embedding to MLOps pipeline with GitHub CI/CD, Docker, AWS, and TensorFlow for automated ML model deployment and management. Create a web interface using Gradio.

  2. Assistant

    Implementing a sentiment analysis pipeline for the IMDB dataset using BERT and RoBERTa models, integrated with an MLOps pipeline (GitHub CI/CD, Docker, AWS, TensorFlow), and deploying a web interface using Gradio is a comprehensive task. Below is a step-by-step guide to achieve this. Each step includes code snippets and explanations to guide you through the process.

    1. Setup and Prerequisites

    Before you begin, ensure you have the following installed:

    • Python 3.7+
    • TensorFlow
    • Transformers (by Hugging Face)
    • Gradio
    • Docker
    • Git
    • AWS CLI

    You can install the Python dependencies using pip:

    pip install tensorflow transformers gradio datasets
    

    2. Data Preparation

    We'll use the IMDB dataset available through Hugging Face's datasets library.

    # data_preparation.py
    
    from datasets import load_dataset
    
    def load_imdb_dataset():
        dataset = load_dataset('imdb')
        train_data = dataset['train']
        test_data = dataset['test']
        return train_data, test_data
    
    if __name__ == "__main__":
        train, test = load_imdb_dataset()
        print(f"Training samples: {len(train)}, Testing samples: {len(test)}")
    

    Run the script:

    python data_preparation.py
    

    3. Tokenization and Embedding

    We'll use BERT and RoBERTa tokenizers from Hugging Face.

    # tokenization.py
    
    from transformers import BertTokenizer, RobertaTokenizer
    from datasets import load_dataset
    
    def tokenize_data(tokenizer, data, max_length=256):
        return tokenizer(
            data['text'],
            padding='max_length',
            truncation=True,
            max_length=max_length
        )
    
    def prepare_tokenized_datasets():
        train_data, test_data = load_dataset('imdb')['train'], load_dataset('imdb')['test']
        
        bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
        roberta_tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
        
        tokenized_train_bert = tokenize_data(bert_tokenizer, train_data)
        tokenized_test_bert = tokenize_data(bert_tokenizer, test_data)
        
        tokenized_train_roberta = tokenize_data(roberta_tokenizer, train_data)
        tokenized_test_roberta = tokenize_data(roberta_tokenizer, test_data)
        
        return {
            'bert': {
                'train': tokenized_train_bert,
                'test': tokenized_test_bert
            },
            'roberta': {
                'train': tokenized_train_roberta,
                'test': tokenized_test_roberta
            }
        }
    
    if __name__ == "__main__":
        tokenized = prepare_tokenized_datasets()
        print("Tokenization complete.")
    

    Run the script:

    python tokenization.py
    

    4. Model Training with TensorFlow

    We'll create a TensorFlow model using BERT and RoBERTa for sentiment analysis.

    # train.py
    
    import tensorflow as tf
    from transformers import TFBertModel, TFRobertaModel
    from datasets import load_dataset
    from transformers import BertTokenizer, RobertaTokenizer
    import numpy as np
    
    def load_and_tokenize(model_name='bert'):
        dataset = load_dataset('imdb')
        tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') if model_name == 'bert' else RobertaTokenizer.from_pretrained('roberta-base')
        
        def tokenize(batch):
            return tokenizer(batch['text'], padding='max_length', truncation=True, max_length=256)
        
        train = dataset['train'].map(tokenize, batched=True)
        test = dataset['test'].map(tokenize, batched=True)
        
        # Convert labels to numpy arrays
        train_labels = np.array(train['label'])
        test_labels = np.array(test['label'])
        
        # Convert to TensorFlow datasets
        train_dataset = tf.data.Dataset.from_tensor_slices((
            dict(train),
            train_labels
        )).shuffle(10000).batch(32)
        
        test_dataset = tf.data.Dataset.from_tensor_slices((
            dict(test),
            test_labels
        )).batch(32)
        
        return train_dataset, test_dataset, tokenizer
    
    def build_model(model_name='bert'):
        if model_name == 'bert':
            transformer = TFBertModel.from_pretrained('bert-base-uncased')
        else:
            transformer = TFRobertaModel.from_pretrained('roberta-base')
        
        input_ids = tf.keras.layers.Input(shape=(256,), dtype=tf.int32, name='input_ids')
        attention_mask = tf.keras.layers.Input(shape=(256,), dtype=tf.int32, name='attention_mask')
        
        transformer_outputs = transformer(input_ids, attention_mask=attention_mask)
        cls_token = transformer_outputs.last_hidden_state[:,0,:]  # CLS token
        dropout = tf.keras.layers.Dropout(0.3)(cls_token)
        output = tf.keras.layers.Dense(1, activation='sigmoid')(dropout)
        
        model = tf.keras.Model(inputs=[input_ids, attention_mask], outputs=output)
        
        model.compile(
            optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5),
            loss='binary_crossentropy',
            metrics=['accuracy']
        )
        
        return model
    
    def train_and_save(model_name='bert', epochs=3):
        train_ds, test_ds, tokenizer = load_and_tokenize(model_name)
        model = build_model(model_name)
        model.fit(train_ds, epochs=epochs, validation_data=test_ds)
        model.save(f'saved_models/{model_name}_model')
        tokenizer.save_pretrained(f'saved_models/{model_name}_tokenizer')
        print(f"{model_name.capitalize()} model trained and saved.")
    
    if __name__ == "__main__":
        # Train BERT model
        train_and_save('bert')
        
        # Train RoBERTa model
        train_and_save('roberta')
    

    Run the script:

    python train.py
    

    Note: Training BERT/RoBERTa models can be resource-intensive. Consider using GPUs (e.g., through Google Colab or AWS EC2 instances with GPU support).

    5. Building the MLOps Pipeline

    We'll set up GitHub CI/CD, Dockerization, and deployment to AWS.

    5.1. Dockerize the Application

    Create a Dockerfile for the application.

    # Dockerfile
    
    FROM python:3.8-slim
    
    # Set environment variables
    ENV PYTHONDONTWRITEBYTECODE=1
    ENV PYTHONUNBUFFERED=1
    
    # Set work directory
    WORKDIR /app
    
    # Install dependencies
    COPY requirements.txt .
    RUN pip install --upgrade pip
    RUN pip install -r requirements.txt
    
    # Copy application code
    COPY . .
    
    # Expose port
    EXPOSE 7860
    
    # Command to run the app
    CMD ["python", "app.py"]
    

    Create requirements.txt:

    tensorflow
    transformers
    gradio
    datasets
    boto3
    

    5.2. Create the Gradio Web Interface

    # app.py
    
    import gradio as gr
    import tensorflow as tf
    from transformers import BertTokenizer, RobertaTokenizer
    import numpy as np
    
    # Load BERT model and tokenizer
    bert_model = tf.keras.models.load_model('saved_models/bert_model')
    bert_tokenizer = BertTokenizer.from_pretrained('saved_models/bert_tokenizer')
    
    # Load RoBERTa model and tokenizer
    roberta_model = tf.keras.models.load_model('saved_models/roberta_model')
    roberta_tokenizer = RobertaTokenizer.from_pretrained('saved_models/roberta_tokenizer')
    
    def predict_sentiment(text, model_choice):
        if model_choice == 'BERT':
            tokenizer = bert_tokenizer
            model = bert_model
        else:
            tokenizer = roberta_tokenizer
            model = roberta_model
        
        inputs = tokenizer(
            text,
            padding='max_length',
            truncation=True,
            max_length=256,
            return_tensors='tf'
        )
        predictions = model(inputs)
        score = predictions.numpy()[0][0]
        sentiment = 'Positive' if score > 0.5 else 'Negative'
        return {"Sentiment": sentiment, "Confidence": float(score)}
    
    # Create Gradio Interface
    iface = gr.Interface(
        fn=predict_sentiment,
        inputs=[
            gr.inputs.Textbox(lines=5, placeholder="Enter movie review here..."),
            gr.inputs.Radio(['BERT', 'RoBERTa'], label="Choose Model")
        ],
        outputs=[
            gr.outputs.Label(num_top_classes=2)
        ],
        title="IMDB Sentiment Analysis",
        description="Classify movie reviews as Positive or Negative using BERT and RoBERTa models."
    )
    
    if __name__ == "__main__":
        iface.launch(host="0.0.0.0", port=7860)
    

    5.3. GitHub Actions for CI/CD

    Create a .github/workflows/docker-image.yml file for building and pushing the Docker image.

    # .github/workflows/docker-image.yml
    
    name: Build and Deploy Docker Image
    
    on:
      push:
        branches: [ main ]
    
    jobs:
      build:
        runs-on: ubuntu-latest
    
        steps:
        - name: Checkout code
          uses: actions/checkout@v2
    
        - name: Set up Docker Buildx
          uses: docker/setup-buildx-action@v1
    
        - name: Log in to Docker Hub
          uses: docker/login-action@v2
          with:
            username: ${{ secrets.DOCKER_USERNAME }}
            password: ${{ secrets.DOCKER_PASSWORD }}
    
        - name: Build and push Docker image
          uses: docker/build-push-action@v3
          with:
            push: true
            tags: yourdockerhubusername/imdb-sentiment-analysis:latest
    
        - name: Deploy to AWS (using AWS CLI)
          env:
            AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
            AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
            AWS_REGION: 'us-east-1'
          run: |
            # Example using AWS ECS
            aws ecs update-service --cluster your-cluster --service your-service --force-new-deployment
    

    Notes:

    • Secrets: In your GitHub repository, add the following secrets:

      • DOCKER_USERNAME and DOCKER_PASSWORD for Docker Hub.
      • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for AWS permissions.
    • Deployment: The above example assumes you're using AWS ECS. Adjust the deployment step based on your AWS setup (EKS, EC2, etc.).

    5.4. AWS Setup

    Ensure you have an AWS account and set up the necessary services:

    1. ECR (Elastic Container Registry): To store Docker images.
    2. ECS (Elastic Container Service): To deploy Docker containers.
    3. IAM Roles: Assign appropriate permissions to allow GitHub Actions to deploy to ECS.

    Detailed AWS setup is beyond this guide's scope, but AWS documentation provides comprehensive steps.

    6. Testing the Web Interface Locally

    Before deploying, test the Gradio app locally.

    python app.py
    

    Open your browser and navigate to http://localhost:7860 to see the web interface.

    7. Deploying to AWS

    Assuming you've set up ECS and pushed your Docker image to ECR or Docker Hub, your GitHub Actions workflow can handle deployment automatically on each push to the main branch.

    Accessing the Deployed Application:

    Once deployed, navigate to the AWS ECS service's public endpoint or the load balancer associated with your service to access the Gradio web interface.

    8. Monitoring and Model Management

    For model management and monitoring, consider integrating AWS services such as:

    • CloudWatch: For logging and monitoring application metrics.
    • SageMaker: For more advanced model deployment and management features.

    Implement automated scaling, health checks, and rollback strategies within your ECS or other deployment services to ensure high availability and reliability.

    9. Optional Enhancements

    • Containerization of Training: Use separate Docker images or services for training models.
    • Model Versioning: Implement model versioning to track different model iterations.
    • Continuous Evaluation: Set up pipelines to regularly evaluate model performance on new data.
    • Security: Implement authentication and authorization for your web interface.

    Conclusion

    This guide provides a comprehensive overview of building a sentiment analysis pipeline using BERT and RoBERTa with MLOps best practices. Each component—from data preparation to deployment—can be further expanded based on specific requirements and infrastructure. Leveraging cloud services and automation tools ensures scalability, maintainability, and reliability of your machine learning applications.