write a step by step code to implement the sentiment analysis of imdb dataset using bert…
- User
write a step by step code to implement the sentiment analysis of imdb dataset using bert and roberta models from tokeniztion , embedding to MLOps pipeline with GitHub CI/CD, Docker, AWS, and TensorFlow for automated ML model deployment and management. Create a web interface using Gradio.
- Assistant
Implementing a sentiment analysis pipeline for the IMDB dataset using BERT and RoBERTa models, integrated with an MLOps pipeline (GitHub CI/CD, Docker, AWS, TensorFlow), and deploying a web interface using Gradio is a comprehensive task. Below is a step-by-step guide to achieve this. Each step includes code snippets and explanations to guide you through the process.
1. Setup and Prerequisites
Before you begin, ensure you have the following installed:
- Python 3.7+
- TensorFlow
- Transformers (by Hugging Face)
- Gradio
- Docker
- Git
- AWS CLI
You can install the Python dependencies using
pip:pip install tensorflow transformers gradio datasets2. Data Preparation
We'll use the IMDB dataset available through Hugging Face's
datasetslibrary.# data_preparation.py from datasets import load_dataset def load_imdb_dataset(): dataset = load_dataset('imdb') train_data = dataset['train'] test_data = dataset['test'] return train_data, test_data if __name__ == "__main__": train, test = load_imdb_dataset() print(f"Training samples: {len(train)}, Testing samples: {len(test)}")Run the script:
python data_preparation.py3. Tokenization and Embedding
We'll use BERT and RoBERTa tokenizers from Hugging Face.
# tokenization.py from transformers import BertTokenizer, RobertaTokenizer from datasets import load_dataset def tokenize_data(tokenizer, data, max_length=256): return tokenizer( data['text'], padding='max_length', truncation=True, max_length=max_length ) def prepare_tokenized_datasets(): train_data, test_data = load_dataset('imdb')['train'], load_dataset('imdb')['test'] bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') roberta_tokenizer = RobertaTokenizer.from_pretrained('roberta-base') tokenized_train_bert = tokenize_data(bert_tokenizer, train_data) tokenized_test_bert = tokenize_data(bert_tokenizer, test_data) tokenized_train_roberta = tokenize_data(roberta_tokenizer, train_data) tokenized_test_roberta = tokenize_data(roberta_tokenizer, test_data) return { 'bert': { 'train': tokenized_train_bert, 'test': tokenized_test_bert }, 'roberta': { 'train': tokenized_train_roberta, 'test': tokenized_test_roberta } } if __name__ == "__main__": tokenized = prepare_tokenized_datasets() print("Tokenization complete.")Run the script:
python tokenization.py4. Model Training with TensorFlow
We'll create a TensorFlow model using BERT and RoBERTa for sentiment analysis.
# train.py import tensorflow as tf from transformers import TFBertModel, TFRobertaModel from datasets import load_dataset from transformers import BertTokenizer, RobertaTokenizer import numpy as np def load_and_tokenize(model_name='bert'): dataset = load_dataset('imdb') tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') if model_name == 'bert' else RobertaTokenizer.from_pretrained('roberta-base') def tokenize(batch): return tokenizer(batch['text'], padding='max_length', truncation=True, max_length=256) train = dataset['train'].map(tokenize, batched=True) test = dataset['test'].map(tokenize, batched=True) # Convert labels to numpy arrays train_labels = np.array(train['label']) test_labels = np.array(test['label']) # Convert to TensorFlow datasets train_dataset = tf.data.Dataset.from_tensor_slices(( dict(train), train_labels )).shuffle(10000).batch(32) test_dataset = tf.data.Dataset.from_tensor_slices(( dict(test), test_labels )).batch(32) return train_dataset, test_dataset, tokenizer def build_model(model_name='bert'): if model_name == 'bert': transformer = TFBertModel.from_pretrained('bert-base-uncased') else: transformer = TFRobertaModel.from_pretrained('roberta-base') input_ids = tf.keras.layers.Input(shape=(256,), dtype=tf.int32, name='input_ids') attention_mask = tf.keras.layers.Input(shape=(256,), dtype=tf.int32, name='attention_mask') transformer_outputs = transformer(input_ids, attention_mask=attention_mask) cls_token = transformer_outputs.last_hidden_state[:,0,:] # CLS token dropout = tf.keras.layers.Dropout(0.3)(cls_token) output = tf.keras.layers.Dense(1, activation='sigmoid')(dropout) model = tf.keras.Model(inputs=[input_ids, attention_mask], outputs=output) model.compile( optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5), loss='binary_crossentropy', metrics=['accuracy'] ) return model def train_and_save(model_name='bert', epochs=3): train_ds, test_ds, tokenizer = load_and_tokenize(model_name) model = build_model(model_name) model.fit(train_ds, epochs=epochs, validation_data=test_ds) model.save(f'saved_models/{model_name}_model') tokenizer.save_pretrained(f'saved_models/{model_name}_tokenizer') print(f"{model_name.capitalize()} model trained and saved.") if __name__ == "__main__": # Train BERT model train_and_save('bert') # Train RoBERTa model train_and_save('roberta')Run the script:
python train.pyNote: Training BERT/RoBERTa models can be resource-intensive. Consider using GPUs (e.g., through Google Colab or AWS EC2 instances with GPU support).
5. Building the MLOps Pipeline
We'll set up GitHub CI/CD, Dockerization, and deployment to AWS.
5.1. Dockerize the Application
Create a
Dockerfilefor the application.# Dockerfile FROM python:3.8-slim # Set environment variables ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 # Set work directory WORKDIR /app # Install dependencies COPY requirements.txt . RUN pip install --upgrade pip RUN pip install -r requirements.txt # Copy application code COPY . . # Expose port EXPOSE 7860 # Command to run the app CMD ["python", "app.py"]Create
requirements.txt:tensorflow transformers gradio datasets boto35.2. Create the Gradio Web Interface
# app.py import gradio as gr import tensorflow as tf from transformers import BertTokenizer, RobertaTokenizer import numpy as np # Load BERT model and tokenizer bert_model = tf.keras.models.load_model('saved_models/bert_model') bert_tokenizer = BertTokenizer.from_pretrained('saved_models/bert_tokenizer') # Load RoBERTa model and tokenizer roberta_model = tf.keras.models.load_model('saved_models/roberta_model') roberta_tokenizer = RobertaTokenizer.from_pretrained('saved_models/roberta_tokenizer') def predict_sentiment(text, model_choice): if model_choice == 'BERT': tokenizer = bert_tokenizer model = bert_model else: tokenizer = roberta_tokenizer model = roberta_model inputs = tokenizer( text, padding='max_length', truncation=True, max_length=256, return_tensors='tf' ) predictions = model(inputs) score = predictions.numpy()[0][0] sentiment = 'Positive' if score > 0.5 else 'Negative' return {"Sentiment": sentiment, "Confidence": float(score)} # Create Gradio Interface iface = gr.Interface( fn=predict_sentiment, inputs=[ gr.inputs.Textbox(lines=5, placeholder="Enter movie review here..."), gr.inputs.Radio(['BERT', 'RoBERTa'], label="Choose Model") ], outputs=[ gr.outputs.Label(num_top_classes=2) ], title="IMDB Sentiment Analysis", description="Classify movie reviews as Positive or Negative using BERT and RoBERTa models." ) if __name__ == "__main__": iface.launch(host="0.0.0.0", port=7860)5.3. GitHub Actions for CI/CD
Create a
.github/workflows/docker-image.ymlfile for building and pushing the Docker image.# .github/workflows/docker-image.yml name: Build and Deploy Docker Image on: push: branches: [ main ] jobs: build: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v2 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v1 - name: Log in to Docker Hub uses: docker/login-action@v2 with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} - name: Build and push Docker image uses: docker/build-push-action@v3 with: push: true tags: yourdockerhubusername/imdb-sentiment-analysis:latest - name: Deploy to AWS (using AWS CLI) env: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_REGION: 'us-east-1' run: | # Example using AWS ECS aws ecs update-service --cluster your-cluster --service your-service --force-new-deploymentNotes:
-
Secrets: In your GitHub repository, add the following secrets:
DOCKER_USERNAMEandDOCKER_PASSWORDfor Docker Hub.AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYfor AWS permissions.
-
Deployment: The above example assumes you're using AWS ECS. Adjust the deployment step based on your AWS setup (EKS, EC2, etc.).
5.4. AWS Setup
Ensure you have an AWS account and set up the necessary services:
- ECR (Elastic Container Registry): To store Docker images.
- ECS (Elastic Container Service): To deploy Docker containers.
- IAM Roles: Assign appropriate permissions to allow GitHub Actions to deploy to ECS.
Detailed AWS setup is beyond this guide's scope, but AWS documentation provides comprehensive steps.
6. Testing the Web Interface Locally
Before deploying, test the Gradio app locally.
python app.pyOpen your browser and navigate to
http://localhost:7860to see the web interface.7. Deploying to AWS
Assuming you've set up ECS and pushed your Docker image to ECR or Docker Hub, your GitHub Actions workflow can handle deployment automatically on each push to the
mainbranch.Accessing the Deployed Application:
Once deployed, navigate to the AWS ECS service's public endpoint or the load balancer associated with your service to access the Gradio web interface.
8. Monitoring and Model Management
For model management and monitoring, consider integrating AWS services such as:
- CloudWatch: For logging and monitoring application metrics.
- SageMaker: For more advanced model deployment and management features.
Implement automated scaling, health checks, and rollback strategies within your ECS or other deployment services to ensure high availability and reliability.
9. Optional Enhancements
- Containerization of Training: Use separate Docker images or services for training models.
- Model Versioning: Implement model versioning to track different model iterations.
- Continuous Evaluation: Set up pipelines to regularly evaluate model performance on new data.
- Security: Implement authentication and authorization for your web interface.
Conclusion
This guide provides a comprehensive overview of building a sentiment analysis pipeline using BERT and RoBERTa with MLOps best practices. Each component—from data preparation to deployment—can be further expanded based on specific requirements and infrastructure. Leveraging cloud services and automation tools ensures scalability, maintainability, and reliability of your machine learning applications.