Write code for document classification in most effective way
- User
Write code for document classification in most effective way
- Assistant
Here's an effective and scalable approach to document classification using Python, combining traditional machine learning and modern deep learning techniques. The code is optimized for performance, readability, and flexibility.
🧠 Key Principles for Effective Document Classification
- Efficient Preprocessing: Clean text, remove noise, and normalize.
- Feature Extraction: Use TF-IDF or pre-trained embeddings (e.g., BERT).
- Model Selection: Balance accuracy and computational cost.
- Evaluation: Use metrics like accuracy, F1-score, and confusion matrices.
- Scalability: Leverage pipelines and libraries like
scikit-learn,TensorFlow, orHugging Face.
✅ 1. Basic Pipeline with Scikit-Learn (Traditional ML)
import numpy as np import pandas as pd from sklearn.datasets import fetch_20newsgroups from sklearn.model_selection import train_test_split from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.naive_bayes import MultinomialNB from sklearn.pipeline import Pipeline from sklearn.metrics import classification_report, accuracy_score # Load dataset (example: 20 Newsgroups) categories = ['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.space'] newsgroups = fetch_20newsgroups(subset='all', categories=categories) X, y = newsgroups.data, newsgroups.target # Split data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) # Create a pipeline: TF-IDF + Classifier model = Pipeline([ ('tfidf', TfidfVectorizer(stop_words='english', max_features=10000)), ('clf', MultinomialNB()) # Can replace with SVM, Random Forest, etc. ]) # Train and evaluate model.fit(X_train, y_train) y_pred = model.predict(X_test) print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}") print(classification_report(y_test, y_pred, target_names=newsgroups.target_names))
🚀 2. Advanced Approach with Transformers (Hugging Face + PyTorch)
For state-of-the-art accuracy, use pre-trained models like BERT:
from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments from sklearn.model_selection import train_test_split import torch # Load dataset (same as before) # X, y = ... (from previous example) # Split data X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) # Load pre-trained BERT model and tokenizer model_name = "bert-base-uncased" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=len(categories)) # Tokenize data train_encodings = tokenizer(X_train, truncation=True, padding=True, max_length=512) val_encodings = tokenizer(X_val, truncation=True, padding=True, max_length=512) # Convert to PyTorch Dataset class NewsDataset(torch.utils.data.Dataset): def __init__(self, encodings, labels): self.encodings = encodings self.labels = labels def __getitem__(self, idx): item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()} item['labels'] = torch.tensor(self.labels[idx]) return item def __len__(self): return len(self.labels) train_dataset = NewsDataset(train_encodings, y_train) val_dataset = NewsDataset(val_encodings, y_val) # Training arguments training_args = TrainingArguments( output_dir="./results", num_train_epochs=3, per_device_train_batch_size=16, per_device_eval_batch_size=16, warmup_steps=500, weight_decay=0.01, logging_dir="./logs", logging_steps=10, evaluation_strategy="epoch" ) # Trainer trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=val_dataset ) # Train and evaluate trainer.train() results = trainer.evaluate() print("Validation Accuracy:", results["eval_accuracy"])
🔍 3. Tips for Optimization
- Hyperparameter Tuning: Use
GridSearchCVorOptunafor ML models. - Efficient Vectorization: Use
HashingVectorizerfor memory efficiency. - Distributed Training: Use
DaskorRayfor large datasets. - Model Compression: Prune BERT for faster inference (e.g.,
transformers'pruneAPI).
📊 Evaluation Metrics
Use
classification_reportfor detailed metrics:print(classification_report(y_test, y_pred, target_names=newsgroups.target_names))
🧩 When to Choose Which Approach
Approach Use Case Pros Cons TF-IDF + Naive Bayes Small datasets, fast prototyping Lightweight, simple Less accurate for complex texts BERT/Transformer High accuracy required State-of-the-art performance Computationally heavy
📦 Dependencies
Install required libraries:
pip install scikit-learn pandas numpy torch transformers
This approach balances efficiency and accuracy, leveraging both classical and modern techniques. For production, consider model serving (e.g.,
FastAPI+TorchScript) or cloud deployment (e.g., AWS SageMaker).
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.