Skip to content
The Chat Archive
qwen3-30b-a3b· Alibaba (Qwen) LMArena Human Preference 140K

Write code for document classification in most effective way

  1. User

    Write code for document classification in most effective way

  2. Assistant

    Here's an effective and scalable approach to document classification using Python, combining traditional machine learning and modern deep learning techniques. The code is optimized for performance, readability, and flexibility.


    🧠 Key Principles for Effective Document Classification

    1. Efficient Preprocessing: Clean text, remove noise, and normalize.
    2. Feature Extraction: Use TF-IDF or pre-trained embeddings (e.g., BERT).
    3. Model Selection: Balance accuracy and computational cost.
    4. Evaluation: Use metrics like accuracy, F1-score, and confusion matrices.
    5. Scalability: Leverage pipelines and libraries like scikit-learn, TensorFlow, or Hugging Face.

    ✅ 1. Basic Pipeline with Scikit-Learn (Traditional ML)

    import numpy as np
    import pandas as pd
    from sklearn.datasets import fetch_20newsgroups
    from sklearn.model_selection import train_test_split
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.naive_bayes import MultinomialNB
    from sklearn.pipeline import Pipeline
    from sklearn.metrics import classification_report, accuracy_score
    
    # Load dataset (example: 20 Newsgroups)
    categories = ['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.space']
    newsgroups = fetch_20newsgroups(subset='all', categories=categories)
    X, y = newsgroups.data, newsgroups.target
    
    # Split data
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
    
    # Create a pipeline: TF-IDF + Classifier
    model = Pipeline([
        ('tfidf', TfidfVectorizer(stop_words='english', max_features=10000)),
        ('clf', MultinomialNB())  # Can replace with SVM, Random Forest, etc.
    ])
    
    # Train and evaluate
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    
    print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
    print(classification_report(y_test, y_pred, target_names=newsgroups.target_names))
    

    🚀 2. Advanced Approach with Transformers (Hugging Face + PyTorch)

    For state-of-the-art accuracy, use pre-trained models like BERT:

    from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
    from sklearn.model_selection import train_test_split
    import torch
    
    # Load dataset (same as before)
    # X, y = ... (from previous example)
    
    # Split data
    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
    
    # Load pre-trained BERT model and tokenizer
    model_name = "bert-base-uncased"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=len(categories))
    
    # Tokenize data
    train_encodings = tokenizer(X_train, truncation=True, padding=True, max_length=512)
    val_encodings = tokenizer(X_val, truncation=True, padding=True, max_length=512)
    
    # Convert to PyTorch Dataset
    class NewsDataset(torch.utils.data.Dataset):
        def __init__(self, encodings, labels):
            self.encodings = encodings
            self.labels = labels
    
        def __getitem__(self, idx):
            item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
            item['labels'] = torch.tensor(self.labels[idx])
            return item
    
        def __len__(self):
            return len(self.labels)
    
    train_dataset = NewsDataset(train_encodings, y_train)
    val_dataset = NewsDataset(val_encodings, y_val)
    
    # Training arguments
    training_args = TrainingArguments(
        output_dir="./results",
        num_train_epochs=3,
        per_device_train_batch_size=16,
        per_device_eval_batch_size=16,
        warmup_steps=500,
        weight_decay=0.01,
        logging_dir="./logs",
        logging_steps=10,
        evaluation_strategy="epoch"
    )
    
    # Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=val_dataset
    )
    
    # Train and evaluate
    trainer.train()
    results = trainer.evaluate()
    print("Validation Accuracy:", results["eval_accuracy"])
    

    🔍 3. Tips for Optimization

    • Hyperparameter Tuning: Use GridSearchCV or Optuna for ML models.
    • Efficient Vectorization: Use HashingVectorizer for memory efficiency.
    • Distributed Training: Use Dask or Ray for large datasets.
    • Model Compression: Prune BERT for faster inference (e.g., transformers' prune API).

    📊 Evaluation Metrics

    Use classification_report for detailed metrics:

    print(classification_report(y_test, y_pred, target_names=newsgroups.target_names))
    

    🧩 When to Choose Which Approach

    ApproachUse CaseProsCons
    TF-IDF + Naive BayesSmall datasets, fast prototypingLightweight, simpleLess accurate for complex texts
    BERT/TransformerHigh accuracy requiredState-of-the-art performanceComputationally heavy

    📦 Dependencies

    Install required libraries:

    pip install scikit-learn pandas numpy torch transformers
    

    This approach balances efficiency and accuracy, leveraging both classical and modern techniques. For production, consider model serving (e.g., FastAPI + TorchScript) or cloud deployment (e.g., AWS SageMaker).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.