gpt-4.1-2025-04-14· OpenAI LMArena Human Preference 140K
Write code for document classification in most effective way
- User
Write code for document classification in most effective way
- Assistant
Certainly! Document classification is a common NLP task. The most effective way as of 2024 is to use transformer-based models (like BERT, RoBERTa, or DistilBERT) with transfer learning. Below is a complete, practical example using the Hugging Face
transformerslibrary and PyTorch. This code can be adapted for your dataset.
1. Install Required Libraries
pip install torch transformers datasets scikit-learn
2. Prepare Your Data
Assume you have a CSV file:
data.csvwith columns:textandlabel.
3. Code: Document Classification with Transformers
import pandas as pd from datasets import Dataset, ClassLabel from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer from sklearn.model_selection import train_test_split import torch # 1. Load your data df = pd.read_csv('data.csv') train_df, test_df = train_test_split(df, test_size=0.2, stratify=df['label'], random_state=42) # 2. Convert to HuggingFace Dataset train_dataset = Dataset.from_pandas(train_df) test_dataset = Dataset.from_pandas(test_df) # 3. Encode labels labels = sorted(df['label'].unique()) label2id = {label: i for i, label in enumerate(labels)} id2label = {i: label for label, i in label2id.items()} def encode_labels(example): example['label'] = label2id[example['label']] return example train_dataset = train_dataset.map(encode_labels) test_dataset = test_dataset.map(encode_labels) # 4. Tokenization model_name = "distilbert-base-uncased" # Or "bert-base-uncased", etc. tokenizer = AutoTokenizer.from_pretrained(model_name) def tokenize_function(example): return tokenizer( example["text"], padding="max_length", truncation=True, max_length=256, ) train_dataset = train_dataset.map(tokenize_function, batched=True) test_dataset = test_dataset.map(tokenize_function, batched=True) # 5. Set format for PyTorch train_dataset.set_format(type='torch', columns=['input_ids', 'attention_mask', 'label']) test_dataset.set_format(type='torch', columns=['input_ids', 'attention_mask', 'label']) # 6. Load model model = AutoModelForSequenceClassification.from_pretrained( model_name, num_labels=len(labels), id2label=id2label, label2id=label2id ) # 7. Training arguments training_args = TrainingArguments( output_dir="./results", evaluation_strategy="epoch", save_strategy="epoch", learning_rate=2e-5, per_device_train_batch_size=16, per_device_eval_batch_size=16, num_train_epochs=3, weight_decay=0.01, load_best_model_at_end=True, metric_for_best_model="accuracy", ) # 8. Metrics import numpy as np from sklearn.metrics import accuracy_score, f1_score def compute_metrics(eval_pred): logits, labels = eval_pred preds = np.argmax(logits, axis=1) return { "accuracy": accuracy_score(labels, preds), "f1": f1_score(labels, preds, average="weighted"), } # 9. Trainer trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=test_dataset, compute_metrics=compute_metrics, ) # 10. Train trainer.train() # 11. Evaluate results = trainer.evaluate() print(results)
Tips for Effectiveness
- Use a pretrained transformer (BERT, RoBERTa, etc.) for best results.
- Fine-tune on your own data.
- Use GPU for training (set up CUDA if available).
- For large datasets, consider using the
datasetslibrary for efficient data handling.
This approach is state-of-the-art for most document classification tasks as of 2024.
If you have a specific dataset or need a different language/model, let me know!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.