Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro-preview-05-06· GoogleHugging Face LMArena Human Preference 140K

how can I use transformer-based models on a dataframe with queries and a list of…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with o3-mini’s answer.
  1. User

    how can I use transformer-based models on a dataframe with queries and a list of categories

  2. Gemini Assistant

    Okay, let's walk through how to use transformer-based models (like BERT, RoBERTa, DistilBERT, etc.) for categorizing user queries when you have a Pandas DataFrame with query_text and category_label columns.

    We'll primarily use the Hugging Face transformers library, which is the standard for this.

    Core Steps:

    1. Prepare Your Data:
      • Load DataFrame.
      • Encode category labels into numerical format.
      • Split data into training and validation sets.
    2. Choose a Pre-trained Model and Tokenizer:
      • Select a suitable transformer model (e.g., distilbert-base-uncased for a good balance of speed and accuracy).
      • Load its corresponding tokenizer.
    3. Tokenize Your Text Data:
      • Convert your query text into the numerical input format the transformer model expects.
    4. Create Datasets:
      • Wrap your tokenized data and labels into PyTorch Dataset objects (or Hugging Face Dataset objects).
    5. Set Up the Trainer:
      • Define TrainingArguments.
      • Initialize the Trainer class.
    6. Train (Fine-tune) the Model:
      • Run the training process.
    7. Evaluate the Model:
      • Assess performance on the validation set.
    8. Make Predictions on New Queries:
      • Use the fine-tuned model to categorize new, unseen queries.

    Let's break down each step with conceptual code examples:

    Assumptions:

    • You have a Pandas DataFrame, e.g., df, with columns named query_text and category_label.
    • You have Python, PyTorch, and the Hugging Face transformers and datasets libraries installed.
      pip install pandas torch transformers datasets scikit-learn
      

    Step 1: Prepare Your Data

    import pandas as pd
    from sklearn.model_selection import train_test_split
    from sklearn.preprocessing import LabelEncoder
    from datasets import Dataset, DatasetDict
    
    # Sample DataFrame (replace with your actual data loading)
    data = {
        'query_text': [
            "what is our current revenue ytd?",
            "how many open support tickets are there?",
            "explain the new expense policy",
            "projected sales for next quarter",
            "current cash balance",
            "status of invoice #12345",
            "define accrual accounting",
            "variance between actual and budget for R&D"
        ],
        'category_label': [
            "Financial Reporting",
            "Operational Inquiry",
            "Policy Question",
            "Forecasting",
            "Financial Reporting",
            "Transactional Inquiry",
            "Definition Request",
            "Variance Analysis"
        ]
    }
    df = pd.DataFrame(data)
    
    # --- Encode Category Labels ---
    label_encoder = LabelEncoder()
    df['encoded_label'] = label_encoder.fit_transform(df['category_label'])
    num_labels = len(label_encoder.classes_)
    print(f"Number of unique labels: {num_labels}")
    print(f"Label mapping: {dict(zip(label_encoder.classes_, label_encoder.transform(label_encoder.classes_)))}")
    
    # --- Split Data ---
    # Stratify ensures similar class distribution in train and val sets
    train_df, val_df = train_test_split(
        df,
        test_size=0.2, # 20% for validation
        random_state=42,
        stratify=df['encoded_label'] # Important for classification
    )
    
    # Convert Pandas DataFrames to Hugging Face Dataset objects
    # The 'label' column is expected by many Hugging Face trainers for classification
    train_dataset_hf = Dataset.from_pandas(train_df.rename(columns={'encoded_label': 'label'}))
    val_dataset_hf = Dataset.from_pandas(val_df.rename(columns={'encoded_label': 'label'}))
    
    # Optional: Create a DatasetDict (useful for organizing)
    dataset_dict = DatasetDict({
        'train': train_dataset_hf,
        'validation': val_dataset_hf
    })
    
    print(dataset_dict)
    

    Step 2: Choose Pre-trained Model and Tokenizer

    from transformers import AutoTokenizer, AutoModelForSequenceClassification
    
    model_name = "distilbert-base-uncased" # A good starting point
    # model_name = "bert-base-uncased"
    # model_name = "roberta-base"
    
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=num_labels)
    
    • AutoTokenizer: Automatically selects the correct tokenizer for the model_name.
    • AutoModelForSequenceClassification: Loads the pre-trained model with a classification head on top, configured for num_labels output classes.

    Step 3: Tokenize Your Text Data

    def tokenize_function(examples):
        # Tokenize the 'query_text' column.
        # padding=True will pad to the longest sequence in the batch.
        # truncation=True will truncate sequences longer than the model's max input length.
        return tokenizer(examples["query_text"], padding="max_length", truncation=True)
    
    tokenized_datasets = dataset_dict.map(tokenize_function, batched=True)
    
    # Remove original text columns as they are not needed by the model
    tokenized_datasets = tokenized_datasets.remove_columns(["query_text", "category_label", "__index_level_0__"]) # remove original index if present
    tokenized_datasets.set_format("torch") # Ensure tensors are PyTorch tensors
    
    print(tokenized_datasets['train'][0]) # See what a tokenized example looks like
    

    Step 4: Create Datasets (Already done with Hugging Face Dataset in Step 1 & 3)

    The tokenized_datasets object is now ready for the Trainer.

    Step 5: Set Up the Trainer

    from transformers import TrainingArguments, Trainer
    import numpy as np
    from sklearn.metrics import accuracy_score, f1_score
    
    def compute_metrics(eval_pred):
        logits, labels = eval_pred
        predictions = np.argmax(logits, axis=-1)
        acc = accuracy_score(labels, predictions)
        f1 = f1_score(labels, predictions, average='weighted') # Use 'weighted' for imbalanced classes
        return {"accuracy": acc, "f1": f1}
    
    training_args = TrainingArguments(
        output_dir="./results",          # Directory to save model checkpoints and logs
        num_train_epochs=3,              # Total number of training epochs
        per_device_train_batch_size=4,   # Batch size for training (adjust based on GPU memory)
        per_device_eval_batch_size=8,    # Batch size for evaluation
        warmup_steps=100,                # Number of warmup steps for learning rate scheduler
        weight_decay=0.01,               # Strength of weight decay
        logging_dir="./logs",            # Directory for storing logs
        logging_steps=5,                # Log metrics every X updates steps
        evaluation_strategy="epoch",     # Evaluate at the end of each epoch
        save_strategy="epoch",           # Save checkpoint at the end of each epoch
        load_best_model_at_end=True,     # Load the best model (by validation loss) at the end
        metric_for_best_model="f1",      # Use f1-score to determine the best model
        # report_to="wandb" # Optional: for Weights & Biases logging
    )
    
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_datasets["train"],
        eval_dataset=tokenized_datasets["validation"],
        tokenizer=tokenizer, # Pass the tokenizer for padding consistency
        compute_metrics=compute_metrics
    )
    

    Step 6: Train (Fine-tune) the Model

    trainer.train()
    

    This will start the fine-tuning process. If you have a GPU, Hugging Face Trainer will automatically use it.

    Step 7: Evaluate the Model

    The Trainer automatically evaluates on the validation set during training if evaluation_strategy is set. You can also explicitly evaluate:

    eval_results = trainer.evaluate()
    print(f"Evaluation results: {eval_results}")
    

    Step 8: Make Predictions on New Queries

    # Save the fine-tuned model and tokenizer
    model_save_path = "./fine_tuned_query_classifier"
    trainer.save_model(model_save_path)
    tokenizer.save_pretrained(model_save_path)
    
    # Load the fine-tuned model and tokenizer later for inference
    loaded_model = AutoModelForSequenceClassification.from_pretrained(model_save_path)
    loaded_tokenizer = AutoTokenizer.from_pretrained(model_save_path)
    
    # Make sure model is in evaluation mode and on the correct device
    import torch
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    loaded_model.to(device)
    loaded_model.eval()
    
    
    def predict_category(query_text):
        inputs = loaded_tokenizer(query_text, return_tensors="pt", padding=True, truncation=True)
        inputs = {k: v.to(device) for k, v in inputs.items()} # Move inputs to the same device as model
    
        with torch.no_grad(): # Disable gradient calculations for inference
            logits = loaded_model(**inputs).logits
    
        predicted_class_id = torch.argmax(logits, dim=-1).item()
        predicted_category = label_encoder.inverse_transform([predicted_class_id])[0]
        
        # Optional: Get probabilities
        probabilities = torch.softmax(logits, dim=-1)[0].cpu().numpy()
        confidence = probabilities[predicted_class_id]
        
        return predicted_category, confidence, dict(zip(label_encoder.classes_, probabilities))
    
    
    # Example usage:
    new_query1 = "What was our profit margin last year?"
    category1, confidence1, all_probs1 = predict_category(new_query1)
    print(f"Query: '{new_query1}' -> Predicted Category: '{category1}' (Confidence: {confidence1:.4f})")
    # print(f"All probabilities: {all_probs1}")
    
    new_query2 = "how do i submit a PO?"
    category2, confidence2, _ = predict_category(new_query2)
    print(f"Query: '{new_query2}' -> Predicted Category: '{category2}' (Confidence: {confidence2:.4f})")
    

    Key Considerations:

    • Computational Resources: Fine-tuning transformers, even smaller ones like DistilBERT, can be computationally intensive. A GPU is highly recommended for reasonable training times. If you don't have one, consider using Google Colab (which offers free GPU access) or cloud ML platforms.
    • Hyperparameter Tuning: The TrainingArguments and model choice can significantly impact performance. Experiment with learning rates, batch sizes, number of epochs, and different pre-trained models.
    • Dataset Size: Transformers generally benefit from larger datasets. If your labeled dataset is very small, you might see limited improvement from fine-tuning, or you might need techniques like few-shot learning or data augmentation.
    • Imbalanced Classes: If some query categories are much more frequent than others, this can bias the model. The f1_score with average='weighted' in compute_metrics helps, and you might consider techniques like class weighting in the loss function (can be set in Trainer) or over/undersampling.
    • Error Analysis: After training, examine the queries that your model misclassifies. This can give you insights into where the model struggles and how you might improve your data or model.

    This comprehensive guide should get you started with using transformer models for query categorization from your DataFrame.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt