Skip to content
TheChatArchiveThe Chat Archive
o3-mini· OpenAIHugging Face LMArena Human Preference 140K

how can I use transformer-based models on a dataframe with queries and a list of…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-pro-preview-05-06’s answer.
  1. User

    how can I use transformer-based models on a dataframe with queries and a list of categories

  2. Assistant

    Below is a step-by-step guide, including sample code, on how to use transformer-based models (e.g., BERT) for classifying user queries based on a DataFrame. In this example, we'll use Hugging Face's Transformers and Datasets libraries to fine-tune a BERT-based classifier.


    1. Prepare Your Data

    Assume you have a DataFrame (df) with at least two columns:

    • "query": the text of the user query.
    • "category": the label/category for the query.

    First, install the required libraries (if you haven’t already):

    pip install transformers datasets scikit-learn pandas torch
    

    2. Map Categories to Integers

    Transformers expect numerical labels. Create a mapping between your category names and integer IDs.

    import pandas as pd
    
    # Example DataFrame
    data = {
        "query": [
            "What is our projected revenue growth?",
            "How do I calculate NPV?",
            "Show me last quarter's expenses.",
            "What are the compliance requirements for tax filings?"
        ],
        "category": ["Forecasting", "Calculation", "Data Retrieval", "Compliance"]
    }
    df = pd.DataFrame(data)
    
    # Create mapping for categories
    categories = df["category"].unique()
    category2id = {cat: i for i, cat in enumerate(categories)}
    id2category = {i: cat for cat, i in category2id.items()}
    
    # Map the category column to ids
    df["label"] = df["category"].map(category2id)
    

    3. Split the Data and Convert to Hugging Face Dataset

    Use scikit-learn to split the DataFrame into training and validation sets and convert them into the Hugging Face Dataset format.

    from sklearn.model_selection import train_test_split
    from datasets import Dataset
    
    # Split the DataFrame
    train_df, val_df = train_test_split(df, test_size=0.2, random_state=42, stratify=df["label"])
    
    # Convert pandas DataFrame to Hugging Face datasets
    train_dataset = Dataset.from_pandas(train_df)
    val_dataset = Dataset.from_pandas(val_df)
    

    4. Tokenize the Data

    Load a pretrained tokenizer (e.g., bert-base-uncased) and tokenize your queries.

    from transformers import AutoTokenizer
    
    model_name = "bert-base-uncased"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    
    # Define a tokenization function
    def tokenize_function(examples):
        return tokenizer(examples["query"], truncation=True, padding="max_length", max_length=128)
    
    # Apply the tokenizer to the dataset
    train_dataset = train_dataset.map(tokenize_function, batched=True)
    val_dataset = val_dataset.map(tokenize_function, batched=True)
    

    Make sure to remove columns that are not used by the model during training:

    train_dataset = train_dataset.remove_columns(["query", "category", "__index_level_0__"])
    val_dataset = val_dataset.remove_columns(["query", "category", "__index_level_0__"])
    

    5. Load and Prepare the Model

    Load a pre-trained transformer model for sequence classification and specify the number of output labels.

    from transformers import AutoModelForSequenceClassification
    
    num_labels = len(category2id)  # Number of unique categories
    
    model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=num_labels)
    

    6. Set Up Training Arguments and the Trainer

    Define training arguments and instantiate the Trainer from Hugging Face.

    from transformers import TrainingArguments, Trainer
    import numpy as np
    import evaluate
    
    # Define a metric (accuracy for simplicity)
    accuracy_metric = evaluate.load("accuracy")
    
    def compute_metrics(eval_pred):
        logits, labels = eval_pred
        predictions = np.argmax(logits, axis=-1)
        return accuracy_metric.compute(predictions=predictions, references=labels)
    
    training_args = TrainingArguments(
        output_dir="./results",          # output directory
        evaluation_strategy="epoch",     # evaluation strategy to adopt during training
        learning_rate=2e-5,
        per_device_train_batch_size=16,
        per_device_eval_batch_size=16,
        num_train_epochs=3,              # total training epochs
        weight_decay=0.01,               # strength of weight decay
    )
    
    # Create the trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=val_dataset,
        compute_metrics=compute_metrics,
    )
    

    7. Train the Model

    Kick off training using the trainer.train() method.

    trainer.train()
    

    After training, you can evaluate your model:

    eval_results = trainer.evaluate()
    print(eval_results)
    

    8. Categorize New User Queries

    Once the model is trained, you can predict the category for new queries. Here's how to do it:

    # Define a function to predict category
    def predict_category(query):
        inputs = tokenizer(query, return_tensors="pt", truncation=True, padding="max_length", max_length=128)
        outputs = model(**inputs)
        logits = outputs.logits
        predicted_class_id = logits.argmax().item()
        return id2category[predicted_class_id]
    
    # Example usage:
    new_query = "Could you forecast our quarterly earnings?"
    predicted_category = predict_category(new_query)
    print("Predicted Category:", predicted_category)
    

    Final Thoughts

    • Fine-tuning: The transformer model is fine-tuned on your labeled queries, meaning it has adapted to the nuances of your data.
    • Performance: If you have a large dataset and need further performance improvements, consider hyperparameter tuning or using larger transformer models.
    • Deployment: Once satisfied with performance, you can deploy this as an API using frameworks such as FastAPI or Flask.

    This workflow enables you to leverage a transformer-based model on a DataFrame containing queries and their categories, efficiently categorizing new queries based on the learned patterns. Feel free to ask if you need further clarification or help with any step!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.