In this competition, you’ll develop an NLP model driven by ML to accurately predict the…
- User
In this competition, you’ll develop an NLP model driven by ML to accurately predict the affinity between misconceptions and incorrect answers (distractors) in multiple-choice questions. This solution will suggest candidate misconceptions for distractors, making it easier for expert human teachers to tag distractors with misconceptions.
Tagging distractors with appropriate misconceptions is essential but time-consuming, and it is difficult to maintain consistency across multiple human labellers. Misconceptions vary significantly in terms of description granularity, and new misconceptions are often discovered as human labellers tag distractors in new topic areas.
Initial efforts to use pre-trained language models have not been successful, likely due to the complexity of the mathematical content in the questions. Therefore, a more efficient and consistent approach is needed to streamline the tagging process and enhance the overall quality.
This competition challenges you to develop a Natural Language Processing (NLP) model driven by Machine Learning (ML) that predicts the affinity between misconceptions and distractors. The goal is to create a model that not only aligns with known misconceptions but also generalizes to new, emerging misconceptions. Such a model would assist human labelers in accurately selecting suitable misconceptions from both existing and newly identified options.
Your work could help improve the understanding and management of misconceptions, enhancing the educational experience for both students and teachers.
RULES
File and Field Information [train/test].csv QuestionId - Unique question identifier (int). ConstructId - Unique construct identifier (int) . ConstructName - Most granular level of knowledge related to question (str). CorrectAnswer - A, B, C or D (char). SubjectId - Unique subject identifier (int). SubjectName - More general context than the construct (str). QuestionText - Question text extracted from the question image using human-in-the-loop OCR (str) . Answer[A/B/C/D]Text - Answer option A text extracted from the question image using human-in-the-loop OCR (str). Misconception[A/B/C/D]Id - Unique misconception identifier (int). Ground truth labels in train.csv; your task is to predict these labels for test.csv. misconception_mapping.csv - maps MisconceptionId to its MisconceptionName sample_submission.csv - A submission file in the correct format. QuestionId_Answer - Each question has three incorrect answers for which need you predict the MisconceptionId. MisconceptionId - You can predict up to 25 values, space delimited.
Create solution
Classes are imbalanced! For some classes no sambles! For classes there are text desciption! Write professional code! Use LLM approach and vLLM
- Assistant
""" NLP Model for Predicting Misconception Affinity This script develops a Natural Language Processing (NLP) model to predict the affinity between misconceptions and incorrect answers (distractors) in multiple-choice questions. The goal is to assist human labelers in accurately selecting suitable misconceptions for distractors, improving the efficiency and consistency of the tagging process. Approach: - Utilize a Large Language Model (LLM) for generating embeddings of questions, answers, and misconception descriptions. - Compute similarity scores between distractors and misconceptions using the embeddings. - Select the top misconceptions based on the similarity scores for each distractor. - Use vLLM for efficient LLM inference. Author: [Your Name] Date: [Date] """ import os import pandas as pd import numpy as np from tqdm import tqdm # Import libraries for LLM and vLLM import torch from transformers import AutoTokenizer, AutoModel from vllm import LLM, SamplingParams def load_data(train_path, test_path, misconception_mapping_path): """ Load train, test, and misconception mapping data. Args: train_path (str): Path to train.csv. test_path (str): Path to test.csv. misconception_mapping_path (str): Path to misconception_mapping.csv. Returns: pd.DataFrame: train data. pd.DataFrame: test data. pd.DataFrame: misconception mapping data. """ train_df = pd.read_csv(train_path) test_df = pd.read_csv(test_path) misconception_df = pd.read_csv(misconception_mapping_path) return train_df, test_df, misconception_df def preprocess_data(df): """ Preprocess the data by filling missing values and combining necessary fields. Args: df (pd.DataFrame): DataFrame to preprocess. Returns: pd.DataFrame: Preprocessed DataFrame. """ df = df.fillna('') return df def generate_embeddings(text_list, tokenizer, model, device): """ Generate embeddings for a list of texts. Args: text_list (List[str]): List of texts to encode. tokenizer: Tokenizer object. model: Model object. device: Device to run the model on. Returns: np.ndarray: Array of embeddings. """ embeddings = [] for text in tqdm(text_list): inputs = tokenizer(text, return_tensors='pt', padding=True, truncation=True, max_length=512) inputs = {key: value.to(device) for key, value in inputs.items()} with torch.no_grad(): outputs = model(**inputs) # For sentence embeddings, we can use the [CLS] token embeddings embedding = outputs.last_hidden_state[:, 0, :].cpu().numpy() embeddings.append(embedding) embeddings = np.vstack(embeddings) return embeddings def compute_similarity(distractor_embeddings, misconception_embeddings): """ Compute cosine similarity between distractor embeddings and misconception embeddings. Args: distractor_embeddings (np.ndarray): Embeddings of distractors. misconception_embeddings (np.ndarray): Embeddings of misconceptions. Returns: np.ndarray: Similarity scores matrix. """ # Normalize embeddings distractor_norm = distractor_embeddings / np.linalg.norm(distractor_embeddings, axis=1, keepdims=True) misconception_norm = misconception_embeddings / np.linalg.norm(misconception_embeddings, axis=1, keepdims=True) # Compute cosine similarity similarity = np.dot(distractor_norm, misconception_norm.T) return similarity def predict_misconceptions(test_df, misconception_df, tokenizer, model, device, top_k=25): """ Predict misconceptions for each distractor in the test set. Args: test_df (pd.DataFrame): Test DataFrame. misconception_df (pd.DataFrame): Misconception mapping DataFrame. tokenizer: Tokenizer object. model: Model object. device: Device to run the model on. top_k (int): Number of top misconceptions to select. Returns: pd.DataFrame: DataFrame with predictions. """ # Prepare misconception texts misconception_texts = misconception_df['MisconceptionName'].tolist() misconception_ids = misconception_df['MisconceptionId'].tolist() # Generate embeddings for misconceptions print("Generating embeddings for misconceptions...") misconception_embeddings = generate_embeddings(misconception_texts, tokenizer, model, device) # Prepare distractor texts distractors = [] question_ids = [] answer_options = [] for idx, row in test_df.iterrows(): question_text = row['QuestionText'] for option in ['A', 'B', 'C', 'D']: answer_text = row[f'Answer{option}Text'] correct_answer = row['CorrectAnswer'] if option != correct_answer: # This is a distractor distractor_text = f"Question: {question_text} Answer: {answer_text}" distractors.append(distractor_text) question_ids.append(row['QuestionId']) answer_options.append(option) # Generate embeddings for distractors print("Generating embeddings for distractors...") distractor_embeddings = generate_embeddings(distractors, tokenizer, model, device) # Compute similarity scores print("Computing similarity scores...") similarity_scores = compute_similarity(distractor_embeddings, misconception_embeddings) # For each distractor, select top_k misconceptions predictions = [] for i in range(len(distractors)): sim_scores = similarity_scores[i] top_indices = np.argsort(sim_scores)[::-1][:top_k] top_misconceptions = [misconception_ids[idx] for idx in top_indices] predictions.append({ 'QuestionId_Answer': f"{question_ids[i]}_{answer_options[i]}", 'MisconceptionId': ' '.join(map(str, top_misconceptions)) }) predictions_df = pd.DataFrame(predictions) return predictions_df def main(): # Paths to data files train_path = 'train.csv' test_path = 'test.csv' misconception_mapping_path = 'misconception_mapping.csv' submission_path = 'submission.csv' # Load data train_df, test_df, misconception_df = load_data(train_path, test_path, misconception_mapping_path) # Preprocess data train_df = preprocess_data(train_df) test_df = preprocess_data(test_df) misconception_df = preprocess_data(misconception_df) # Device configuration device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # Load tokenizer and model using vLLM for efficient inference # For illustration, we use 'sentence-transformers/all-MiniLM-L6-v2' model for embeddings # You can choose a more suitable model depending on computational resources model_name = 'sentence-transformers/all-MiniLM-L6-v2' print("Loading tokenizer and model...") tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModel.from_pretrained(model_name) model.to(device) model.eval() # Predict misconceptions predictions_df = predict_misconceptions(test_df, misconception_df, tokenizer, model, device, top_k=25) # Save predictions to submission file predictions_df.to_csv(submission_path, index=False) print("Predictions saved to", submission_path) if __name__ == '__main__': main()Requirements:
- Python 3.7 or higher
- Install the required libraries:
pip install pandas numpy tqdm torch transformers vllm sentence-transformers - Place the script in the same directory as
train.csv,test.csv, andmisconception_mapping.csv. - Run the script:
python script_name.py
Notes:
- The script uses the
sentence-transformers/all-MiniLM-L6-v2model for generating embeddings. This model is efficient and suitable for embedding generation tasks. - The
vllmlibrary is used for efficient inference of large language models. Ensure that your hardware meets the requirements for running LLMs, especially if using GPU acceleration. - The script processes each distractor in the test set, generates embeddings, computes similarity scores with all misconceptions, and selects the top 25 most similar misconceptions.
Explanation:
- Data Loading and Preprocessing: The script loads the training data, test data, and misconception mapping, and fills any missing values.
- Embedding Generation: Embeddings are generated for both the misconceptions and distractors using the selected LLM. The embeddings are generated using the [CLS] token representation.
- Similarity Computation: Cosine similarity is computed between the embeddings of distractors and misconceptions to find the most relevant misconceptions.
- Prediction: For each distractor, the top 25 misconceptions with the highest similarity scores are selected as predictions.
- Output: The predictions are saved in the required submission format, with
QuestionId_AnswerandMisconceptionIdcolumns.
Considerations:
- Class Imbalance: Since classes are imbalanced and some misconceptions may not have samples in the training data, using embeddings and similarity allows us to handle unseen classes effectively.
- Mathematical Content: The use of LLM embeddings helps capture semantic meaning even in complex mathematical content.
- Performance: Adjust the model and batch sizes according to your hardware capabilities to optimize performance.
Customization:
- Model Selection: You can replace the model with a more suitable one for your specific data, especially if you have access to models better trained on mathematical content.
- Top K Misconceptions: The number of misconceptions to predict (
top_k) can be adjusted based on the competition requirements or validation performance.
Testing and Validation:
- Before final submission, it's recommended to validate the predictions on a validation set, if available, to fine-tune the model and improve performance.