another way to compute bleu4 import sacrebleu from nltk.tokenize import wordtokenize…
- User
another way to compute bleu4 import sacrebleu from nltk.tokenize import word_tokenize
Load reference answers from tokenized_test_dataset
reference_answers_list = [tokenized_test_dataset[i]['query'] for i in range(500)]
Assuming df_generated_questions is already defined
Extract generated questions from the DataFrame
generated_questions_list = df_generated_questions['generated_question'].tolist()
Tokenize reference answers and generated questions using nltk
tokenized_reference_answers = [word_tokenize(reference) for reference in reference_answers_list] tokenized_generated_questions = [word_tokenize(generated) for generated in generated_questions_list]
Compute BLEU-4 score using sacrebleu
bleu_score = sacrebleu.corpus_bleu(tokenized_generated_questions, [tokenized_reference_answers])
Print the BLEU-4 score
print(f'BLEU-4 Score: {bleu_score.score * 100:.2f}')
- Assistant
In your code snippet, you're attempting to compute the BLEU-4 score using the
sacrebleupackage after tokenizing your reference answers and generated questions withnltk. However, you've made a small mistake in thesacrebleu.corpus_bleufunction call. Thesacrebleupackage expects the reference to be a list of lists of strings, where each sublist corresponds to a set of reference translations for a single sentence. You need to make sure you're providing references in the format expected bysacrebleu.Here’s an alternative way to compute the BLEU-4 score using the
sacrebleupackage, but with the correct formatting of references:import sacrebleu # Assuming the lists 'reference_answers_list' and 'generated_questions_list' are obtained as shown previously # NOTE: 'sacrebleu' expects a list of reference translations for each sentence. Here, # we're assuming 'reference_answers_list' represents a single list of references (one per sentence). # Thus, we wrap each answer inside another list to simulate multiple references per sentence. refs = [[ref] for ref in reference_answers_list] # Assume 'generated_questions_list' represents hypotheses (one per sentence) hyps = generated_questions_list # Compute BLEU score bleu = sacrebleu.corpus_bleu(hyps, refs) # Print BLEU score print(f'BLEU-4 Score: {bleu.score:.2f}')By wrapping each reference within another list, we create the expected structure for
sacrebleu. It should be noted thatsacrebleuautomatically tokenizes the text internally, so you don’t necessarily need to tokenize your sentences withnltkbefore passing them tosacrebleu; you can pass the raw, untokenized sentences directly.Also,
sacrebleureturns the BLEU score as a percentage, so you don’t need to multiply by 100. If you prefer to use tokenized text, you should disable the internal tokenizer by setting thetokenizer='none'parameter in thecorpus_bleufunction. - User
this # Function to generate a question using a given model and context def generate_question(model, tokenizer, context, max_length=128, device="cuda" if torch.cuda.is_available() else "cpu"): # Move the model to the specified device model.to(device)
# Tokenize and move tensors to the appropriate device inputs = tokenizer(context, return_tensors="pt", max_length=max_length, padding="max_length", truncation=True) inputs = {key: value.to(device) for key, value in inputs.items()} # Generate question using the model generated_question_ids = model.generate(**inputs) generated_question = tokenizer.decode(generated_question_ids[0], skip_special_tokens=True) return generated_questionho to compute bleu4
- Assistant
To compute the BLEU-4 score for a set of generated questions against reference questions, you would typically follow these steps:
- Generate questions using the
generate_questionfunction for each context. - Collect all generated questions and their corresponding reference questions.
- Use a BLEU score computation function to compare the generated text against references.
Here's how you can compute the BLEU-4 score for a set of generated questions:
import sacrebleu # Let's assume you have a list of contexts and their corresponding reference questions contexts = [...] reference_questions = [...] # Use your function to generate questions for each context generated_questions = [generate_question(model, tokenizer, context) for context in contexts] # Wrap reference questions for BLEU evaluation (assuming one reference per context) refs = [[ref] for ref in reference_questions] # Each reference wrapped in a list # Compute BLEU score using sacrebleu bleu = sacrebleu.corpus_bleu(generated_questions, refs) # Print BLEU score print(f'BLEU-4 Score: {bleu.score:.2f}')Please remember that BLEU scores can be a useful metric for evaluating the quality of generated text, but there are limitations to consider. BLEU scores primarily focus on n-gram precision and do not account for semantic accuracy, relevance, fluency, or factual correctness. It is often best to use BLEU scores as part of a suite of evaluation metrics rather than in isolation.
- Generate questions using the