Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· openaiHugging Face Suzhen/CodeChat

another way to compute bleu4 import sacrebleu from nltk.tokenize import wordtokenize…

  1. User

    another way to compute bleu4 import sacrebleu from nltk.tokenize import word_tokenize

    Load reference answers from tokenized_test_dataset

    reference_answers_list = [tokenized_test_dataset[i]['query'] for i in range(500)]

    Assuming df_generated_questions is already defined

    Extract generated questions from the DataFrame

    generated_questions_list = df_generated_questions['generated_question'].tolist()

    Tokenize reference answers and generated questions using nltk

    tokenized_reference_answers = [word_tokenize(reference) for reference in reference_answers_list] tokenized_generated_questions = [word_tokenize(generated) for generated in generated_questions_list]

    Compute BLEU-4 score using sacrebleu

    bleu_score = sacrebleu.corpus_bleu(tokenized_generated_questions, [tokenized_reference_answers])

    Print the BLEU-4 score

    print(f'BLEU-4 Score: {bleu_score.score * 100:.2f}')

  2. Assistant

    In your code snippet, you're attempting to compute the BLEU-4 score using the sacrebleu package after tokenizing your reference answers and generated questions with nltk. However, you've made a small mistake in the sacrebleu.corpus_bleu function call. The sacrebleu package expects the reference to be a list of lists of strings, where each sublist corresponds to a set of reference translations for a single sentence. You need to make sure you're providing references in the format expected by sacrebleu.

    Here’s an alternative way to compute the BLEU-4 score using the sacrebleu package, but with the correct formatting of references:

    import sacrebleu
    
    # Assuming the lists 'reference_answers_list' and 'generated_questions_list' are obtained as shown previously
    
    # NOTE: 'sacrebleu' expects a list of reference translations for each sentence. Here,
    # we're assuming 'reference_answers_list' represents a single list of references (one per sentence).
    # Thus, we wrap each answer inside another list to simulate multiple references per sentence.
    refs = [[ref] for ref in reference_answers_list]
    
    # Assume 'generated_questions_list' represents hypotheses (one per sentence)
    hyps = generated_questions_list
    
    # Compute BLEU score
    bleu = sacrebleu.corpus_bleu(hyps, refs)
    
    # Print BLEU score
    print(f'BLEU-4 Score: {bleu.score:.2f}')
    

    By wrapping each reference within another list, we create the expected structure for sacrebleu. It should be noted that sacrebleu automatically tokenizes the text internally, so you don’t necessarily need to tokenize your sentences with nltk before passing them to sacrebleu; you can pass the raw, untokenized sentences directly.

    Also, sacrebleu returns the BLEU score as a percentage, so you don’t need to multiply by 100. If you prefer to use tokenized text, you should disable the internal tokenizer by setting the tokenizer='none' parameter in the corpus_bleu function.

  3. User

    this # Function to generate a question using a given model and context def generate_question(model, tokenizer, context, max_length=128, device="cuda" if torch.cuda.is_available() else "cpu"): # Move the model to the specified device model.to(device)

    # Tokenize and move tensors to the appropriate device
    inputs = tokenizer(context, return_tensors="pt", max_length=max_length, padding="max_length", truncation=True)
    inputs = {key: value.to(device) for key, value in inputs.items()}
    
    # Generate question using the model
    generated_question_ids = model.generate(**inputs)
    generated_question = tokenizer.decode(generated_question_ids[0], skip_special_tokens=True)
    
    return generated_question
    

    ho to compute bleu4

  4. Assistant

    To compute the BLEU-4 score for a set of generated questions against reference questions, you would typically follow these steps:

    1. Generate questions using the generate_question function for each context.
    2. Collect all generated questions and their corresponding reference questions.
    3. Use a BLEU score computation function to compare the generated text against references.

    Here's how you can compute the BLEU-4 score for a set of generated questions:

    import sacrebleu
    
    # Let's assume you have a list of contexts and their corresponding reference questions
    contexts = [...]
    reference_questions = [...]
    
    # Use your function to generate questions for each context
    generated_questions = [generate_question(model, tokenizer, context) for context in contexts]
    
    # Wrap reference questions for BLEU evaluation (assuming one reference per context)
    refs = [[ref] for ref in reference_questions]  # Each reference wrapped in a list
    
    # Compute BLEU score using sacrebleu
    bleu = sacrebleu.corpus_bleu(generated_questions, refs)
    
    # Print BLEU score
    print(f'BLEU-4 Score: {bleu.score:.2f}')
    

    Please remember that BLEU scores can be a useful metric for evaluating the quality of generated text, but there are limitations to consider. BLEU scores primarily focus on n-gram precision and do not account for semantic accuracy, relevance, fluency, or factual correctness. It is often best to use BLEU scores as part of a suite of evaluation metrics rather than in isolation.