Skip to content
The Chat Archive
Hugging Face Space: tomoniai-mixtral-chat ShareLM (browser-extension contributions)

using huggingface's transformers library, how can I enforce my model to generate one of…

  1. User

    using huggingface's transformers library, how can I enforce my model to generate one of ['A', 'B', 'C', 'D'] only? it should output the option with the higher probability given the input

  2. Assistant

    To enforce your model to generate one of ['A', 'B', 'C', 'D'] only and output the option with the highest probability given the input, you can use the generate method of the Hugging Face Transformers library and set the do_sample parameter to False and the top_k parameter to 4. This will force the model to only generate one of the specified tokens and select the one with the highest probability. Here's an example:

    from transformers import AutoTokenizer, AutoModelForSequenceClassification

    tokenizer = AutoTokenizer.from_pretrained("model_name") model = AutoModelForSequenceClassification.from_pretrained("model_name", num_labels=4)

    input_text = "This is some input text" input_ids = tokenizer.encode(input_text, return_tensors="pt")

    outputs = model.generate( input_ids, max_length=1, do_sample=False, top_k=4, pad_token_id=tokenizer.pad_token_id,

  3. User

    but what if I want to use AutoModelForSeq2SeqLM model, and not AutoModelForSequenceClassification?

  4. Assistant

    If you are using AutoModelForSeq2SeqLM model, you can enforce your model to generate one of ['A', 'B', 'C', 'D'] only and output the option with the highest probability given the input by using the generate method and setting the num_beams parameter to 4 and the early_stopping parameter to True. This will force the model to consider only the top 4 beam search results and return the one that ends with the specified token. Here's an example:

    from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

    tokenizer = AutoTokenizer.from_pretrained("model_name") model = AutoModelForSeq2SeqLM.from_pretrained("model_name")

    input_text = "This is some input text" input_ids = tokenizer.encode(input_text, return_tensors="pt")

    outputs = model.generate( input_ids, max_length=1, num_beams=4, early_stopping=True, pad_token_id=tokenizer

  5. User

    this is not right

  6. Assistant

    I apologize for the confusion. The previous answer I provided is not correct for AutoModelForSeq2SeqLM model. To enforce your model to generate one of ['A', 'B', 'C', 'D'] only and output the option with the highest probability given the input, you can use the generate method and set the do_sample parameter to False and the top_k parameter to 4. Then you can post-process the output to select the token from the list ['A', 'B', 'C', 'D'] that has the highest probability. Here's an example:

    from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

    tokenizer = AutoTokenizer.from_pretrained("model_name") model = AutoModelForSeq2SeqLM.from_pretrained("model_name")

    input_text = "This is some input text" input_ids = tokenizer.encode(input_text, return_tensors="pt")

    outputs = model.generate( input_ids, max_length=1, do_sample=False