Skip to content
TheChatArchiveThe Chat Archive
DeepSeekdeepseek-r1-0528· DeepSeekHugging Face LMArena Human Preference 140K

What standard of format do the chat templates in the huggingface transformers library…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Hunyuanhunyuan-turbos-20250416’s answer.
  1. User

    What standard of format do the chat templates in the huggingface transformers library follow?

  2. DeepSeek Assistant

    The chat templates in the Hugging Face Transformers library follow a Jinja2 templating standard to dynamically format multi-turn conversations for text generation models. This system is model-specific but adheres to common conventions:

    1. Core Structure

    • Messages are represented as dictionaries in a list:
      messages = [
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "What is AI?"},
          {"role": "assistant", "content": "AI is..."),
          {"role": "user", "content": "How does it work?"}
      ]
      
    • Roles: Typically system, user, assistant.

    2. Jinja2 Template Components

    • Loops: Iterate over messages with {% for message in messages %}.
    • Conditionals: Handle roles via {% if message['role'] == 'user' %}.
    • Variables: Insert content using {{ message['content'] }}.
    • Special Tokens: Reference tokens like bos_token, eos_token.

    3. Common Conventions

    • Role-Based Wrapping:
      Example (Llama 2-style):
      {% if messages[0]['role'] == 'system' %}
          {% set loop_messages = messages[1:] %}
          {% set system_message = messages[0]['content'] %}
      {% endif %}
      {{ bos_token }} + "{% for message in loop_messages %}"
          {% if message['role'] == 'user' %}
              {{ '[INST] ' + message['content'] + ' [/INST]' }}
          {% else %}
              {{ message['content'] + eos_token }}
          {% endif %}
      {% endfor %}
      
    • Token Handling: Explicit bos_token/eos_token insertion.
    • System Prompt: Often separated from dialogue turns.

    4. Usage via apply_chat_template

    from transformers import AutoTokenizer
    
    tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
    formatted = tokenizer.apply_chat_template(
        messages, 
        tokenize=False,  # Return string (set True for token IDs)
        add_generation_prompt=True  # Add assistant prompt for reply
    )
    

    Output Format (Llama 2 example):

    <s>[INST] <<SYS>>
    You are a helpful assistant.
    <</SYS>>
    
    What is AI? [/INST] AI is... </s><s>[INST] How does it work? [/INST]
    

    5. Key Standards

    • Model-Specific Templates: Predefined in tokenizer_config.json (check via tokenizer.chat_template).
    • Flexibility: Customizable via Jinja2 (override tokenizer.chat_template).
    • Add Generation Prompt: Use add_generation_prompt=True to append an assistant turn starter (e.g., [/INST]).

    6. Examples by Model Family

    ModelTemplate Style
    Llama 2[INST]/[/INST] with system prompts
    MistralSimilar to Llama 2
    ZephyrUses `<
    OpenAI (ChatGPT)`"<

    Resources:

    To inspect a model's template:

    print(tokenizer.chat_template)
    

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.