What standard of format do the chat templates in the huggingface transformers library…
- User
What standard of format do the chat templates in the huggingface transformers library follow?
Assistant
The chat templates in the Hugging Face Transformers library follow a Jinja2 templating standard to dynamically format multi-turn conversations for text generation models. This system is model-specific but adheres to common conventions:
1. Core Structure
- Messages are represented as dictionaries in a list:
messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is AI?"}, {"role": "assistant", "content": "AI is..."), {"role": "user", "content": "How does it work?"} ] - Roles: Typically
system,user,assistant.
2. Jinja2 Template Components
- Loops: Iterate over messages with
{% for message in messages %}. - Conditionals: Handle roles via
{% if message['role'] == 'user' %}. - Variables: Insert content using
{{ message['content'] }}. - Special Tokens: Reference tokens like
bos_token,eos_token.
3. Common Conventions
- Role-Based Wrapping:
Example (Llama 2-style):{% if messages[0]['role'] == 'system' %} {% set loop_messages = messages[1:] %} {% set system_message = messages[0]['content'] %} {% endif %} {{ bos_token }} + "{% for message in loop_messages %}" {% if message['role'] == 'user' %} {{ '[INST] ' + message['content'] + ' [/INST]' }} {% else %} {{ message['content'] + eos_token }} {% endif %} {% endfor %} - Token Handling: Explicit
bos_token/eos_tokeninsertion. - System Prompt: Often separated from dialogue turns.
4. Usage via
apply_chat_templatefrom transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf") formatted = tokenizer.apply_chat_template( messages, tokenize=False, # Return string (set True for token IDs) add_generation_prompt=True # Add assistant prompt for reply )Output Format (Llama 2 example):
<s>[INST] <<SYS>> You are a helpful assistant. <</SYS>> What is AI? [/INST] AI is... </s><s>[INST] How does it work? [/INST]5. Key Standards
- Model-Specific Templates: Predefined in
tokenizer_config.json(check viatokenizer.chat_template). - Flexibility: Customizable via Jinja2 (override
tokenizer.chat_template). - Add Generation Prompt: Use
add_generation_prompt=Trueto append an assistant turn starter (e.g.,[/INST]).
6. Examples by Model Family
Model Template Style Llama 2 [INST]/[/INST]with system promptsMistral Similar to Llama 2 Zephyr Uses `< OpenAI (ChatGPT) `"< Resources:
- Hugging Face Chat Template Guide
- Common Template Examples
- Customization: See Jinja2 docs for advanced syntax.
To inspect a model's template:
print(tokenizer.chat_template)- Messages are represented as dictionaries in a list:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.