I have saved my llama model in google drive and I want it to inference in google colab.…
- User
I have saved my llama model in google drive and I want it to inference in google colab. The github guide: Name Last commit message Last commit date parent directory .. chat_completion Fix quantization for inference 4 months ago README.md Fixed readme with renaming file name 4 days ago inference.py Make gradio and langchain optional dependencies (#676) 2 months ago multi_modal_infer.py convertion missing preprocessor_config.json. 2 weeks ago multi_modal_infer_gradio_UI.py Renamed the file and made that G small of gradio 4 days ago samsum_prompt.txt recipes/quickstart folder updated 5 months ago README.md Local Inference Hugging face setup Important Note: Before running the inference, you'll need your Hugging Face access token, which you can get at your Settings page here. Then run huggingface-cli login and copy and paste your Hugging Face access token to complete the login to make sure the scripts can download Hugging Face models if needed.
Multimodal Inference For Multi-Modal inference we have added multi_modal_infer.py which uses the transformers library.
The way to run this would be:
python multi_modal_infer.py --image_path PATH_TO_IMAGE --prompt_text "Describe this image" --temperature 0.5 --top_p 0.8 --model_name "meta-llama/Llama-3.2-11B-Vision-Instruct" Multi-modal Inferencing Using gradio UI for inferencing For multi-modal inferencing using gradio UI we have added multi_modal_infer_gradio_UI.py which used gradio and transformers library.
Steps to Run The way to run this would be:
Ensure having proper access to llama 3.2 vision models, then run the command given below python multi_modal_infer_gradio_UI.py --hf_token Text-only Inference For local inference we have provided an inference script. Depending on the type of finetuning performed during training the inference script takes different arguments.
To finetune all model parameters the output dir of the training has to be given as --model_name argument. In the case of a parameter efficient method like lora the base model has to be given as --model_name and the output dir of the training has to be given as --peft_model argument. Additionally, a prompt for the model in the form of a text file has to be provided. The prompt file can either be piped through standard input or given as --prompt_file parameter.
Content Safety The inference script also supports safety checks for both user prompt and model outputs. In particular, we use two packages, AuditNLG and Azure content safety.
Note If using Azure content Safety, please make sure to get the endpoint and API key as described here and add them as the following environment variables,CONTENT_SAFETY_ENDPOINT and CONTENT_SAFETY_KEY.
Examples:
Full finetuning of all parameters
cat <test_prompt_file> | python inference.py --model_name <training_config.output_dir> --use_auditnlg
PEFT method
cat <test_prompt_file> | python inference.py --model_name <training_config.model_name> --peft_model <training_config.output_dir> --use_auditnlg
prompt as parameter
python inference.py --model_name <training_config.output_dir> --prompt_file <test_prompt_file> --use_auditnlg The folder contains test prompts for summarization use-case:
samsum_prompt.txt ... Note on Llama version < 3.1 The default padding token in HuggingFace Tokenizer is None. To use padding the padding token needs to be added as a special token to the tokenizer, which in this case requires to resize the token_embeddings as shown below:
tokenizer.add_special_tokens( {
"pad_token": "<PAD>", } )model.resize_token_embeddings(model.config.vocab_size + 1) Padding would be required for batched inference. In this example, batch size = 1 so essentially padding is not required. However, we added the code pointer as an example in case of batch inference. For Llama version 3.1 use the special token <|finetune_right_pad_id|> (128004) for padding.
Chat completion The inference folder also includes a chat completion example, that adds built-in safety features in fine-tuned models to the prompt tokens. To run the example:
python chat_completion/chat_completion.py --model_name "PATH/TO/MODEL/7B/" --prompt_file chat_completion/chats.json --quantization 8bit --use_auditnlg Flash Attention and Xformer Memory Efficient Kernels Setting use_fast_kernels will enable using of Flash Attention or Xformer memory-efficient kernels based on the hardware being used. This would speed up inference when used for batched inputs. This has been enabled in optimum library from HuggingFace as a one-liner API, please read more here.
python chat_completion/chat_completion.py --model_name "PATH/TO/MODEL/7B/" --prompt_file chat_completion/chats.json --quantization 8bit --use_auditnlg --use_fast_kernels
python inference.py --model_name <training_config.output_dir> --peft_model <training_config.output_dir> --prompt_file <test_prompt_file> --use_auditnlg --use_fast_kernels Inference with FSDP checkpoints In case you have fine-tuned your model with pure FSDP and saved the checkpoints with "SHARDED_STATE_DICT" as shown here, you can use this converter script to convert the FSDP Sharded checkpoints into HuggingFace checkpoints. This enables you to use the inference script normally as mentioned above. To convert the checkpoint use the following command:
This is helpful if you have fine-tuned you model using FSDP only as follows:
torchrun --nnodes 1 --nproc_per_node 8 recipes/quickstart/finetuning/finetuning.py --enable_fsdp --model_name /path_of_model_folder/7B --dist_checkpoint_root_folder model_checkpoints --dist_checkpoint_folder fine-tuned --fsdp_config.pure_bf16 Then convert your FSDP checkpoint to HuggingFace checkpoints using:
python -m llama_recipes.inference.checkpoint_converter_fsdp_hf --fsdp_checkpoint_path PATH/to/FSDP/Checkpoints --consolidated_model_path PATH/to/save/checkpoints --HF_model_path_or_name PATH/or/HF/model_name
--HF_model_path_or_name specifies the HF Llama model name or path where it has config.json and tokenizer.json
By default, training parameter are saved in train_params.yaml in the path where FSDP checkpoints are saved, in the converter script we frist try to find the HugingFace model name used in the fine-tuning to load the model with configs from there, if not found user need to provide it.
Then run inference using:
python inference.py --model_name <training_config.output_dir> --prompt_file <test_prompt_file> Inference on large models like Meta Llama 405B The FP8 quantized variants of Meta Llama (i.e. meta-llama/Meta-Llama-3.1-405B-FP8 and meta-llama/Meta-Llama-3.1-405B-Instruct-FP8) can be executed on a single node with 8x80GB H100 using the scripts located in this folder. To run the unquantized Meta Llama 405B variants (i.e. meta-llama/Meta-Llama-3.1-405B and meta-llama/Meta-Llama-3.1-405B-Instruct) we need to use a multi-node setup for inference. The llama-recipes inference script currently does not allow multi-node inference. To run this model you can use vLLM with pipeline and tensor parallelism as showed in this example.
The github path: https://github.com/meta-llama/llama-recipes.git recipes/quickstart/inference/local_inference
Write the code for google colab. Show full code.
- Assistant
To utilize your LLaMA model saved in Google Drive for inference in Google Colab, you'll primarily need to take the following steps:
- Mount Google Drive to access your model.
- Install the necessary libraries and dependencies.
- Clone or download the repository containing the inference script.
- Set up configuration and run the inference.
Here's how you can do it:
# Mount Google Drive from google.colab import drive drive.mount('/content/drive') # Install required packages !pip install transformers !pip install gradio !pip install optimum # Clone the Github repository !git clone https://github.com/meta-llama/llama-recipes.git %cd llama-recipes # Optional: Hugging Face authentication (if needed) # !huggingface-cli login # Input your Hugging Face token when prompted # Set up the paths and configurations MODEL_NAME = '/content/drive/MyDrive/path_to_your_model_folder' # change to the path where your model is saved in Google Drive PROMPT_FILE = 'recipes/quickstart/inference/local_inference/samsum_prompt.txt' # adjust this as required # Run the inference !python inference.py --model_name $MODEL_NAME --prompt_file $PROMPT_FILEExplanation
-
Mount Google Drive: You'll first mount Google Drive where your model is stored so that Colab can access it.
-
Install Libraries: Install the required Python packages. Depending on your specific needs, you might have to install additional packages, such as
transformers,gradio,optimum, etc. -
Clone Repository: This step downloads the GitHub repository that contains your inference scripts and related files.
-
Configure Paths and Run: Define the model's path and any other configuration you need for running inference. Then, use the provided script to execute it.
Ensure that the paths such as
MODEL_NAMEmatch where your model is actually saved, and update any other configurations necessary for your specific use-case. Also, if you need to authenticate with Hugging Face to download some dependencies, usehuggingface-cli login.Finally, keep an eye on Colab's memory and runtime limits, especially if dealing with large models, as this might constrain your ability to run the model.
- User
Ok, what about the Chat completion ?
- Assistant
For chat completion using your LLaMA model in Google Colab, you need to follow a similar setup process, but with slight adjustments for running the chat completion script. Here's a Colab script that covers the steps:
# Step 1: Mount Google Drive to access your model from google.colab import drive drive.mount('/content/drive') # Step 2: Install necessary libraries !pip install transformers !pip install optimum # If AuditNLG or any other library is needed, install accordingly # !pip install auditnlg or any other necessary dependencies # Step 3: Clone the GitHub repository with the chat completion script !git clone https://github.com/meta-llama/llama-recipes.git %cd llama-recipes # Optional: Authenticate Hugging Face (if required) # !huggingface-cli login # Step 4: Set up the paths for your model and prompt file # Replace the path below with the path to your model in Google Drive MODEL_NAME = '/content/drive/MyDrive/path_to_your_model_folder' # Point to the chat completion JSON file or create one PROMPT_FILE = 'chat_completion/chats.json' # Adjust this path as necessary # Set other relevant parameters for the script QUANTIZATION = '8bit' # You can adjust according to your needs, like 16bit, 32bit, etc. USE_AUDITNLG = True # Set True if using AuditNLG for content safety; otherwise, False or remove the flag # Step 5: Run the chat completion script !python chat_completion/chat_completion.py \ --model_name $MODEL_NAME \ --prompt_file $PROMPT_FILE \ --quantization $QUANTIZATION \ --use_auditnlg $USE_AUDITNLGKey Points:
-
Prompt File: Make sure you have a JSON file (
chats.jsonor another file) with the prompts you wish to use for chat completion. This file should be in a format that the script expects and it should reside within the cloned repo or specify a full path to it if stored elsewhere. -
Quantization and Safety: Adjust the quantization and any content safety features as needed. For large models, quantization can help reduce memory usage.
-
Library Dependencies: Ensure all required libraries are installed. Depending on your model and which features or safety checks are enabled, additional packages might be required.
-
Environment Setup: Verify that your environment supports the execution of large models, especially if you're using Colab's free version. For better performance, consider upgrading to Colab Pro, which offers additional memory.
-
Adapt Paths as Needed: Make sure all paths, such as the model path (
MODEL_NAME) and the prompt file path (PROMPT_FILE), are correctly set according to where your files are stored in Google Drive or locally within Colab.
This script should help you set up a chat completion scenario using your saved LLaMA model in Google Colab. Adjust parameters and paths as necessary based on your specific setup and requirements.
-
- User
How to create the json file
- Assistant
Creating a JSON file for chat completion typically involves structuring your data in a way that the model can interpret as input prompts. This structure can vary depending on the specific requirements of the script you're using, but generally, a simple structure might involve a list of objects where each object contains a prompt and possibly an expected response or context.
Here's a basic example of how you might structure such a JSON file:
[ { "prompt": "Hello, how are you?", "context": "A casual conversation with a friend." }, { "prompt": "What's the weather like today?", "context": "Inquiry about current weather conditions." }, { "prompt": "Tell me a joke.", "context": "Looking for something light-hearted." } ]Steps to Create a JSON File
-
Define Your Prompts:
- Think about the questions or statements you want your model to respond to.
- Optionally, include additional context if your application requires it.
-
Write the JSON File:
- You can use any text editor to create the file (e.g., Notepad on Windows, TextEdit on macOS, or any other text editor you prefer).
-
Save the File:
- Save the file with the
.jsonextension, for examplechats.json.
- Save the file with the
-
Upload to Google Drive (optional):
- If you're using Google Colab and want the file to be easily accessible, you can upload it to your Google Drive.
Example Using Python to Create the JSON File
Here's how you can create a JSON file using Python code, which can be run directly within a Colab cell or locally:
import json # Define your data chat_data = [ {"prompt": "Hello, how are you?", "context": "A casual conversation with a friend."}, {"prompt": "What's the weather like today?", "context": "Inquiry about current weather conditions."}, {"prompt": "Tell me a joke.", "context": "Looking for something light-hearted."} ] # Specify file path file_path = '/content/chats.json' # If running locally, change this to an appropriate path # Write to JSON file with open(file_path, 'w') as f: json.dump(chat_data, f, indent=4) # Verify by reading the file contents (optional) with open(file_path, 'r') as f: file_contents = json.load(f) print(json.dumps(file_contents, indent=4))This script will create a JSON file (
chats.json) in your Colab environment. If you're working locally, adjust thefile_pathaccordingly. The file contains prompts and contexts that you'd like your model to use for generating responses during chat completion. -
- User
Show chat_data = [ {"prompt": "Hello, how are you?", "context": "A casual conversation with a friend."}, {"prompt": "What's the weather like today?", "context": "Inquiry about current weather conditions."}, {"prompt": "Tell me a joke.", "context": "Looking for something light-hearted."} ] in json format.
- Assistant
Certainly! Here's how the
chat_datalist of dictionaries would look in JSON format:[ { "prompt": "Hello, how are you?", "context": "A casual conversation with a friend." }, { "prompt": "What's the weather like today?", "context": "Inquiry about current weather conditions." }, { "prompt": "Tell me a joke.", "context": "Looking for something light-hearted." } ]This JSON structure consists of an array of objects, with each object containing a
promptand acontext. This format is common for scenarios where you provide both an input and some contextual information for the model to consider when generating responses. - User
Ok, from your previous response: For chat completion using your LLaMA model in Google Colab, you need to follow a similar setup process, but with slight adjustments for running the chat completion script. Here's a Colab script that covers the steps:
Step 1: Mount Google Drive to access your model
from google.colab import drive drive.mount('/content/drive')
Step 2: Install necessary libraries
!pip install transformers !pip install optimum
If AuditNLG or any other library is needed, install accordingly
!pip install auditnlg or any other necessary dependencies
Step 3: Clone the GitHub repository with the chat completion script
!git clone https://github.com/meta-llama/llama-recipes.git %cd llama-recipes
Optional: Authenticate Hugging Face (if required)
!huggingface-cli login
Step 4: Set up the paths for your model and prompt file
Replace the path below with the path to your model in Google Drive
MODEL_NAME = '/content/drive/MyDrive/path_to_your_model_folder'
Point to the chat completion JSON file or create one
PROMPT_FILE = 'chat_completion/chats.json' # Adjust this path as necessary
Set other relevant parameters for the script
QUANTIZATION = '8bit' # You can adjust according to your needs, like 16bit, 32bit, etc. USE_AUDITNLG = True # Set True if using AuditNLG for content safety; otherwise, False or remove the flag
Step 5: Run the chat completion script
!python chat_completion/chat_completion.py
--model_name PROMPT_FILE
--quantization USE_AUDITNLG Output: 2024-11-01 02:44:28.968304: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-11-01 02:44:29.003819: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-11-01 02:44:29.013839: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-11-01 02:44:29.035202: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-11-01 02:44:30.734932: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT /usr/local/lib/python3.10/dist-packages/llama_recipes/model_checkpointing/checkpoint_handler.py:17: DeprecationWarning:torch.distributed._shard.checkpointwill be deprecated, usetorch.distributed.checkpointinstead from torch.distributed._shard.checkpoint import ( User dialogs: [[{'role': 'user', 'content': 'what is the recipe of mayonnaise?'}], [{'role': 'user', 'content': 'I am going to Paris, what should I see?'}, {'role': 'assistant', 'content': "Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city. 2. The Louvre Museum: The Louvre is one of the world's largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa. 3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.These are just a few of the many attractions that Paris has to offer. With so much to see and do, it's no wonder that Paris is one of the most popular tourist destinations in the world."}, {'role': 'user', 'content': 'What is so great about #1?'}], [{'role': 'system', 'content': 'Always answer with Haiku'}, {'role': 'user', 'content': 'I am going to Paris, what should I see?'}], [{'role': 'system', 'content': 'Always answer with emojis'}, {'role': 'user', 'content': 'How to go from Beijing to NY?'}], [{'role': 'system', 'content': "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information."}, {'role': 'user', 'content': 'Write a brief birthday message to John'}]]==================================
use_fast_kernelsFalse Traceback (most recent call last): File "/content/llama-recipes/llama-recipes/recipes/quickstart/inference/local_inference/chat_completion/chat_completion.py", line 141, in fire.Fire(main) File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/content/llama-recipes/llama-recipes/recipes/quickstart/inference/local_inference/chat_completion/chat_completion.py", line 76, in main chats = [tokenizer.apply_chat_template(dialog) for dialog in dialogs] File "/content/llama-recipes/llama-recipes/recipes/quickstart/inference/local_inference/chat_completion/chat_completion.py", line 76, in chats = [tokenizer.apply_chat_template(dialog) for dialog in dialogs] File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 1803, in apply_chat_template chat_template = self.get_chat_template(chat_template, tools) File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 1967, in get_chat_template raise ValueError( ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating
- Assistant
The error you're encountering is related to the absence of a chat template, which is required for the script to format the dialogue data correctly for the model. In this situation, you need to ensure the tokenizer is equipped with a valid chat template configuration.
Here’s how you might resolve this issue and structure a working setup:
Steps to Address the Error
-
Update the
chat_completion.pyScript:- Ensure that the script includes a mechanism to apply a chat template or references such functionality correctly if it's supposed to be integrated via the Hugging Face Transformers library.
-
Provide a Chat Template:
- Since the error suggests that a chat template isn't provided, you'll need to either supply one manually to the tokenizer or ensure your model and tokenizer are compatible with chat-based completion tasks.
-
Configure or Create a Chat Template:
- If you’re extending the tokenizer to work with a customized chat template, you might need to define and pass it to the tokenizer programmatically.
Example Solutions:
You can modify or troubleshoot the script by following these potential solutions:
Define and Pass a Chat Template:
-
Modify the Code to Set a Chat Template:
If you can, modify the code to set a chat template if it's missing:
from transformers import AutoTokenizer # Load tokenizer tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) # Define a chat template (if not already defined) if not hasattr(tokenizer, 'chat_template'): tokenizer.chat_template = { "user": "<|user|>", "assistant": "<|assistant|>", "system": "<|system|>", "eos": "<|endofresponse|>" } # Continue with the script operation chats = [tokenizer.apply_chat_template(dialog) for dialog in dialogs]Adjust the placeholder values (
<|user|>,<|assistant|>, etc.) to align with your model's requirements. -
Ensure Chat Template with the Tokenizer:
Make sure the tokenizer and model you're using have built-in support for chat templates. This sometimes involves checking documentation for specific models if extending beyond common use cases.
Check for Model Updates:
-
Up-to-date library versions: Ensure your
transformerslibrary is up to date, as newer versions might include more functionality or fixes regarding chat templates. -
Model compatibility: Confirm the model used for chat completion supports chat tasks natively and whether it necessitates specific configurations.
Integration with Hugging Face:
- Explore Documentation: See Hugging Face's Transformers documentation for detailed instructions about setting up and utilizing chat templates with specific tokenizers or models.
By addressing the issues with the chat template setup, you should be able to avoid the error and perform chat completion tasks effectively using your LLaMA model on Google Colab.
-
- User
No, the code in chat_completion.py:
Copyright (c) Meta Platforms, Inc. and affiliates.
This software may be used and distributed according to the terms of the Llama 2 Community License Agreement.
from accelerate import init_empty_weights, load_checkpoint_and_dispatch
import fire import json import os import sys
import torch from transformers import AutoTokenizer
from llama_recipes.inference.chat_utils import read_dialogs_from_file from llama_recipes.inference.model_utils import load_model, load_peft_model from llama_recipes.inference.safety_utils import get_safety_checker from accelerate.utils import is_xpu_available
def main( model_name, peft_model: str=None, quantization: str = None, # Options: 4bit, 8bit max_new_tokens =256, #The maximum numbers of tokens to generate min_new_tokens:int=0, #The minimum numbers of tokens to generate prompt_file: str=None, seed: int=42, #seed value for reproducibility safety_score_threshold: float=0.5, do_sample: bool=True, #Whether or not to use sampling ; use greedy decoding otherwise. use_cache: bool=True, #[optional] Whether or not the model should use the past last key/values attentions Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding. top_p: float=1.0, # [optional] If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation. temperature: float=1.0, # [optional] The value used to modulate the next token probabilities. top_k: int=50, # [optional] The number of highest probability vocabulary tokens to keep for top-k-filtering. repetition_penalty: float=1.0, #The parameter for repetition penalty. 1.0 means no penalty. length_penalty: int=1, #[optional] Exponential penalty to the length that is used with beam-based generation. enable_azure_content_safety: bool=False, # Enable safety check with Azure content safety api enable_sensitive_topics: bool=False, # Enable check for sensitive topics using AuditNLG APIs enable_saleforce_content_safety: bool=True, # Enable safety check woth Saleforce safety flan t5 use_fast_kernels: bool = False, # Enable using SDPA from PyTorch Accelerated Transformers, make use Flash Attention and Xformer memory-efficient kernels enable_llamaguard_content_safety: bool = False, **kwargs ): if prompt_file is not None: assert os.path.exists( prompt_file ), f"Provided Prompt file does not exist {prompt_file}"
dialogs= read_dialogs_from_file(prompt_file) elif not sys.stdin.isatty(): dialogs = "\n".join(sys.stdin.readlines()) try: dialogs = json.loads(dialogs) except: print("Could not parse json from stdin. Please provide a json file with the user prompts. Exiting.") sys.exit(1) else: print("No user prompt provided. Exiting.") sys.exit(1) print(f"User dialogs:\n{dialogs}") print("\n==================================\n") # Set the seeds for reproducibility if is_xpu_available(): torch.xpu.manual_seed(seed) else: torch.cuda.manual_seed(seed) torch.manual_seed(seed) model = load_model(model_name, quantization, use_fast_kernels, **kwargs) if peft_model: model = load_peft_model(model, peft_model) tokenizer = AutoTokenizer.from_pretrained(model_name) chats = [tokenizer.apply_chat_template(dialog) for dialog in dialogs] with torch.no_grad(): for idx, chat in enumerate(chats): safety_checker = get_safety_checker(enable_azure_content_safety, enable_sensitive_topics, enable_saleforce_content_safety, enable_llamaguard_content_safety, ) # Safety check of the user prompt safety_results = [check(dialogs[idx][0]["content"]) for check in safety_checker] are_safe = all([r[1] for r in safety_results]) if are_safe: print(f"User prompt deemed safe.") print("User prompt:\n", dialogs[idx][0]["content"]) print("\n==================================\n") else: print("User prompt deemed unsafe.") for method, is_safe, report in safety_results: if not is_safe: print(method) print(report) print("Skipping the inferece as the prompt is not safe.") sys.exit(1) # Exit the program with an error status tokens= torch.tensor(chat).long() tokens= tokens.unsqueeze(0) attention_mask = torch.ones_like(tokens) if is_xpu_available(): tokens= tokens.to("xpu:0") else: tokens= tokens.to("cuda:0") outputs = model.generate( input_ids=tokens, attention_mask=attention_mask, max_new_tokens=max_new_tokens, do_sample=do_sample, top_p=top_p, temperature=temperature, use_cache=use_cache, top_k=top_k, repetition_penalty=repetition_penalty, length_penalty=length_penalty, **kwargs ) output_text = tokenizer.decode(outputs[0], skip_special_tokens=True) # Safety check of the model output safety_results = [check(output_text) for check in safety_checker] are_safe = all([r[1] for r in safety_results]) if are_safe: print("User input and model output deemed safe.") print(f"Model output:\n{output_text}") print("\n==================================\n") else: print("Model output deemed unsafe.") for method, is_safe, report in safety_results: if not is_safe: print(method) print(report)if name == "main": fire.Fire(main)
- Assistant
Thanks for sharing the
chat_completion.pycode. It looks like the issue arises from applying a chat template function (tokenizer.apply_chat_template) without having a defined chat template. The script assumes that a chat template is either preset within the tokenizer or provided as an argument, which isn’t the case in your setup.Steps to Fix the Issue:
-
Check Tokenizer Configuration: Ensure that the tokenizer loaded in your script supports chat templates. Some tokenizers or model setups might not have this feature enabled by default.
-
Modify the Code to Set a Chat Template:
If the tokenizer doesn’t come with a pre-configured chat template, you might need to manually define one. You can modify the script to provide a basic chat template like this:
from transformers import AutoTokenizer # Load tokenizer tokenizer = AutoTokenizer.from_pretrained(model_name) # Define a chat template manually if not already present if not hasattr(tokenizer, 'apply_chat_template'): print("Setting up a basic chat template.") tokenizer.apply_chat_template = lambda dialog: [ f"<|user|> {turn['content']} <|endofturn|>" if turn['role'] == 'user' else f"<|assistant|> {turn['content']} <|endofturn|>" for turn in dialog ]This snippet constructs a basic chat template directly in the Python script instead of relying on built-in tokenizer support. It uses example delimiters like
<|user|>,<|assistant|>, and<|endofturn|>to differentiate parts of the conversation. Adjust these tags if they don’t suit your model's setup. -
Review Model and Library Versions: Ensure that your version of the
transformerslibrary is compatible with the approach you're taking. Sometimes updating the library or reviewing its documentation can provide insights into newly supported features or necessary adjustments. -
Test with Example Inputs: Test the modified code with a simple chat JSON file to ensure it's functioning after adjustments:
[ {"role": "user", "content": "Hello, how are you?"}, {"role": "assistant", "content": "I'm doing well, thank you! How can I assist you today?"} ]
By making these changes, you should be able to rectify the error and facilitate chat completion using your setup. Adjustments may be necessary based on specific requirements or constraints of the models and tasks you are working with.
-
- User
Still ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating
- Assistant
The error you're encountering indicates that the tokenizer you're using expects a chat template to be explicitly defined, as it doesn't have a default one set. This is a feature introduced in some Hugging Face models to allow for structured handling of conversational data. Here's how you can resolve this:
Steps to Define a Chat Template
-
Check Documentation for Chat Template: First, review the Hugging Face documentation on chat templating to understand how chat templates are expected to be structured.
-
Define a Chat Template: You need to explicitly set a chat template for your tokenizer. This template is generally a Python dictionary or a structured set of instructions that define how prompts and responses are formatted.
-
Setting Up and Using a Chat Template:
Here’s a template setup you can manually define in your script prior to tokenizing your chats:
from transformers import AutoTokenizer # Load the tokenizer tokenizer = AutoTokenizer.from_pretrained(model_name) # Define your chat template, assuming a typical conversation structure chat_template = { "roles": { "user": "<|user|>", "assistant": "<|assistant|>", "system": "<|system|>" }, "delimiter": "<|endofturn|>\n", "prompt_structure": "{system}{dialog}", "dialog_turn": "{role}{content}{delimiter}" } # Assign the chat template to the tokenizer (if your tokenizer supports this operation) try: tokenizer.chat_template = chat_template except AttributeError: # Manually apply chat formatting if assign direct isn't supported def format_chat(dialog): formatted = "" for turn in dialog: role = chat_template["roles"].get(turn["role"], "") content = turn["content"] delimiter = chat_template["delimiter"] formatted += f"{role} {content} {delimiter}" return formatted # Overwrite apply_chat_template with manual formatter if needed tokenizer.apply_chat_template = lambda dialog: format_chat(dialog) # Continue with your script, utilizing this templateKey Considerations
- Structure Adherence: Ensure the JSON or dialogue passed follows the format expected by your template.
- Testing and Debugging: After defining and setting up your chat template, ensure your conversation examples are consistent with these rules and aim to test it with basic dialog inputs.
- Update Libraries: Consider checking for updates to the
transformerslibrary to ensure support for new features or templates.
Further Troubleshooting
- Library Compatibility: The capability to set a chat template may depend on the
transformersversion and the model you are using. Double-check any related updates or documentation for specific requirements. - Verbose Output: Add verbosity to your script's print statements or use debugging to ensure the chat templates are applied correctly.
This approach should help integrate a custom chat template into the tokenizer, resolve errors, and facilitate proper conversation formatting.
-
- User
ValueError: This model has multiple chat templates with no default specified! Please either pass a chat template or the name of the template you wish to use to the
chat_templateargument. Available template names are ['delimiter', 'dialog_turn', 'prompt_structure', 'roles']. - Assistant
The error message you're seeing indicates that the model or tokenizer has multiple chat templates defined, but none is set as the default. This situation requires you to specify which template to use when applying the chat structure to your dialogue data.
Here’s how you can handle this situation:
Approach to Specify a Chat Template
-
Understand the Available Templates: The error message lists the available template names:
'delimiter','dialog_turn','prompt_structure','roles'. Each represents a part of the template configuration used by the tokenizer or model to structure chat dialogues. -
Choose the Appropriate Template: Since you need to specify one, decide which part of the template you want to use based on how your scripts expect to format the data.
-
Set the
chat_templateArgument:Modify the function or method call where you apply the chat template to include a specific chat template choice. If you have a generic
apply_chat_templateor similar method in your workflow, specify the template you want to use directly in this call.
Here’s an example to set and use a specific chat template:
from transformers import AutoTokenizer # Load the tokenizer tokenizer = AutoTokenizer.from_pretrained(model_name) # Assuming you want to use the full chat structure including formatting instructions, # here, integrate all your template selections or logic if the library expects them separately. chat_template = { "roles": { "user": "<|user|>", "assistant": "<|assistant|>", "system": "<|system|>" }, "delimiter": "<|endofturn|>", "prompt_structure": "{system}{dialog}", "dialog_turn": "{role}{content}{delimiter}" } # Assign the chat template directly with the appropriate template name # You might have to integrate these choices into a single configuration or execution call dialog_data = [{'role': 'user', 'content': 'Hello, how are you?'}] # Manually construct based on the structure formatted_dialog = [ f"{chat_template['roles']['user']} {utterance['content']} {chat_template['delimiter']}" for utterance in dialog_data ] # Now use this formatted dialog in your pipeline print("Formatted Dialog:", formatted_dialog)Considerations
- Integration Checks: Ensure that wherever you attempt to use the
chat_template, such assignments are compatible with the function or system that processes your dialogue. - Documentation Review: Check the model's documentation for any specifics on chat templates if they are provided as part of configuration options.
Alternate Strategy
- Explicit Passing: If possible, pass the intended template exact usage to the method expecting it, ensuring it aligns with expected format parameters in
dialog_turnor another applicable section.
Following these steps should help you properly configure and specify the chat template, avoiding ambiguity about which part of the template to utilize. Adjust the approach based on specific method signatures or model configurations.
-
- User
the code in chat_completion.py:
Copyright (c) Meta Platforms, Inc. and affiliates. This software may be used and distributed according to the terms of the Llama 2 Community License Agreement. from accelerate import init_empty_weights, load_checkpoint_and_dispatch import fire import json import os import sys
import torch from transformers import AutoTokenizer
from llama_recipes.inference.chat_utils import read_dialogs_from_file from llama_recipes.inference.model_utils import load_model, load_peft_model from llama_recipes.inference.safety_utils import get_safety_checker from accelerate.utils import is_xpu_available
def main( model_name, peft_model: str=None, quantization: str = None, # Options: 4bit, 8bit max_new_tokens =256, #The maximum numbers of tokens to generate min_new_tokens:int=0, #The minimum numbers of tokens to generate prompt_file: str=None, seed: int=42, #seed value for reproducibility safety_score_threshold: float=0.5, do_sample: bool=True, #Whether or not to use sampling ; use greedy decoding otherwise. use_cache: bool=True, #[optional] Whether or not the model should use the past last key/values attentions Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding. top_p: float=1.0, # [optional] If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation. temperature: float=1.0, # [optional] The value used to modulate the next token probabilities. top_k: int=50, # [optional] The number of highest probability vocabulary tokens to keep for top-k-filtering. repetition_penalty: float=1.0, #The parameter for repetition penalty. 1.0 means no penalty. length_penalty: int=1, #[optional] Exponential penalty to the length that is used with beam-based generation. enable_azure_content_safety: bool=False, # Enable safety check with Azure content safety api enable_sensitive_topics: bool=False, # Enable check for sensitive topics using AuditNLG APIs enable_saleforce_content_safety: bool=True, # Enable safety check woth Saleforce safety flan t5 use_fast_kernels: bool = False, # Enable using SDPA from PyTorch Accelerated Transformers, make use Flash Attention and Xformer memory-efficient kernels enable_llamaguard_content_safety: bool = False, **kwargs ): if prompt_file is not None: assert os.path.exists( prompt_file ), f"Provided Prompt file does not exist {prompt_file}"
dialogs= read_dialogs_from_file(prompt_file)elif not sys.stdin.isatty(): dialogs = "\n".join(sys.stdin.readlines()) try: dialogs = json.loads(dialogs) except: print("Could not parse json from stdin. Please provide a json file with the user prompts. Exiting.") sys.exit(1) else: print("No user prompt provided. Exiting.") sys.exit(1)
print(f"User dialogs:\n{dialogs}") print("\n==================================\n")
Set the seeds for reproducibility
if is_xpu_available(): torch.xpu.manual_seed(seed) else: torch.cuda.manual_seed(seed) torch.manual_seed(seed)
model = load_model(model_name, quantization, use_fast_kernels, **kwargs) if peft_model: model = load_peft_model(model, peft_model)
tokenizer = AutoTokenizer.from_pretrained(model_name)
chats = [tokenizer.apply_chat_template(dialog) for dialog in dialogs]
with torch.no_grad(): for idx, chat in enumerate(chats): safety_checker = get_safety_checker(enable_azure_content_safety, enable_sensitive_topics, enable_saleforce_content_safety, enable_llamaguard_content_safety, ) # Safety check of the user prompt safety_results = [check(dialogs[idx][0]["content"]) for check in safety_checker] are_safe = all([r[1] for r in safety_results]) if are_safe: print(f"User prompt deemed safe.") print("User prompt:\n", dialogs[idx][0]["content"]) print("\n==================================\n") else: print("User prompt deemed unsafe.") for method, is_safe, report in safety_results: if not is_safe: print(method) print(report) print("Skipping the inferece as the prompt is not safe.") sys.exit(1) # Exit the program with an error status tokens= torch.tensor(chat).long() tokens= tokens.unsqueeze(0) attention_mask = torch.ones_like(tokens) if is_xpu_available(): tokens= tokens.to("xpu:0") else: tokens= tokens.to("cuda:0") outputs = model.generate( input_ids=tokens, attention_mask=attention_mask, max_new_tokens=max_new_tokens, do_sample=do_sample, top_p=top_p, temperature=temperature, use_cache=use_cache, top_k=top_k, repetition_penalty=repetition_penalty, length_penalty=length_penalty, **kwargs )
output_text = tokenizer.decode(outputs[0], skip_special_tokens=True) # Safety check of the model output safety_results = [check(output_text) for check in safety_checker] are_safe = all([r[1] for r in safety_results]) if are_safe: print("User input and model output deemed safe.") print(f"Model output:\n{output_text}") print("\n==================================\n") else: print("Model output deemed unsafe.") for method, is_safe, report in safety_results: if not is_safe: print(method) print(report)if name == "main": fire.Fire(main)
- Assistant
The code you've provided suggests that the script is designed to apply a chat template during tokenization of dialogues, but it's encountering a configuration issue related to specifying a chat template. Let's address this situation.
To resolve the
ValueErrorabout multiple chat templates, you need to specify which template to use when callingtokenizer.apply_chat_template. However, if the tokenizer doesn't have this attribute or requires manual handling of chat templates, you might need to adjust how dialogues are formatted before tokenization.Here’s a focused approach to updating your script for handling chat templates manually, given your tokenizer setup does not directly support chat templates:
Updating the Script
-
Manually Format Dialogues:
If direct chat templating isn't supported by the tokenizer, format the dialogues manually before processing:
from transformers import AutoTokenizer # Load tokenizer tokenizer = AutoTokenizer.from_pretrained(model_name) # Sample chat template: Define roles and delimiters chat_template = { "roles": { "user": "<|user|>", "assistant": "<|assistant|>", "system": "<|system|>", }, "turn_delimiter": "<|endofturn|>\n" } # Function to manually apply a chat template def apply_chat_template(dialog, template): formatted_chat = [] for turn in dialog: role = template["roles"].get(turn["role"], "") content = turn["content"] delimiter = template["turn_delimiter"] formatted_chat.append(f"{role} {content} {delimiter}") return ''.join(formatted_chat) # Prepare the dialogues using the manually defined template chats = [apply_chat_template(dialog, chat_template) for dialog in dialogs] # Continue with tokenization and model processing tokens_list = [tokenizer(chat, return_tensors="pt") for chat in chats] # Process tokens as before with torch.no_grad(): for idx, tokens in enumerate(tokens_list): input_ids = tokens['input_ids'].to(model.device) attention_mask = tokens['attention_mask'].to(model.device) # Your generation logic here outputs = model.generate( input_ids=input_ids, attention_mask=attention_mask, max_new_tokens=max_new_tokens, do_sample=do_sample, top_p=top_p, temperature=temperature, use_cache=use_cache, top_k=top_k, repetition_penalty=repetition_penalty, length_penalty=length_penalty, **kwargs ) # Decode and print output_text = tokenizer.decode(outputs[0], skip_special_tokens=True) print(f"Model output:\n{output_text}\n==================================\n")Key Adjustments Explained
-
Manual Formatting: Before tokenizing, dialogues are manually structured using a custom-defined chat template. This compensates for the lack of direct chat template support in the tokenizer.
-
Tokenization Settings: Adjusted to use
return_tensors="pt", ensuring compatibility with PyTorch models.
Why This Matters
By manually formatting the dialogues in a consistent manner, you ensure that the inputs to your model align with expected conversational structures, handling situations where built-in tokenizer methods like
apply_chat_templatearen't applicable or throw errors in your environment setup. This approach bypasses the need for implicit template handling and grants you control over how dialogues are structured for generation tasks. -