provide an end to end code to fine tune llava-1.6-vicuna-7b on a custom dataset. Include…
- User
provide an end to end code to fine tune llava-1.6-vicuna-7b on a custom dataset. Include also how the dataset must be constructed (what format should it be in). The dataset is a visual question answering dataset
Assistant
Okay, let's set up the process for fine-tuning
llava-1.6-vicuna-7bon your custom Visual Question Answering (VQA) dataset. This involves preparing the dataset in the correct format and then writing the Python script using Hugging Face'stransformerslibrary.Assumptions:
- You have a machine with a suitable GPU (LLaVA-7B requires significant VRAM, often >16GB, especially for training).
- You have Python installed, along with
piporconda. - Your images are stored in a directory.
1. Dataset Construction and Format
LLaVA models are typically trained on conversational data paired with images. For VQA fine-tuning, you need to structure your dataset similarly. The standard format is often a JSON Lines file (
.jsonl), where each line is a JSON object representing one data point.Required Format:
Each JSON object in your
.jsonlfile should contain:id: A unique identifier for the data point (optional but good practice).image: The relative or absolute path to the image file associated with this data point.conversations: A list of dictionaries, representing turns in a conversation. For VQA, this will typically be a two-turn conversation: the user's question and the assistant's answer. The structure must include the<image/>token where the image context is expected.
Example
dataset.jsonlfile:{"id": "sample_1", "image": "images/cat_on_sofa.jpg", "conversations": [{"from": "human", "value": "<image/>\nWhat is the animal doing?"}, {"from": "gpt", "value": "The cat is sleeping on the sofa."}]} {"id": "sample_2", "image": "images/street_scene.png", "conversations": [{"from": "human", "value": "<image/>\nHow many cars are visible?"}, {"from": "gpt", "value": "There are three cars visible in the street scene."}]} {"id": "sample_3", "image": "path/to/your/images/kitchen.jpeg", "conversations": [{"from": "human", "value": "<image/>\nWhat color is the countertop?"}, {"from": "gpt", "value": "The countertop is black granite."}]}Key Points:
<image/>Token: The user's turn (from: human) must include the special<image/>token, usually followed by a newline (\n) and then the actual question text. This token signals to the model where to incorporate the image features.fromfield: Use"human"for the user's question turn and"gpt"(or sometimes"assistant") for the model's answer turn. Consistency is important.- Image Paths: Ensure the paths in the
imagefield correctly point to your image files relative to where you run the script, or use absolute paths. - File Structure: You'll typically have:
your_project_directory/finetune_llava.py(The script below)dataset.jsonl(Your data definitions)images/(Directory containing all your image files likecat_on_sofa.jpg,street_scene.png,kitchen.jpeg)
2. End-to-End Fine-tuning Code
This script uses Hugging Face
transformers,datasets,accelerate, andbitsandbytes(optional, for memory efficiency like 4-bit training).Step 1: Install Libraries
pip install transformers datasets accelerate bitsandbytes peft # PEFT for potential LoRA pip install torch torchvision torchaudio # Or install PyTorch matching your CUDA version pip install Pillow # For image loadingStep 2: Fine-tuning Script (
finetune_llava.py)import os import json import torch from PIL import Image from datasets import load_dataset from transformers import ( AutoProcessor, LlavaForConditionalGeneration, TrainingArguments, Trainer, BitsAndBytesConfig # Optional: for 4-bit quantization ) # --- Configuration --- # Model ID from Hugging Face Hub model_id = "llava-hf/llava-1.6-vicuna-7b-hf" # Path to your JSON Lines dataset file dataset_path = "dataset.jsonl" # Directory where your images are stored (ensure paths in dataset.jsonl are relative to this or absolute) # If paths in jsonl are absolute, this can be None or "" image_dir = "images/" # Output directory to save the fine-tuned model output_dir = "./llava-1.6-vicuna-7b-finetuned-vqa" # Training parameters training_epochs = 1 batch_size = 4 # Adjust based on your GPU VRAM gradient_accumulation_steps = 4 # Effective batch size = batch_size * gradient_accumulation_steps learning_rate = 1e-5 # LLaVA authors suggest 1e-5 or 2e-5 for fine-tuning logging_steps = 50 save_steps = 500 # Save checkpoints periodically # Optional: Use 4-bit quantization for memory saving (requires bitsandbytes) use_4bit_quantization = True # Set to False if you have enough VRAM or encounter issues # --- Quantization Configuration (Optional) --- bnb_config = None if use_4bit_quantization: bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16 # Or torch.bfloat16 if supported ) # --- Load Model and Processor --- print("Loading model and processor...") processor = AutoProcessor.from_pretrained(model_id) model = LlavaForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.float16, # Use float16 for faster training and less memory low_cpu_mem_usage=True, # Load model layers incrementally quantization_config=bnb_config, # Apply quantization if configured device_map="auto" # Automatically distribute model across available GPUs ) print("Model and processor loaded.") # --- Prepare Chat Template --- # LLaVA uses a specific chat template (usually Vicuna). The processor should handle this, # but we ensure the tokenizer knows about the '<image/>' token. # The processor should already add special tokens if they aren't there. # We just need to be sure padding is handled correctly. processor.tokenizer.padding_side = "right" # Ensure padding is on the right # --- Load and Preprocess Dataset --- print("Loading dataset...") # Load from JSON Lines file raw_dataset = load_dataset("json", data_files=dataset_path, split="train") # Preprocessing function def preprocess_data(examples): processed_texts = [] image_paths = [] # Construct the conversation text in the required format for the processor for i in range(len(examples['conversations'])): # Assumes structure like: [{"from": "human", "value": "<image/>\nQuestion?"}, {"from": "gpt", "value": "Answer."}] conversation = examples['conversations'][i] if len(conversation) == 2 and conversation[0]['from'] == 'human' and conversation[1]['from'] == 'gpt': # Format using the chat template implicitly applied by the processor # The processor expects a chat structure or raw text + images # Let's create the text that includes the prompt and the answer for tokenization # The processor's chat template will typically format this correctly # Example structure needed by processor might look like: # "<s>[INST] <image/>\n{question} [/INST] {answer} </s>" (depending on the base model, e.g., Vicuna) question = conversation[0]['value'] # e.g., "<image/>\nWhat is this?" answer = conversation[1]['value'] # e.g., "This is a cat." # The processor expects the prompt text for processing. # We need to format the full conversation for input_ids and labels. # The processor.apply_chat_template might be useful if available and correctly configured for LLaVA. # Let's manually construct the full text for tokenization: # NOTE: Check the exact template LLaVA 1.6 Vicuna expects. Often it's something like: # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image/>\n{question} ASSISTANT: {answer}" # Or using the more common Vicuna structure: prompt = f"<s>[INST] {question} [/INST]" # Part the model conditions on full_text = f"<s>[INST] {question} [/INST] {answer}</s>" # Full sequence including answer processed_texts.append(full_text) # Construct the full image path img_path = os.path.join(image_dir, examples['image'][i]) if image_dir else examples['image'][i] image_paths.append(img_path) else: print(f"Skipping invalid conversation structure: {conversation}") # Add placeholders to maintain batch structure, will be filtered later if necessary processed_texts.append("") image_paths.append(None) # Load images - Handle potential errors images = [] valid_indices = [] for idx, path in enumerate(image_paths): if path and os.path.exists(path): try: images.append(Image.open(path).convert("RGB")) valid_indices.append(idx) except Exception as e: print(f"Warning: Could not load image {path}:</s>- User
if the followinf is the script that I use for inference using llava model: as this is the way I do inference on the modek: say I am using llava-1.6-vicuna-7b, instead of 13b, provide end to end fine tuning script on gaudi, that I can run by simply copy and pasting the code snippet you provide.
here is a sample code for running an inference if you might need that import habana_frameworks.torch as ht import habana_frameworks.torch.core as htcore
from optimum.habana.transformers.modeling_utils import adapt_transformers_to_gaudi adapt_transformers_to_gaudi()
from transformers import LlavaNextProcessor, AutoProcessor, AutoConfig, LlavaNextForConditionalGeneration import torch from PIL import Image import requests import os import time import re
device = torch.device("hpu") args_model_name_or_path = "/workspace/models/model_llava_v1_6_vicuna_7b" model_type = AutoConfig.from_pretrained(args_model_name_or_path).model_type
print("Loading the processor") processor = AutoProcessor.from_pretrained(args_model_name_or_path)
print("Loading the model") model = LlavaNextForConditionalGeneration.from_pretrained( args_model_name_or_path, torch_dtype=torch.bfloat16, ) model.to("hpu")
fire_sequences_root = "/workspace/Full_data/test_sequences"
Initialize results dictionary
prediction_results = {}
Get all fire sequence folders
fire_sequence_folders = sorted([ os.path.join(fire_sequences_root, folder) for folder in os.listdir(fire_sequences_root) if os.path.isdir(os.path.join(fire_sequences_root, folder)) ])
print(f"Found {len(fire_sequence_folders)} fire sequence folders.")
Start processing each fire sequence folder
start = time.perf_counter()
batch_size = 4 generate_kwargs = {"max_new_tokens": 200, "do_sample": False} prediction_results_per_image_file = {}
Initialize confusion matrix metrics
metrics = { "true_positive": 0, "false_negative": 0, "false_positive": 0, "true_negative": 0 }
Setting up the conversation for the wildfire detection (original prompt)
fire_conversation = [ { "role": "user", "content": [ {"type": "text", "text": "You are an expert in detecting wildfire smokes starting from their ignition. Can you see one in the image? Respond in simple 'YES' or 'NO'. No explanation needed."}, {"type": "image"}, ], } ]
#With description prompt
fire_conversation = [
{
"role": "user",
"content": [
{"type": "text", "text": "You are an expert in detecting wildfire smokes starting from their ignition. Can you see one in the image? Respond in simple 'YES' or 'NO' as the first sentence. In the second sentence provide a brief description of what you see."},
{"type": "image"},
],
}
]
fire_prompt = processor.apply_chat_template(fire_conversation, add_generation_prompt=True)
for fire_sequence_folder in fire_sequence_folders: fire_sequence_name = os.path.basename(fire_sequence_folder)
prediction_results[fire_sequence_name] = { "original": {"before": 0, "after": 0}, "prediction": {"before": 0, "after": 0} } fire_image_paths = sorted([ os.path.join(fire_sequence_folder, file) for file in os.listdir(fire_sequence_folder) if file.endswith(".jpg") ]) print(f"Processing {fire_sequence_name} with {len(fire_image_paths)} images...") # Batch processing for idx in range(0, len(fire_image_paths), batch_size): batch_images = [Image.open(image_path) for image_path in fire_image_paths[idx : idx + batch_size]] batch_paths = fire_image_paths[idx : idx + batch_size] current_batch_inputs = processor( images=batch_images, text=[fire_prompt] * len(batch_images), padding=True, return_tensors="pt" ).to(model.device) current_batch_generate_ids = model.generate(**current_batch_inputs, max_new_tokens=30) current_batch_results = processor.batch_decode( current_batch_generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False ) for image_path, result in zip(batch_paths, current_batch_results): current_fire_image_path = os.path.basename(image_path) processed_result = result.split("ASSISTANT")[-1].strip().upper() print(f"File: {current_fire_image_path}, Result: {processed_result}") is_fire = "+" in current_fire_image_path prediction_is_fire = "YES" in processed_result # Update confusion matrix metrics if is_fire and prediction_is_fire: metrics["true_positive"] += 1 elif is_fire and not prediction_is_fire: metrics["false_negative"] += 1 elif not is_fire and prediction_is_fire: metrics["false_positive"] += 1 elif not is_fire and not prediction_is_fire: metrics["true_negative"] += 1 if is_fire: prediction_results[fire_sequence_name]["original"]["after"] += 1 else: prediction_results[fire_sequence_name]["original"]["before"] += 1 if prediction_is_fire: prediction_results[fire_sequence_name]["prediction"]["after"] += 1 else: prediction_results[fire_sequence_name]["prediction"]["before"] += 1 prediction_results_per_image_file[image_path] = processed_resultend = time.perf_counter() duration = end - start print(f"Total duration: {duration:.2f} seconds")
Print summary of results
for seq_name, results in prediction_results.items(): print(f"\nSummary for {seq_name}:") print(f" Original - Before: {results['original']['before']}, After: {results['original']['after']}") print(f" Prediction - Before: {results['prediction']['before']}, After: {results['prediction']['after']}")
Print confusion matrix metrics
print("\nConfusion Matrix Metrics:") print(f" True Positives: {metrics['true_positive']}") print(f" False Negatives: {metrics['false_negative']}") print(f" False Positives: {metrics['false_positive']}") print(f" True Negatives: {metrics['true_negative']}")
does it mean the dataset foemat could be the same as I use here to do inference? in particular, can I create a dataset from simply the processed inputs (thus, using current_batch_inputs = processor( images=batch_images, text=[fire_prompt] * len(batch_images), padding=True, return_tensors="pt" ).to(model.device))
does it mean I can use the processed data (after doing processor(...)) to create a dataset for fine tuning or should I create a dataset of a different format
Assistant
Okay, let's break down the Gaudi fine-tuning process for
llava-1.6-vicuna-7b.1. Can you use the processed
current_batch_inputsfrom your inference script for training?No, you cannot directly use the
current_batch_inputsgenerated during inference for training. Here's why:- Missing Labels: Training requires knowing the correct answer (the "labels") for each input. Your inference
current_batch_inputsonly contains the processed image and the prompt (the question). It doesn't contain the expected ground-truth answer. The model needs the answer to calculate the loss and learn. - Training Goal: The goal of training is to adjust the model's weights so that when given the image and the prompt (
<s>[INST] <image/>\nQuestion? [/INST]), it predicts the tokens corresponding to the correct answer (Answer</s>). - Inference Goal: The goal of inference is just to generate the answer based on the image and prompt.
Therefore, you absolutely need a dataset structured differently for training, one that explicitly pairs the question with its corresponding ground-truth answer.
2. Dataset Format for Fine-tuning
The dataset format remains the same as described previously for general LLaVA fine-tuning. You need a JSON Lines file (
.jsonl) where each line represents a data point containing the image path and the full conversation (question and answer).Example
dataset_gaudi.jsonl:{"id": "fire_sample_1", "image": "path/to/your/images/no_fire_scene_01.jpg", "conversations": [{"from": "human", "value": "<image/>\nYou are an expert in detecting wildfire smokes starting from their ignition. Can you see one in the image? Respond in simple 'YES' or 'NO'."}, {"from": "gpt", "value": "NO"}]} {"id": "fire_sample_2", "image": "path/to/your/images/early_fire_scene_05.jpg", "conversations": [{"from": "human", "value": "<image/>\nYou are an expert in detecting wildfire smokes starting from their ignition. Can you see one in the image? Respond in simple 'YES' or 'NO'."}, {"from": "gpt", "value": "YES"}]} {"id": "fire_sample_3", "image": "path/to/your/images/another_no_fire_02.png", "conversations": [{"from": "human", "value": "<image/>\nYou are an expert in detecting wildfire smokes starting from their ignition. Can you see one in the image? Respond in simple 'YES' or 'NO'."}, {"from": "gpt", "value": "NO"}]}Key Points:
- Use the exact prompt structure you intend to use during inference in the
"human"turn's"value". - Provide the correct ground-truth answer (e.g., "YES" or "NO") in the
"gpt"turn's"value". - Ensure the
imagepaths are correct relative to where you run the script, or use absolute paths. - The
<image/>token is crucial.
3. End-to-End Fine-tuning Script for Gaudi
This script uses
optimum-habanawhich provides Gaudi-optimized versions of the Hugging Face Trainer and utilities.Step 1: Install Libraries
# Install base libraries pip install torch torchvision torchaudio pip install transformers datasets accelerate Pillow # Install Gaudi specific libraries pip install optimum[habana]Step 2: Fine-tuning Script (
finetune_llava_gaudi.py)import os import json import torch from PIL import Image from datasets import load_dataset from transformers import ( AutoProcessor, LlavaForConditionalGeneration, # Or LlavaNextForConditionalGeneration if specifically needed/available TrainingArguments, DataCollatorForSeq2Seq # Might be needed for label padding ) from optimum.habana import GaudiConfig, GaudiTrainer, GaudiTrainingArguments from optimum.habana.transformers.modeling_utils import adapt_transformers_to_gaudi import habana_frameworks.torch.core as htcore # Ensure Gaudi environment is active # --- Configuration --- # Model ID from Hugging Face Hub or local path # Use the same base model you used for inference model_id = "/workspace/models/model_llava_v1_6_vicuna_7b" # Or "llava-hf/llava-1.6-vicuna-7b-hf" # Path to your training JSON Lines dataset file dataset_path = "dataset_gaudi.jsonl" # Directory where your images are stored (ensure paths in dataset.jsonl are relative to this or absolute) image_dir = "/workspace/Full_data/" # Adjust if needed, or use absolute paths in jsonl # Output directory to save the fine-tuned model output_dir = "./llava-1.6-vicuna-7b-finetuned-gaudi" # Training parameters training_epochs = 3 # Adjust as needed batch_size_per_hpu = 4 # Adjust based on Gaudi card memory gradient_accumulation_steps = 4 # Effective batch size = batch_size * grad_accum * num_hpus learning_rate = 1e-5 # Common starting point for LLaVA fine-tuning logging_steps = 20 save_steps = 200 # Save checkpoints periodically warmup_steps = 50 # Gaudi specific settings use_hpu_graphs = True # Enable HPU graphs for potential speedup use_lazy_mode = True # Standard for Gaudi training bf16_training = True # Use bfloat16 mixed precision (recommended for Gaudi 2) # --- Adapt Transformers for Gaudi --- print("Adapting transformers for Gaudi...") adapt_transformers_to_gaudi() print("Adaptation complete.") # --- Gaudi Configuration --- # You might need a specific Gaudi configuration, but often defaults work. # If needed, create a GaudiConfig object: # gaudi_config = GaudiConfig(use_fused_adam=True, use_fused_clip_norm=True) # And pass it to GaudiTrainingArguments via `gaudi_config_name` or `gaudi_config` # --- Load Model and Processor --- print(f"Loading processor from {model_id}...") processor = AutoProcessor.from_pretrained(model_id) # Ensure tokenizer padding side is correct for training/generation consistency # Right padding is generally preferred for decoder models during training processor.tokenizer.padding_side = "right" # Add pad token if missing (though Vicuna based models usually have it) if processor.tokenizer.pad_token is None: processor.tokenizer.pad_token = processor.tokenizer.eos_token print("Set pad_token to eos_token") print(f"Loading model from {model_id}...") model = LlavaForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.bfloat16 if bf16_training else torch.float32, # No device_map="auto" or low_cpu_mem_usage with GaudiTrainer, it handles placement ) print("Model and processor loaded.") # --- Load and Preprocess Dataset --- print("Loading dataset...") raw_dataset = load_dataset("json", data_files=dataset_path, split="train") # Define the prompt structure (should match inference) # Ensure this structure allows easy separation of prompt/answer for label masking # Vicuna structure: "<s>[INST] {system_prompt} {user_prompt} [/INST] {assistant_response}</s>" # LLaVA often uses: "USER: <image/>\n{question} ASSISTANT: {answer}" or similar. # Let's stick to the Vicuna format which `apply_chat_template` often defaults to for Vicuna-based models. # Crucially, the processor should handle the correct template application. # Preprocessing function IGNORE_INDEX = -100 # Standard ignore index for labels def preprocess_data(examples): processed_texts = [] images_list = [] full_labels = [] for i in range(len(examples['conversations'])): conv = examples['conversations'][i] image_path = examples['image'][i] # Construct full path if image_dir is provided and path is relative if image_dir and not os.path.isabs(image_path): full_image_path = os.path.join(image_dir, image_path) else: full_image_path = image_path if not os.path.exists(full_image_path): print(f"Warning: Image not found at {full_image_path}, skipping example.") continue try: image = Image.open(full_image_path).convert("RGB") except Exception as- Missing Labels: Training requires knowing the correct answer (the "labels") for each input. Your inference
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.