Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro-exp-03-25-grounding· GoogleHugging Face LMArena Search Arena 24K

provide an end to end code to fine tune llava-1.6-vicuna-7b on a custom dataset. Include…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.0-flash-grounding’s answer.
  1. User

    provide an end to end code to fine tune llava-1.6-vicuna-7b on a custom dataset. Include also how the dataset must be constructed (what format should it be in). The dataset is a visual question answering dataset

  2. Gemini Assistant

    Okay, let's set up the process for fine-tuning llava-1.6-vicuna-7b on your custom Visual Question Answering (VQA) dataset. This involves preparing the dataset in the correct format and then writing the Python script using Hugging Face's transformers library.

    Assumptions:

    • You have a machine with a suitable GPU (LLaVA-7B requires significant VRAM, often >16GB, especially for training).
    • You have Python installed, along with pip or conda.
    • Your images are stored in a directory.

    1. Dataset Construction and Format

    LLaVA models are typically trained on conversational data paired with images. For VQA fine-tuning, you need to structure your dataset similarly. The standard format is often a JSON Lines file (.jsonl), where each line is a JSON object representing one data point.

    Required Format:

    Each JSON object in your .jsonl file should contain:

    1. id: A unique identifier for the data point (optional but good practice).
    2. image: The relative or absolute path to the image file associated with this data point.
    3. conversations: A list of dictionaries, representing turns in a conversation. For VQA, this will typically be a two-turn conversation: the user's question and the assistant's answer. The structure must include the <image/> token where the image context is expected.

    Example dataset.jsonl file:

    {"id": "sample_1", "image": "images/cat_on_sofa.jpg", "conversations": [{"from": "human", "value": "<image/>\nWhat is the animal doing?"}, {"from": "gpt", "value": "The cat is sleeping on the sofa."}]}
    {"id": "sample_2", "image": "images/street_scene.png", "conversations": [{"from": "human", "value": "<image/>\nHow many cars are visible?"}, {"from": "gpt", "value": "There are three cars visible in the street scene."}]}
    {"id": "sample_3", "image": "path/to/your/images/kitchen.jpeg", "conversations": [{"from": "human", "value": "<image/>\nWhat color is the countertop?"}, {"from": "gpt", "value": "The countertop is black granite."}]}
    

    Key Points:

    • <image/> Token: The user's turn (from: human) must include the special <image/> token, usually followed by a newline (\n) and then the actual question text. This token signals to the model where to incorporate the image features.
    • from field: Use "human" for the user's question turn and "gpt" (or sometimes "assistant") for the model's answer turn. Consistency is important.
    • Image Paths: Ensure the paths in the image field correctly point to your image files relative to where you run the script, or use absolute paths.
    • File Structure: You'll typically have:
      • your_project_directory/
        • finetune_llava.py (The script below)
        • dataset.jsonl (Your data definitions)
        • images/ (Directory containing all your image files like cat_on_sofa.jpg, street_scene.png, kitchen.jpeg)

    2. End-to-End Fine-tuning Code

    This script uses Hugging Face transformers, datasets, accelerate, and bitsandbytes (optional, for memory efficiency like 4-bit training).

    Step 1: Install Libraries

    pip install transformers datasets accelerate bitsandbytes peft # PEFT for potential LoRA
    pip install torch torchvision torchaudio # Or install PyTorch matching your CUDA version
    pip install Pillow # For image loading
    

    Step 2: Fine-tuning Script (finetune_llava.py)

    import os
    import json
    import torch
    from PIL import Image
    from datasets import load_dataset
    from transformers import (
        AutoProcessor,
        LlavaForConditionalGeneration,
        TrainingArguments,
        Trainer,
        BitsAndBytesConfig # Optional: for 4-bit quantization
    )
    
    # --- Configuration ---
    # Model ID from Hugging Face Hub
    model_id = "llava-hf/llava-1.6-vicuna-7b-hf"
    # Path to your JSON Lines dataset file
    dataset_path = "dataset.jsonl"
    # Directory where your images are stored (ensure paths in dataset.jsonl are relative to this or absolute)
    # If paths in jsonl are absolute, this can be None or ""
    image_dir = "images/"
    # Output directory to save the fine-tuned model
    output_dir = "./llava-1.6-vicuna-7b-finetuned-vqa"
    # Training parameters
    training_epochs = 1
    batch_size = 4 # Adjust based on your GPU VRAM
    gradient_accumulation_steps = 4 # Effective batch size = batch_size * gradient_accumulation_steps
    learning_rate = 1e-5 # LLaVA authors suggest 1e-5 or 2e-5 for fine-tuning
    logging_steps = 50
    save_steps = 500 # Save checkpoints periodically
    # Optional: Use 4-bit quantization for memory saving (requires bitsandbytes)
    use_4bit_quantization = True # Set to False if you have enough VRAM or encounter issues
    
    # --- Quantization Configuration (Optional) ---
    bnb_config = None
    if use_4bit_quantization:
        bnb_config = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_quant_type="nf4",
            bnb_4bit_compute_dtype=torch.float16 # Or torch.bfloat16 if supported
        )
    
    # --- Load Model and Processor ---
    print("Loading model and processor...")
    processor = AutoProcessor.from_pretrained(model_id)
    model = LlavaForConditionalGeneration.from_pretrained(
        model_id,
        torch_dtype=torch.float16, # Use float16 for faster training and less memory
        low_cpu_mem_usage=True,    # Load model layers incrementally
        quantization_config=bnb_config, # Apply quantization if configured
        device_map="auto" # Automatically distribute model across available GPUs
    )
    print("Model and processor loaded.")
    
    # --- Prepare Chat Template ---
    # LLaVA uses a specific chat template (usually Vicuna). The processor should handle this,
    # but we ensure the tokenizer knows about the '<image/>' token.
    # The processor should already add special tokens if they aren't there.
    # We just need to be sure padding is handled correctly.
    processor.tokenizer.padding_side = "right" # Ensure padding is on the right
    
    # --- Load and Preprocess Dataset ---
    print("Loading dataset...")
    # Load from JSON Lines file
    raw_dataset = load_dataset("json", data_files=dataset_path, split="train")
    
    # Preprocessing function
    def preprocess_data(examples):
        processed_texts = []
        image_paths = []
    
        # Construct the conversation text in the required format for the processor
        for i in range(len(examples['conversations'])):
            # Assumes structure like: [{"from": "human", "value": "<image/>\nQuestion?"}, {"from": "gpt", "value": "Answer."}]
            conversation = examples['conversations'][i]
            if len(conversation) == 2 and conversation[0]['from'] == 'human' and conversation[1]['from'] == 'gpt':
                # Format using the chat template implicitly applied by the processor
                # The processor expects a chat structure or raw text + images
                # Let's create the text that includes the prompt and the answer for tokenization
                # The processor's chat template will typically format this correctly
                # Example structure needed by processor might look like:
                # "<s>[INST] <image/>\n{question} [/INST] {answer} </s>" (depending on the base model, e.g., Vicuna)
    
                question = conversation[0]['value'] # e.g., "<image/>\nWhat is this?"
                answer = conversation[1]['value']   # e.g., "This is a cat."
    
                # The processor expects the prompt text for processing.
                # We need to format the full conversation for input_ids and labels.
                # The processor.apply_chat_template might be useful if available and correctly configured for LLaVA.
                # Let's manually construct the full text for tokenization:
                # NOTE: Check the exact template LLaVA 1.6 Vicuna expects. Often it's something like:
                # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image/>\n{question} ASSISTANT: {answer}"
                # Or using the more common Vicuna structure:
                prompt = f"<s>[INST] {question} [/INST]" # Part the model conditions on
                full_text = f"<s>[INST] {question} [/INST] {answer}</s>" # Full sequence including answer
    
                processed_texts.append(full_text)
                # Construct the full image path
                img_path = os.path.join(image_dir, examples['image'][i]) if image_dir else examples['image'][i]
                image_paths.append(img_path)
            else:
                print(f"Skipping invalid conversation structure: {conversation}")
                # Add placeholders to maintain batch structure, will be filtered later if necessary
                processed_texts.append("")
                image_paths.append(None)
    
        # Load images - Handle potential errors
        images = []
        valid_indices = []
        for idx, path in enumerate(image_paths):
            if path and os.path.exists(path):
                try:
                    images.append(Image.open(path).convert("RGB"))
                    valid_indices.append(idx)
                except Exception as e:
                    print(f"Warning: Could not load image {path}:</s>
    
  3. User

    if the followinf is the script that I use for inference using llava model: as this is the way I do inference on the modek: say I am using llava-1.6-vicuna-7b, instead of 13b, provide end to end fine tuning script on gaudi, that I can run by simply copy and pasting the code snippet you provide.

    here is a sample code for running an inference if you might need that import habana_frameworks.torch as ht import habana_frameworks.torch.core as htcore

    from optimum.habana.transformers.modeling_utils import adapt_transformers_to_gaudi adapt_transformers_to_gaudi()

    from transformers import LlavaNextProcessor, AutoProcessor, AutoConfig, LlavaNextForConditionalGeneration import torch from PIL import Image import requests import os import time import re

    device = torch.device("hpu") args_model_name_or_path = "/workspace/models/model_llava_v1_6_vicuna_7b" model_type = AutoConfig.from_pretrained(args_model_name_or_path).model_type

    print("Loading the processor") processor = AutoProcessor.from_pretrained(args_model_name_or_path)

    print("Loading the model") model = LlavaNextForConditionalGeneration.from_pretrained( args_model_name_or_path, torch_dtype=torch.bfloat16, ) model.to("hpu")

    fire_sequences_root = "/workspace/Full_data/test_sequences"

    Initialize results dictionary

    prediction_results = {}

    Get all fire sequence folders

    fire_sequence_folders = sorted([ os.path.join(fire_sequences_root, folder) for folder in os.listdir(fire_sequences_root) if os.path.isdir(os.path.join(fire_sequences_root, folder)) ])

    print(f"Found {len(fire_sequence_folders)} fire sequence folders.")

    Start processing each fire sequence folder

    start = time.perf_counter()

    batch_size = 4 generate_kwargs = {"max_new_tokens": 200, "do_sample": False} prediction_results_per_image_file = {}

    Initialize confusion matrix metrics

    metrics = { "true_positive": 0, "false_negative": 0, "false_positive": 0, "true_negative": 0 }

    Setting up the conversation for the wildfire detection (original prompt)

    fire_conversation = [ { "role": "user", "content": [ {"type": "text", "text": "You are an expert in detecting wildfire smokes starting from their ignition. Can you see one in the image? Respond in simple 'YES' or 'NO'. No explanation needed."}, {"type": "image"}, ], } ]

    #With description prompt

    fire_conversation = [

    {

    "role": "user",

    "content": [

    {"type": "text", "text": "You are an expert in detecting wildfire smokes starting from their ignition. Can you see one in the image? Respond in simple 'YES' or 'NO' as the first sentence. In the second sentence provide a brief description of what you see."},

    {"type": "image"},

    ],

    }

    ]

    fire_prompt = processor.apply_chat_template(fire_conversation, add_generation_prompt=True)

    for fire_sequence_folder in fire_sequence_folders: fire_sequence_name = os.path.basename(fire_sequence_folder)

    prediction_results[fire_sequence_name] = {
        "original": {"before": 0, "after": 0},
        "prediction": {"before": 0, "after": 0}
    }
    
    fire_image_paths = sorted([
        os.path.join(fire_sequence_folder, file)
        for file in os.listdir(fire_sequence_folder)
        if file.endswith(".jpg")
    ])
    
    print(f"Processing {fire_sequence_name} with {len(fire_image_paths)} images...")
    
    # Batch processing
    for idx in range(0, len(fire_image_paths), batch_size):
        batch_images = [Image.open(image_path) for image_path in fire_image_paths[idx : idx + batch_size]]
        batch_paths = fire_image_paths[idx : idx + batch_size]
    
        current_batch_inputs = processor(
            images=batch_images, 
            text=[fire_prompt] * len(batch_images), 
            padding=True, 
            return_tensors="pt"
        ).to(model.device)
    
        current_batch_generate_ids = model.generate(**current_batch_inputs, max_new_tokens=30)
    
        current_batch_results = processor.batch_decode(
            current_batch_generate_ids, 
            skip_special_tokens=True, 
            clean_up_tokenization_spaces=False
        )
    
        for image_path, result in zip(batch_paths, current_batch_results):
            current_fire_image_path = os.path.basename(image_path)
            processed_result = result.split("ASSISTANT")[-1].strip().upper()
            print(f"File: {current_fire_image_path}, Result: {processed_result}")
    
            is_fire = "+" in current_fire_image_path
            prediction_is_fire = "YES" in processed_result
            
            # Update confusion matrix metrics
            if is_fire and prediction_is_fire:
                metrics["true_positive"] += 1
            elif is_fire and not prediction_is_fire:
                metrics["false_negative"] += 1
            elif not is_fire and prediction_is_fire:
                metrics["false_positive"] += 1
            elif not is_fire and not prediction_is_fire:
                metrics["true_negative"] += 1
    
            if is_fire:
                prediction_results[fire_sequence_name]["original"]["after"] += 1
            else:
                prediction_results[fire_sequence_name]["original"]["before"] += 1
    
            if prediction_is_fire:
                prediction_results[fire_sequence_name]["prediction"]["after"] += 1
            else:
                prediction_results[fire_sequence_name]["prediction"]["before"] += 1
    
            prediction_results_per_image_file[image_path] = processed_result
    

    end = time.perf_counter() duration = end - start print(f"Total duration: {duration:.2f} seconds")

    Print summary of results

    for seq_name, results in prediction_results.items(): print(f"\nSummary for {seq_name}:") print(f" Original - Before: {results['original']['before']}, After: {results['original']['after']}") print(f" Prediction - Before: {results['prediction']['before']}, After: {results['prediction']['after']}")

    Print confusion matrix metrics

    print("\nConfusion Matrix Metrics:") print(f" True Positives: {metrics['true_positive']}") print(f" False Negatives: {metrics['false_negative']}") print(f" False Positives: {metrics['false_positive']}") print(f" True Negatives: {metrics['true_negative']}")

    does it mean the dataset foemat could be the same as I use here to do inference? in particular, can I create a dataset from simply the processed inputs (thus, using current_batch_inputs = processor( images=batch_images, text=[fire_prompt] * len(batch_images), padding=True, return_tensors="pt" ).to(model.device))

    does it mean I can use the processed data (after doing processor(...)) to create a dataset for fine tuning or should I create a dataset of a different format

  4. Gemini Assistant

    Okay, let's break down the Gaudi fine-tuning process for llava-1.6-vicuna-7b.

    1. Can you use the processed current_batch_inputs from your inference script for training?

    No, you cannot directly use the current_batch_inputs generated during inference for training. Here's why:

    • Missing Labels: Training requires knowing the correct answer (the "labels") for each input. Your inference current_batch_inputs only contains the processed image and the prompt (the question). It doesn't contain the expected ground-truth answer. The model needs the answer to calculate the loss and learn.
    • Training Goal: The goal of training is to adjust the model's weights so that when given the image and the prompt (<s>[INST] <image/>\nQuestion? [/INST]), it predicts the tokens corresponding to the correct answer (Answer</s>).
    • Inference Goal: The goal of inference is just to generate the answer based on the image and prompt.

    Therefore, you absolutely need a dataset structured differently for training, one that explicitly pairs the question with its corresponding ground-truth answer.

    2. Dataset Format for Fine-tuning

    The dataset format remains the same as described previously for general LLaVA fine-tuning. You need a JSON Lines file (.jsonl) where each line represents a data point containing the image path and the full conversation (question and answer).

    Example dataset_gaudi.jsonl:

    {"id": "fire_sample_1", "image": "path/to/your/images/no_fire_scene_01.jpg", "conversations": [{"from": "human", "value": "<image/>\nYou are an expert in detecting wildfire smokes starting from their ignition. Can you see one in the image? Respond in simple 'YES' or 'NO'."}, {"from": "gpt", "value": "NO"}]}
    {"id": "fire_sample_2", "image": "path/to/your/images/early_fire_scene_05.jpg", "conversations": [{"from": "human", "value": "<image/>\nYou are an expert in detecting wildfire smokes starting from their ignition. Can you see one in the image? Respond in simple 'YES' or 'NO'."}, {"from": "gpt", "value": "YES"}]}
    {"id": "fire_sample_3", "image": "path/to/your/images/another_no_fire_02.png", "conversations": [{"from": "human", "value": "<image/>\nYou are an expert in detecting wildfire smokes starting from their ignition. Can you see one in the image? Respond in simple 'YES' or 'NO'."}, {"from": "gpt", "value": "NO"}]}
    

    Key Points:

    • Use the exact prompt structure you intend to use during inference in the "human" turn's "value".
    • Provide the correct ground-truth answer (e.g., "YES" or "NO") in the "gpt" turn's "value".
    • Ensure the image paths are correct relative to where you run the script, or use absolute paths.
    • The <image/> token is crucial.

    3. End-to-End Fine-tuning Script for Gaudi

    This script uses optimum-habana which provides Gaudi-optimized versions of the Hugging Face Trainer and utilities.

    Step 1: Install Libraries

    # Install base libraries
    pip install torch torchvision torchaudio
    pip install transformers datasets accelerate Pillow
    
    # Install Gaudi specific libraries
    pip install optimum[habana]
    

    Step 2: Fine-tuning Script (finetune_llava_gaudi.py)

    import os
    import json
    import torch
    from PIL import Image
    from datasets import load_dataset
    from transformers import (
        AutoProcessor,
        LlavaForConditionalGeneration, # Or LlavaNextForConditionalGeneration if specifically needed/available
        TrainingArguments,
        DataCollatorForSeq2Seq # Might be needed for label padding
    )
    from optimum.habana import GaudiConfig, GaudiTrainer, GaudiTrainingArguments
    from optimum.habana.transformers.modeling_utils import adapt_transformers_to_gaudi
    import habana_frameworks.torch.core as htcore # Ensure Gaudi environment is active
    
    # --- Configuration ---
    # Model ID from Hugging Face Hub or local path
    # Use the same base model you used for inference
    model_id = "/workspace/models/model_llava_v1_6_vicuna_7b" # Or "llava-hf/llava-1.6-vicuna-7b-hf"
    # Path to your training JSON Lines dataset file
    dataset_path = "dataset_gaudi.jsonl"
    # Directory where your images are stored (ensure paths in dataset.jsonl are relative to this or absolute)
    image_dir = "/workspace/Full_data/" # Adjust if needed, or use absolute paths in jsonl
    # Output directory to save the fine-tuned model
    output_dir = "./llava-1.6-vicuna-7b-finetuned-gaudi"
    # Training parameters
    training_epochs = 3       # Adjust as needed
    batch_size_per_hpu = 4    # Adjust based on Gaudi card memory
    gradient_accumulation_steps = 4 # Effective batch size = batch_size * grad_accum * num_hpus
    learning_rate = 1e-5      # Common starting point for LLaVA fine-tuning
    logging_steps = 20
    save_steps = 200          # Save checkpoints periodically
    warmup_steps = 50
    # Gaudi specific settings
    use_hpu_graphs = True    # Enable HPU graphs for potential speedup
    use_lazy_mode = True     # Standard for Gaudi training
    bf16_training = True     # Use bfloat16 mixed precision (recommended for Gaudi 2)
    
    # --- Adapt Transformers for Gaudi ---
    print("Adapting transformers for Gaudi...")
    adapt_transformers_to_gaudi()
    print("Adaptation complete.")
    
    # --- Gaudi Configuration ---
    # You might need a specific Gaudi configuration, but often defaults work.
    # If needed, create a GaudiConfig object:
    # gaudi_config = GaudiConfig(use_fused_adam=True, use_fused_clip_norm=True)
    # And pass it to GaudiTrainingArguments via `gaudi_config_name` or `gaudi_config`
    
    # --- Load Model and Processor ---
    print(f"Loading processor from {model_id}...")
    processor = AutoProcessor.from_pretrained(model_id)
    # Ensure tokenizer padding side is correct for training/generation consistency
    # Right padding is generally preferred for decoder models during training
    processor.tokenizer.padding_side = "right"
    # Add pad token if missing (though Vicuna based models usually have it)
    if processor.tokenizer.pad_token is None:
        processor.tokenizer.pad_token = processor.tokenizer.eos_token
        print("Set pad_token to eos_token")
    
    print(f"Loading model from {model_id}...")
    model = LlavaForConditionalGeneration.from_pretrained(
        model_id,
        torch_dtype=torch.bfloat16 if bf16_training else torch.float32,
        # No device_map="auto" or low_cpu_mem_usage with GaudiTrainer, it handles placement
    )
    print("Model and processor loaded.")
    
    # --- Load and Preprocess Dataset ---
    print("Loading dataset...")
    raw_dataset = load_dataset("json", data_files=dataset_path, split="train")
    
    # Define the prompt structure (should match inference)
    # Ensure this structure allows easy separation of prompt/answer for label masking
    # Vicuna structure: "<s>[INST] {system_prompt} {user_prompt} [/INST] {assistant_response}</s>"
    # LLaVA often uses: "USER: <image/>\n{question} ASSISTANT: {answer}" or similar.
    # Let's stick to the Vicuna format which `apply_chat_template` often defaults to for Vicuna-based models.
    # Crucially, the processor should handle the correct template application.
    
    # Preprocessing function
    IGNORE_INDEX = -100 # Standard ignore index for labels
    
    def preprocess_data(examples):
        processed_texts = []
        images_list = []
        full_labels = []
    
        for i in range(len(examples['conversations'])):
            conv = examples['conversations'][i]
            image_path = examples['image'][i]
    
            # Construct full path if image_dir is provided and path is relative
            if image_dir and not os.path.isabs(image_path):
                full_image_path = os.path.join(image_dir, image_path)
            else:
                full_image_path = image_path
    
            if not os.path.exists(full_image_path):
                print(f"Warning: Image not found at {full_image_path}, skipping example.")
                continue
    
            try:
                image = Image.open(full_image_path).convert("RGB")
            except Exception as
    

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.