Skip to content
TheChatArchiveThe Chat Archive
Amazon Novaamazon-nova-experimental-chat-05-14· AmazonHugging Face LMArena Human Preference 140K

When I use Salesforce/blip-image-captioning-large should I scale the images to some…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-opus-4-20250514-thinking-16k’s answer.
  1. User

    When I use Salesforce/blip-image-captioning-large should I scale the images to some standard format before feeding them in? I now do

    processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
    model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
    raw_image = Image.open(img_path).convert("RGB")
    inputs = processor(raw_image, "return_tensors="pt")
    out = model.generate(**inputs)
    caption = processor.decode(out[0], skip_special_tokens=True)
    
  2. Amazon Nova Assistant

    When using the Salesforce/blip-image-captioning-large model for image captioning, it is essential to preprocess the images correctly to ensure optimal performance. Let's go through the necessary steps and address whether you should scale the images to a standard format.

    Image Preprocessing

    The BlipProcessor handles the necessary preprocessing for the images. Here’s a detailed explanation of what happens when you use the processor:

    1. Resizing: The processor will automatically resize the image to the model's expected input size. For the Salesforce/blip-image-captioning-large model the expected input size is 384x384. If the input image is not 384x384 the processor will resize it maintaining the aspect ratio and then centre crop it to 384x384.
    2. Normalisation: The processor will also normalise the pixel values so they are in the correct range for the model.

    Given this, you do not need to manually resize or scale the images to a standard format before feeding them into the processor. The BlipProcessor will take care of resizing and any other necessary preprocessing steps.

    Corrected Code

    Your code is almost correct. Here is the refined version with some minor adjustments for clarity:

    from PIL import Image
    from transformers import BlipProcessor, BlipForConditionalGeneration
    
    # Load processor and model
    processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
    model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
    
    # Open and preprocess the image
    raw_image = Image.open(img_path).convert("RGB")
    inputs = processor(images=raw_image, return_tensors="pt")
    
    # Generate caption
    out = model.generate(**inputs)
    caption = processor.decode(out[0], skip_special_tokens=True)
    
    print(caption)
    

    Key Points:

    1. Image Opening: You correctly open the image and convert it to RGB.
    2. Processor Input: When calling the processor, use images=raw_image to explicitly specify the image input. This is not necessary but improves readability
    3. Return Tensors: You had a minor typo in your code "return_tensors="pt" should be return_tensors="pt
    4. Automatic Resizing: The processor automatically resizes the image to 384x384, so manual resizing is unnecessary.
    5. Generate and Decode: The generation and decoding steps are correctly implemented.

    By relying on the BlipProcessor, you ensure that the image is properly preprocessed to meet the model's requirements. Therefore you should not manually resize the image.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.