Skip to content
The Chat Archive
sonar-reasoning-pro-high LMArena Search Arena 24K

cider score computing library

  1. User

    cider score computing library

  2. Assistant

    CIDEr Score Computing Libraries

    CIDEr (Consensus-based Image Description Evaluation) is a widely used metric for evaluating image captioning systems that captures the consensus among human-written reference captions. Here are the main libraries and tools available for computing CIDEr scores:

    Python Implementations

    1. COCO Caption Evaluation

    The most widely used implementation comes from the MS COCO caption evaluation tools:

    • Repository: tylin/coco-caption
    • Python 3 Fork: Aldenhovel/bleu-rouge-meteor-cider-spice-eval4imagecaption[5]
    • Features:
      • Provides both CIDEr and CIDEr-D metrics
      • CIDEr-D is a more robust version that defends against "gaming effects"
      • Requires specific JSON format for references and candidates

    To use this library, you need to format your references and captions into specific JSON structures:

    # Create JSON files for evaluation
    references = {
        "1": ["this is a tree", "this is an apple", ...], 
        "2": ["a man is sitting", "a man in the street", ...],
    }
    captions = {
        "1": ["this is a big tree"],
        "2": ["a man is sitting"],
    }
    
    # Convert to proper format
    import json
    new_cap = []
    for k, v in captions.items():
        new_cap.append({'image_id': k, 'caption': v})
    
    new_ref = {'images': [], 'annotations': []}
    for k, refs in references.items():
        new_ref['images'].append({'id': k})
        for ref in refs:
            new_ref['annotations'].append({'image_id': k, 'id': k, 'caption': ref})
    
    with open('references.json', 'w') as fgts:
        json.dump(new_ref, fgts)
    with open('captions.json', 'w') as fres:
        json.dump(new_cap, fres)
    

    2. daqingliu/cider

    A standalone Python 3 implementation of CIDEr:

    • Repository: daqingliu/cider[1]
    • Features:
      • Provides both CIDEr and CIDEr-D
      • Important note: CIDEr by default computes IDF values using reference sentences provided, meaning CIDEr score for a dataset with only 1 image will be zero
      • When evaluating with few images, set IDF to "coco-val-df" instead, which uses IDF from MSCOCO Validation Dataset[1]

    3. aac-metrics Package

    A PyPI package for audio captioning metrics that includes CIDEr-D:

    • Installation: pip install aac-metrics
    • Usage:
    from aac_metrics.functional import cider_d
    from aac_metrics.utils.tokenization import preprocess_mono_sents, preprocess_mult_sents
    
    candidates = ["a man is speaking", "rain falls"]
    mult_references = [
        ["a man speaks.", "someone speaks.", "a man is speaking while a bird is chirping in the background"], 
        ["rain is falling hard on a surface"]
    ]
    
    candidates = preprocess_mono_sents(candidates)
    mult_references = preprocess_mult_sents(mult_references)
    
    corpus_scores, sents_scores = cider_d(candidates, mult_references)
    print(corpus_scores)  # {"cider_d": tensor(0.9614)}
    

    R Implementation

    CIDER Package

    The CIDER R package appears to be focused on evaluating single-cell RNA-seq integration quality rather than image captions[2][3]. This is a different tool despite the similar name.

    Technical Details

    CIDEr calculates the consensus between candidate caption and reference captions using:

    1. TF-IDF weighting for n-grams
    2. Cosine similarity between the TF-IDF vectors
    3. Range: [0, 10] with higher values indicating better captions

    The difference between CIDEr and CIDEr-D:

    • CIDEr applies a stemmer to each word before computing n-grams
    • CIDEr-D is more robust against "gaming effects" and is the version used in most benchmarks[4]

    Benchmarking

    CIDEr-D is available as part of the MS COCO evaluation server to enable systematic evaluation and benchmarking of image captioning systems[8]. This allows for standardized comparison between different approaches.

    When using these libraries for evaluation, ensure you have the required dependencies:

    • Java 1.8.0 (for METEOR, SPICE, and PTBTokenizer functionality)
    • Python 3.6 or later
    • For aac-metrics: torch, numpy, pyyaml, tqdm, sentence-transformers, transformers, torchmetrics

    CIDEr has become a standard metric in image captioning evaluation, often used alongside other metrics like BLEU, ROUGE-L, METEOR, and SPICE.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.