cider score computing library
- User
cider score computing library
- Assistant
CIDEr Score Computing Libraries
CIDEr (Consensus-based Image Description Evaluation) is a widely used metric for evaluating image captioning systems that captures the consensus among human-written reference captions. Here are the main libraries and tools available for computing CIDEr scores:
Python Implementations
1. COCO Caption Evaluation
The most widely used implementation comes from the MS COCO caption evaluation tools:
- Repository: tylin/coco-caption
- Python 3 Fork: Aldenhovel/bleu-rouge-meteor-cider-spice-eval4imagecaption[5]
- Features:
- Provides both CIDEr and CIDEr-D metrics
- CIDEr-D is a more robust version that defends against "gaming effects"
- Requires specific JSON format for references and candidates
To use this library, you need to format your references and captions into specific JSON structures:
# Create JSON files for evaluation references = { "1": ["this is a tree", "this is an apple", ...], "2": ["a man is sitting", "a man in the street", ...], } captions = { "1": ["this is a big tree"], "2": ["a man is sitting"], } # Convert to proper format import json new_cap = [] for k, v in captions.items(): new_cap.append({'image_id': k, 'caption': v}) new_ref = {'images': [], 'annotations': []} for k, refs in references.items(): new_ref['images'].append({'id': k}) for ref in refs: new_ref['annotations'].append({'image_id': k, 'id': k, 'caption': ref}) with open('references.json', 'w') as fgts: json.dump(new_ref, fgts) with open('captions.json', 'w') as fres: json.dump(new_cap, fres)2. daqingliu/cider
A standalone Python 3 implementation of CIDEr:
- Repository: daqingliu/cider[1]
- Features:
- Provides both CIDEr and CIDEr-D
- Important note: CIDEr by default computes IDF values using reference sentences provided, meaning CIDEr score for a dataset with only 1 image will be zero
- When evaluating with few images, set IDF to "coco-val-df" instead, which uses IDF from MSCOCO Validation Dataset[1]
3. aac-metrics Package
A PyPI package for audio captioning metrics that includes CIDEr-D:
- Installation:
pip install aac-metrics - Usage:
from aac_metrics.functional import cider_d from aac_metrics.utils.tokenization import preprocess_mono_sents, preprocess_mult_sents candidates = ["a man is speaking", "rain falls"] mult_references = [ ["a man speaks.", "someone speaks.", "a man is speaking while a bird is chirping in the background"], ["rain is falling hard on a surface"] ] candidates = preprocess_mono_sents(candidates) mult_references = preprocess_mult_sents(mult_references) corpus_scores, sents_scores = cider_d(candidates, mult_references) print(corpus_scores) # {"cider_d": tensor(0.9614)}R Implementation
CIDER Package
The CIDER R package appears to be focused on evaluating single-cell RNA-seq integration quality rather than image captions[2][3]. This is a different tool despite the similar name.
Technical Details
CIDEr calculates the consensus between candidate caption and reference captions using:
- TF-IDF weighting for n-grams
- Cosine similarity between the TF-IDF vectors
- Range: [0, 10] with higher values indicating better captions
The difference between CIDEr and CIDEr-D:
- CIDEr applies a stemmer to each word before computing n-grams
- CIDEr-D is more robust against "gaming effects" and is the version used in most benchmarks[4]
Benchmarking
CIDEr-D is available as part of the MS COCO evaluation server to enable systematic evaluation and benchmarking of image captioning systems[8]. This allows for standardized comparison between different approaches.
When using these libraries for evaluation, ensure you have the required dependencies:
- Java 1.8.0 (for METEOR, SPICE, and PTBTokenizer functionality)
- Python 3.6 or later
- For aac-metrics: torch, numpy, pyyaml, tqdm, sentence-transformers, transformers, torchmetrics
CIDEr has become a standard metric in image captioning evaluation, often used alongside other metrics like BLEU, ROUGE-L, METEOR, and SPICE.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.