Skip to content
The Chat Archive
gpt-4.1-2025-04-14· OpenAI LMArena Human Preference 140K

enumerate all text-based image retrieval datasets

  1. User

    enumerate all text-based image retrieval datasets

  2. Assistant

    Certainly! Here’s a comprehensive list of text-based image retrieval datasets (also called image-text retrieval or cross-modal retrieval datasets). These datasets are commonly used for evaluating and training models that retrieve images based on textual queries (and vice versa).


    General-Purpose Datasets

    1. MS COCO (Microsoft Common Objects in Context)

    • Description: 330K images, each with 5 captions.
    • Use: Image-text retrieval, captioning, VQA.
    • Link: MS COCO

    2. Flickr30K

    • Description: 31K images, each with 5 captions.
    • Use: Image-text retrieval, captioning.
    • Link: Flickr30K

    3. Flickr8K

    • Description: 8K images, each with 5 captions.
    • Use: Smaller-scale retrieval, captioning.
    • Link: Flickr8K

    4. Conceptual Captions

    • Description: 3.3M images with captions harvested from the web.
    • Use: Large-scale pretraining, retrieval.
    • Link: Conceptual Captions

    5. SBU Captions

    • Description: 1M images with captions from Flickr.
    • Use: Large-scale pretraining, retrieval.
    • Link: SBU Captions

    6. Visual Genome

    • Description: 108K images with region descriptions, Q&A, and relationships.
    • Use: Dense captioning, retrieval, VQA.
    • Link: Visual Genome

    7. Google Conceptual Captions 12M (CC12M)

    • Description: 12M image-caption pairs.
    • Use: Large-scale pretraining, retrieval.
    • Link: CC12M

    8. YFCC100M (Yahoo Flickr Creative Commons 100M)

    • Description: 100M images/videos with metadata and user tags.
    • Use: Large-scale retrieval, weakly supervised learning.
    • Link: YFCC100M

    Specialized/Challenging Datasets

    9. RSICD (Remote Sensing Image Caption Dataset)

    • Description: 10,921 remote sensing images, each with 5 captions.
    • Use: Remote sensing retrieval.
    • Link: RSICD

    10. Fashion-Gen

    • Description: 293K fashion images with detailed descriptions.
    • Use: Fashion retrieval, generation.
    • Link: Fashion-Gen

    11. CUHK-PEDES

    • Description: 40,206 images of 13,003 persons, each with 2 sentences.
    • Use: Person search by natural language.
    • Link: CUHK-PEDES

    12. Oxford Flowers-102 Captions

    • Description: 8,189 images of 102 flower categories, each with 10 captions.
    • Use: Fine-grained retrieval.
    • Link: Oxford Flowers-102

    13. CUB-200-2011 with Captions

    • Description: 11,788 bird images, each with 10 captions.
    • Use: Fine-grained retrieval.
    • Link: CUB-200-2011

    14. Recipe1M

    • Description: 1M recipes with images and text.
    • Use: Cross-modal retrieval (food domain).
    • Link: Recipe1M

    15. MIR Flickr

    • Description: 25K images with user tags and descriptions.
    • Use: Multi-modal retrieval.
    • Link: MIR Flickr

    Recent Large-Scale/Noisy Datasets

    16. LAION-400M / LAION-5B

    • Description: 400M/5B image-text pairs scraped from the web.
    • Use: Large-scale pretraining, retrieval.
    • Link: LAION

    17. RedCaps

    • Description: 12M image-caption pairs from Reddit.
    • Use: Large-scale pretraining, retrieval.
    • Link: RedCaps

    Other Notable Datasets

    18. VG-COCO (Visual Genome + COCO)

    • Description: Combines region descriptions from Visual Genome with COCO images.
    • Use: Dense retrieval.

    19. TextCaps

    • Description: 145K images with captions focusing on text in images.
    • Use: Scene text retrieval.
    • Link: TextCaps

    20. MSR-VTT

    • Description: 10K video clips with 200K captions.
    • Use: Video-text retrieval.
    • Link: MSR-VTT

    Summary Table

    Dataset# Images/PairsCaptions per ImageDomainLink
    MS COCO330K5GeneralCOCO
    Flickr30K31K5GeneralFlickr30K
    Flickr8K8K5GeneralFlickr8K
    Conceptual Captions3.3M1WebConceptual Captions
    SBU Captions1M1WebSBU Captions
    Visual Genome108KDense regionsGeneralVisual Genome
    CC12M12M1WebCC12M
    YFCC100M100MTags/metadataWebYFCC100M
    RSICD10.9K5Remote SensingRSICD
    Fashion-Gen293K1+FashionFashion-Gen
    CUHK-PEDES40K2PersonCUHK-PEDES
    Oxford Flowers8K10FlowersOxford Flowers-102
    CUB-200-201111.8K10BirdsCUB-200-2011
    Recipe1M1M1+FoodRecipe1M
    MIR Flickr25KTags/desc.GeneralMIR Flickr
    LAION-400M/5B400M/5B1WebLAION
    RedCaps12M1WebRedCaps
    TextCaps145K1Scene TextTextCaps
    MSR-VTT10K videos20VideoMSR-VTT

    If you need a more specific list (e.g., only for English, only for fine-grained, etc.), let me know!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.