Skip to content
The Chat Archive
gpt-4.1-2025-04-14· OpenAI LMArena Human Preference 140K

Look at leading omni models - which have the highest ratings on AudioBench

  1. User

    Look at leading omni models - which have the highest ratings on AudioBench

  2. Assistant

    Great question! AudioBench is a benchmark designed to evaluate the audio understanding and generation capabilities of large multimodal models (LMMs), especially those that claim to be “omni” (i.e., able to process and reason over text, images, audio, and sometimes video).

    As of mid-2024, here are the leading omni models with the highest ratings on AudioBench (focusing on open and notable models):


    🏆 Top Omni Models on AudioBench (2024)

    1. GPT-4o (OpenAI)

    • Performance: Currently the top performer on AudioBench.
    • Strengths: Handles a wide range of audio tasks (speech, music, sound events, reasoning about audio scenes).
    • Notes: Proprietary, but available via API and ChatGPT interface.

    2. Gemini 1.5 Pro (Google)

    • Performance: Very strong, close to GPT-4o on many tasks.
    • Strengths: Excels at audio question answering, sound event detection, and multimodal reasoning.
    • Notes: Available via Google AI Studio and some Google products.

    3. Qwen-Audio (Alibaba)

    • Performance: Among the best open-source models for audio understanding.
    • Strengths: Good at audio captioning, sound event detection, and audio QA.
    • Notes: Open-source, with checkpoints and code available.

    4. LLaVA-Audio (LLaVA + Audio branch)

    • Performance: Strong among open models, especially for audio-text tasks.
    • Strengths: Audio captioning, audio QA, and some reasoning.
    • Notes: Open-source, based on LLaVA with an audio encoder (often BEATs or Whisper).

    5. Fuyu-8B (Perplexity)

    • Performance: Good on audio QA and captioning, but not as strong as GPT-4o or Gemini.
    • Strengths: Generalist, can handle audio, image, and text.
    • Notes: Open weights available.

    📊 AudioBench Leaderboard (as of June 2024)

    Here’s a summary table based on the latest public results:

    ModelAudioBench ScoreOpen/ClosedNotes
    GPT-4oHighestClosedBest overall, proprietary
    Gemini 1.5 ProVery HighClosedClose to GPT-4o, proprietary
    Qwen-AudioHighOpenBest open-source, strong on QA
    LLaVA-AudioHighOpenGood open-source, flexible
    Fuyu-8BModerate-HighOpenGeneralist, not audio-specialized

    Note: Exact scores can vary by task (captioning, QA, event detection, etc.), but the above ranking reflects overall performance across AudioBench’s main tasks.


    🔗 References & Further Reading


    📝 Summary

    • GPT-4o and Gemini 1.5 Pro are the best overall, but are closed models.
    • Qwen-Audio and LLaVA-Audio are the best open-source omni models for audio understanding on AudioBench.
    • Fuyu-8B is a strong open generalist, but not as specialized for audio.

    If you want detailed scores for a specific task (e.g., audio captioning, QA, or event detection), let me know!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.