Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

Can you suggests some performance metrics for text-to-speech models?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-5-sonnet-20241022’s answer.
  1. User

    Can you suggests some performance metrics for text-to-speech models?

  2. Assistant

    Below is a compact “menu” of the most common (and most useful) ways people judge the quality of a Text-to-Speech (TTS) system today.
    I’ve broken them into five buckets so you can quickly pick what matters for your project.


    1. Human-based (Subjective) Metrics 🧑‍⚖️

    MetricWhat it measuresHow it’s collectedTypical scale
    MOS (Mean Opinion Score)Overall naturalness / pleasantnessListeners rate 1–5 (“Bad”→“Excellent”)1 – 5
    CMOS (Comparative MOS)“Which sample is better?” vs. a baselineListeners pick A/B, difference mapped to –3…+3–3 – +3
    MUSHRAFine-grained ranking of multiple systems incl. a hidden referenceListeners rate each 0-1000 – 100
    Pairwise PreferenceSimple binary preference (A vs. B)% votes for each system0 – 100 %
    Diagnostic MOSNaturalness and intelligibility on difficult sentencesSame as MOS, but sentence set is curated for edge cases1 – 5

    Pros: Gold standard—nothing beats human ears.
    Cons: Expensive, slow, high variance → usually complemented with objective metrics below.


    2. Signal-level Objective Metrics 🎛️

    MetricWhat it really checksGood when you need…
    MCD (Mel-Cepstral Distortion)Spectral distance (dB) between generated & reference speech. Lower = better.Fast, model-agnostic proxy for spectral quality.
    F0 RMSE / CCError or correlation in pitch (F0) contours.You care about prosody/pitch accuracy.
    V/UV errorVoiced/Unvoiced decision accuracy.Checking basic prosodic correctness.
    Duration RMSEPhone / syllable duration error.Rhythm & timing fidelity.
    STOIShort-Time Objective Intelligibility (0–1).Speech is noisy or bandwidth-limited.
    PESQ (narrow-band) & POLQA (wide-band)Perceptual speech quality (–0.5…4.5).Telephony / codec scenarios.
    SNR / Segmental SNRNoise level vs. clean reference.Checking denoising / vocoder artifacts.

    Remember: these are reference-based—you need parallel ground-truth recordings.


    3. Intelligibility-oriented Metrics 🧠

    1. ASR-WER: Run a robust ASR on the generated audio and compute Word Error Rate.
      • Fast, language-agnostic proxy for comprehension.
      • Lower WER ⇒ higher intelligibility.

    2. CER or SER: Char / Sentence Error Rate for languages with no clear word boundaries.

    Tip: Keep the same ASR model for all experiments; otherwise the metric drifts.


    4. Speaker Similarity & Style Transfer 🎙️

    MetricExplanation
    Speaker Embedding Cosine SimilarityCompare d-vectors/x-vectors from a speaker-verification network: higher similarity ⇒ better voice cloning.
    EER (Equal Error Rate)Run speaker verification: how often the system confuses synthesized speech for the target speaker. Lower EER = better.
    AB-X TestHuman listeners decide if X is closer to A (target) or B (other).

    5. Efficiency & Deployment Metrics 🚀

    Sometimes “quality” also means “will it run fast enough?”:

    • Real-Time Factor (RTF) = synth_time / audio_duration (RTF < 1 ⇒ faster-than-real-time).
    • Latency (ms) = time from text input to first audio chunk.
    • Model size (MB) & peak RAM/VRAM usage.
    • Throughput (sentences/sec or characters/sec on target hardware).
    • Energy / power draw if you’re targeting edge devices.


    Putting it all together

    A typical academic or production evaluation might look like:

    1. Human MOS on ~50 unseen sentences, n ≈ 20 raters.
    2. MCD, F0 RMSE, ASR-WER on the same set for quick iteration.
    3. Speaker similarity cosine + EER for voice-cloning TTS.
    4. RTF & memory numbers on your deployment GPU / CPU.

    That gives you a 360° view: naturalness, intelligibility, speaker fidelity, and practicality.


    Quick References

    • ITU-T P.800 – MOS procedures
    • ITU-T P.862 & P.863 – PESQ / POLQA
    • MUSHRA (ITU-R BS.1534-1)
    • “TTS MOSNet” (Lo et al., 2019) – a learned MOS predictor

    Feel free to mix-and-match depending on what success looks like for you!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.