Skip to content
TheChatArchiveThe Chat Archive
DeepSeekdeepseek-v3-0324· DeepSeekHugging Face LMArena Human Preference 140K

how does a draft model helps speeding up llm inference? i understand that a draft model…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with MiniMaxminimax-m1’s answer.
  1. User

    how does a draft model helps speeding up llm inference? i understand that a draft model computes "easy" tokens, but how does that speed up the process exactly?

  2. DeepSeek Assistant

    A draft model (or speculative decoding) speeds up LLM inference by reducing the number of sequential steps required to generate text. Here’s how it works in detail:

    Core Idea:

    Instead of running the full LLM for every token, a smaller, faster draft model predicts multiple tokens in advance (a "draft"). The full LLM then verifies these tokens in parallel, accepting correct ones and rejecting incorrect ones. This reduces the number of slow, full-model forward passes.

    Step-by-Step Process:

    1. Drafting Phase:

      • A lightweight draft model (e.g., a smaller LLM or a shallow network) generates a sequence of candidate tokens (e.g., 3–5 tokens) autoregressively.
      • This is fast because the draft model is much smaller than the main LLM.
    2. Verification Phase:

      • The full LLM evaluates the draft tokens in parallel (not sequentially).
      • For each position, the LLM checks if the draft token matches its own prediction.
      • If a token is correct, it’s accepted; if not, the process stops, and the LLM generates the correct token.
    3. Early Termination:

      • If all draft tokens are accepted, the LLM skips multiple sequential steps (e.g., 3–5 steps are done in 1 step).
      • If some are rejected, only the correct prefix is kept, and the process repeats.

    Why It Speeds Up Inference:

    • Parallel Verification: The full LLM checks multiple tokens at once (unlike autoregressive decoding, which is strictly sequential).
    • Fewer Full-Model Calls: The draft model handles "easy" tokens, while the LLM only corrects mistakes.
    • Hardware Efficiency: Batching the verification step leverages GPU/TPU parallelism better.

    Example:

    Suppose the draft model predicts:
    [A, B, C]
    The full LLM verifies in parallel:

    • If A matches, B matches, but C doesn’t, the output is [A, B, X] (where X is the corrected token).
    • The LLM only needed 1 forward pass instead of 3.

    Key Requirements:

    • The draft model must be fast (much cheaper than the full LLM).
    • The draft model should have decent accuracy (otherwise, the LLM frequently rejects tokens, reducing gains).
    • The full LLM must support parallel scoring (e.g., via masked attention).

    Performance Gains:

    • Typically 2–5x speedup (depends on draft model quality and acceptance rate).
    • Works best for tasks where tokens are predictable (e.g., grammar, common phrases).

    Variants:

    • Medusa, Speculative Decoding, Assisted Generation (Hugging Face) use similar ideas.
    • Some methods use multiple draft heads or tree-based drafting for better efficiency.

    In summary, draft models reduce latency by amortizing the cost of full LLM calls over multiple tokens, leveraging parallelism and early termination.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.