Skip to content
TheChatArchiveThe Chat Archive
chatgpt-4o-latest-20250326· OpenAIHugging Face LMArena Human Preference 140K

which model under $10 for output tokens is the fastest TPS output. can you show me the…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-sonnet-4-20250514’s answer.
  1. User

    which model under $10 for output tokens is the fastest TPS output. can you show me the leaderboard for TPS

  2. Assistant

    As of now, token-per-second (TPS) output speed depends on several factors including:

    1. Model architecture
    2. Deployment method (API, local inference)
    3. Hardware (e.g., GPU type, CPU)
    4. Model size (smaller models are generally faster per token)
    5. Context length / Input size

    Since model speeds can vary depending on testing conditions, most public benchmarks don’t always provide TPS output speed in a definitive leaderboard. However, multiple community sources — like LMSYS Chatbot Arena, Hugging Face Open LLM Leaderboard, and OpenRouter / FastChat / VLLM benchmarks — evaluate latency and throughput.

    That said, smaller quantized models (<10B) typically offer the fastest TPS under budget, especially for inference under $10.


    ⚡ Fastest Output TPS Models (Under $10 in cost for outputs)

    ModelTPS (approx)Cost EfficiencyNotes
    Phi-3-mini (3.8B)✅ 200–300 TPS✅ Super cheapVery small, extremely fast, strong performance for size
    Gemma-2B (2B)✅ 300–400 TPS✅ CheapestVery fast due to size, but weaker performance
    Mistral-7B (v0.3)✅ 150–250 TPS✅ Good balanceVery efficient, stronger than most 2B–4B
    LLaMA3-8B✅ 100–200 TPS✅ GoodSlightly slower than smaller models but better output
    Mixtral 8x7B (MoE)❌ ~30–70 TPS❌ Costs moreSparse activation helps TPS, but MoE routing adds overhead
    GPT-3.5-turbo✅ 50–150 TPS✅ Cheap on OpenAIVery fast on OpenAI infra, but still slower than smaller local models
    Claude 3 Haiku✅ 50–100 TPS✅ CheapOptimized for response time, solid for under $10 workloads

    ✅ Best Choices Under $10

    RankModelReason
    🥇Phi-3-mini (3.8B, MS Azure)Fastest + Strong performance for size + Cheap
    🥈Mistral-7B (via vLLM or Ollama)Great for mid-size apps, high TPS
    🥉Gemma-2BBlazing fast for ultra-light applications
    4GPT-3.5 / Claude HaikuFast via API, convenient, still affordable
    5LLaMA3-8BBalanced speed and strength

    💡 Notes on TPS Comparisons:

    • API Hosted Models (OpenAI, Claude, Cohere, etc.) do not expose raw TPS, but tests show GPT-3.5-turbo and Claude Haiku have decent TPS.

    • Locally run models (via vLLM, Ollama, Hugging Face) can exceed 300–400 TPS on RTX 4090 or A100 for models like Phi-3-mini, Gemma, Mistral-7B, due to optimized token streaming.


    📊 TPS Benchmarks (Unofficial)

    Here's an example chart of rough TPS on RTX 4090 (using vLLM or similar):

    ModelSizeTPSFramework
    Phi-3-mini3.8B~300+vLLM/Ollama
    Mistral-7B7B~180–250vLLM/Ollama
    Gemma-2B2B~350–500Ollama
    LLaMA3-8B8B~150vLLM
    GPT-3.5-turbo-~100–150OpenAI API
    Claude 3 Haiku-~100Claude API

    👉 Where to Find Live Benchmarks:


    If you tell me your usage platform (API or local) and hardware (e.g., GPU), I can recommend the fastest TPS model with optimal cost under $10.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.