which model under $10 for output tokens is the fastest TPS output. can you show me the…
- User
which model under $10 for output tokens is the fastest TPS output. can you show me the leaderboard for TPS
- Assistant
As of now, token-per-second (TPS) output speed depends on several factors including:
- Model architecture
- Deployment method (API, local inference)
- Hardware (e.g., GPU type, CPU)
- Model size (smaller models are generally faster per token)
- Context length / Input size
Since model speeds can vary depending on testing conditions, most public benchmarks don’t always provide TPS output speed in a definitive leaderboard. However, multiple community sources — like LMSYS Chatbot Arena, Hugging Face Open LLM Leaderboard, and OpenRouter / FastChat / VLLM benchmarks — evaluate latency and throughput.
That said, smaller quantized models (<10B) typically offer the fastest TPS under budget, especially for inference under $10.
⚡ Fastest Output TPS Models (Under $10 in cost for outputs)
Model TPS (approx) Cost Efficiency Notes Phi-3-mini (3.8B) ✅ 200–300 TPS ✅ Super cheap Very small, extremely fast, strong performance for size Gemma-2B (2B) ✅ 300–400 TPS ✅ Cheapest Very fast due to size, but weaker performance Mistral-7B (v0.3) ✅ 150–250 TPS ✅ Good balance Very efficient, stronger than most 2B–4B LLaMA3-8B ✅ 100–200 TPS ✅ Good Slightly slower than smaller models but better output Mixtral 8x7B (MoE) ❌ ~30–70 TPS ❌ Costs more Sparse activation helps TPS, but MoE routing adds overhead GPT-3.5-turbo ✅ 50–150 TPS ✅ Cheap on OpenAI Very fast on OpenAI infra, but still slower than smaller local models Claude 3 Haiku ✅ 50–100 TPS ✅ Cheap Optimized for response time, solid for under $10 workloads
✅ Best Choices Under $10
Rank Model Reason 🥇 Phi-3-mini (3.8B, MS Azure) Fastest + Strong performance for size + Cheap 🥈 Mistral-7B (via vLLM or Ollama) Great for mid-size apps, high TPS 🥉 Gemma-2B Blazing fast for ultra-light applications 4 GPT-3.5 / Claude Haiku Fast via API, convenient, still affordable 5 LLaMA3-8B Balanced speed and strength
💡 Notes on TPS Comparisons:
-
API Hosted Models (OpenAI, Claude, Cohere, etc.) do not expose raw TPS, but tests show GPT-3.5-turbo and Claude Haiku have decent TPS.
-
Locally run models (via vLLM, Ollama, Hugging Face) can exceed 300–400 TPS on RTX 4090 or A100 for models like Phi-3-mini, Gemma, Mistral-7B, due to optimized token streaming.
📊 TPS Benchmarks (Unofficial)
Here's an example chart of rough TPS on RTX 4090 (using vLLM or similar):
Model Size TPS Framework Phi-3-mini 3.8B ~300+ vLLM/Ollama Mistral-7B 7B ~180–250 vLLM/Ollama Gemma-2B 2B ~350–500 Ollama LLaMA3-8B 8B ~150 vLLM GPT-3.5-turbo - ~100–150 OpenAI API Claude 3 Haiku - ~100 Claude API
👉 Where to Find Live Benchmarks:
- LMSYS Chatbot Arena
- Open LLM Leaderboard on Hugging Face
- OpenRouter.ai (shows latency per model)
- VLLM GitHub (for raw output TPS tests)
If you tell me your usage platform (API or local) and hardware (e.g., GPU), I can recommend the fastest TPS model with optimal cost under $10.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.