Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

what is the best llm to build agentic workflows

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Coherecommand-a-03-2025’s answer.
  1. User

    what is the best llm to build agentic workflows

  2. Assistant

    “Best” depends on what you need in an agentic stack—raw reasoning power, controllability, licensing freedom, latency, or budget. Here’s a quick field-guide that teams building agents usually walk through.

    ────────────────────────────────────────

    1. Shortlist by Use-Case ──────────────────────────────────────── • Heavy reasoning / multi-step tool use → GPT-4, Claude-2, PaLM-2-Bison
      • Very long context (>50k tokens) → Claude-2 (100k), GPT-4-Turbo (128k)
      • Tight budgets / high QPS → GPT-3.5-Turbo, Cohere Command-R
      • Self-host + fine-tune → Llama-2-70B, Mistral-8x7B, Yi-34B
      • Tiny on-device agents → Phi-2, TinyLlama-1.1B, GGUF quantized models
      • Sensitive data / air-gapped → Llama-2-70B-chat, Open-Flamingo, a custom fine-tune

    ──────────────────────────────────────── 2. Quick Capability Matrix ────────────────────────────────────────

    ModelContextFunction-CallingLicensingRAM for Inference (float16)Typical $/1k tokens
    GPT-4 (OpenAI)8k/32k/128kYes (best)Closed SaaSCloud only0.03in/0.03 in / 0.06 out
    Claude-2 (Anthropic)100kTools via JSONClosed SaaSCloud only~0.008/0.008 / 0.024
    PaLM-2-Bison (Google)8k/32kYesClosed SaaSCloud only~0.01/0.01 / 0.005 out
    Llama-2-70B-chat4k (can rope extend)Via RAG wrapperApache-2≈140 GB (fp16)Self-host cost
    Mistral-8x7B (MoE)32kYes (OSS)Apache-2≈32 GBSelf-host cost
    Cohere Command-R128kYesClosed SaaSCloud only~0.002/0.002 / 0.006
    Phi-2 (2.7B)4k—MIT<6 GBFree / local

    ──────────────────────────────────────── 3. What Makes an LLM “Agent-Friendly”? ────────────────────────────────────────

    1. Reasoning depth: passes >70% on AgentBench or tool-use evals.
    2. Interface: structured function-calling / JSON mode to call your tools deterministically.
    3. Low latency + streaming: essential for long multi-step chains.
    4. Determinism knobs: temperature, top-p, logit-bias, etc.
    5. Token limit large enough for memory + current task + planning scratchpad.
    6. Permissive TOS if agents will act on users’ behalf (e.g., avoid “no automation” clauses).

    ──────────────────────────────────────── 4. Toolkit Patterns People Combine ──────────────────────────────────────── • Planner-Executor: a “planner” GPT-4 creates subtasks; cheaper model (gpt-3.5-turbo) executes.
    • Reflection loops: Claude-2 revisits its own plan, critiques, re-plots.
    • Routing: Use OpenAI function calling to decide which micro-service or tool to call.
    • Hybrid search: Retrieval (LlamaIndex) → reasoning (GPT-4) → action (Python tool).
    • Multi-agent crews: CrewAI, AutoGen, or LangGraph orchestrate specialized agents.

    ──────────────────────────────────────── 5. Selection Cheat-Sheet ──────────────────────────────────────── Ask yourself: Recommended baseline ──────────────────────────────────────── ─────────────────────────── Need best-in-class reasoning? GPT-4 or Claude-2
    Need “free as in speech”? Llama-2-70B-chat or Mistral-8x7B
    Need 100k context w/ JSON outputs? Claude-2 (100k)
    Need cheap, scalable executor seats? GPT-3.5-Turbo, Cohere Command-R
    Need on-device demo / IoT agent? Phi-2 or TinyLlama
    Need enterprise on-premise w/ SLA? Azure GPT-4, AWS Bedrock (Claude/Palm), or self-host Llama-2

    ──────────────────────────────────────── 6. Practical Tip ──────────────────────────────────────── Most production systems end up hybrid: • Planner / critic → expensive frontier model (GPT-4, Claude-2).
    • Executing smaller units → cheaper model or fine-tuned open-source model.
    • Retrieval & state stored outside the LLM (vector DB, SQL) for determinism.

    If you have only bandwidth for one model today and can use a hosted API, GPT-4 with function-calling still gives the smoothest agent-building UX. If you must self-host, Llama-2-70B-chat or the new Mistral-8x7B MoE are the current sweet spots.

    Hope that helps—happy agent-building! 🚀

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.