Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-flash· GoogleHugging Face LMArena Human Preference 140K

What is the best local llm to run on a gpu with 24 gigabytes of vram ?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Coherecommand-a-03-2025’s answer.
  1. User

    What is the best local llm to run on a gpu with 24 gigabytes of vram ?

  2. Gemini Assistant

    For 24 gigabytes of VRAM, you're in an excellent position! You can run some of the most capable open-source models with good performance.

    The BEST overall local LLM to run on a GPU with 24GB of VRAM, balancing capability, speed, and VRAM efficiency, is currently:

    Mixtral 8x7B (Instruct v0.1)

    Why Mixtral 8x7B is the best for 24GB VRAM:

    1. Exceptional Performance: Mixtral is a Sparse Mixture of Experts (SMoE) model. This means that while it has a massive total of 47 billion parameters, only 12 billion parameters are "active" (used) per token. This gives it performance comparable to much larger models (like Llama 2 34B or even 70B in some benchmarks) but with a VRAM footprint and inference speed closer to a 13B model.
    2. Perfect VRAM Fit: With 24GB, you can comfortably run quantized versions of Mixtral:
      • Q5_K_M (5-bit quantization): This is often the sweet spot for quality vs. VRAM. It will typically use around 14-16GB of VRAM, leaving you plenty of headroom for context length.
      • Q4_K_M (4-bit quantization): If you want to maximize context length or need slightly more VRAM for other tasks, this will use around 10-12GB, with only a minor perceived drop in quality.
    3. Versatility: Mixtral is highly versatile. It's excellent for:
      • General chat and conversational AI
      • Reasoning and problem-solving
      • Code generation and understanding
      • Creative writing and summarization
      • Instruction following

    Where to get Mixtral 8x7B (GGUF format for llama.cpp compatible runners):


    Runner-Up / Excellent Alternatives (Depending on your needs):

    1. Mistral 7B (Instruct v0.2 / Zephyr / OpenHermes 2.5 Mistral 7B)

      • Why: If Mixtral is still a bit too heavy or you prioritize maximum speed above all else, these 7B models are incredibly powerful for their size. They will run very fast and use minimal VRAM (typically 6-10GB for Q4/Q5). They are fantastic for rapid iteration, simple chat, or if you need to run multiple instances.
      • Where: Look for Mistral-7B-Instruct-v0.2-GGUF, zephyr-7b-beta.Q5_K_M.gguf, or OpenHermes-2.5-Mistral-7B.Q5_K_M.gguf from TheBloke.
    2. Llama 2 34B (Chat/Instruct)

      • Why: If you want a more "traditional" large model that's not an MoE, the 34B models offer a significant step up from 13B. You can fit them on 24GB, but you'll likely need to use Q4_K_M or even Q3_K_S quantization. They will be slower than Mixtral 8x7B but can still offer good quality.
      • VRAM Usage (approx.): Q4_K_M: ~18-20GB.
      • Where: Look for Llama-2-34B-Chat-GGUF or Yi-34B-Chat-GGUF from TheBloke.

    Software to Run Them:

    To run these GGUF format models, you'll want one of these:

    • LM Studio: (Easiest for beginners) A user-friendly desktop application that handles downloading and running models with a nice chat interface.
    • text-generation-webui (Oobabooga): (Very popular, feature-rich) A web-based UI that supports many model types and has tons of extensions. Excellent for more advanced users.
    • KoboldCpp: (Good for storytelling/roleplay) A standalone executable that's easy to use and optimized for creative tasks.
    • llama.cpp: (Command-line, most control) The underlying C++ library. If you're comfortable with the command line, it offers the most direct control and often the best performance.

    Recommendation Summary:

    Start with Mixtral 8x7B Instruct v0.1 Q5_K_M. Download it via LM Studio or text-generation-webui. This will give you the most powerful and versatile model that fits perfectly within your 24GB VRAM. If you find it's still not fast enough or you want to experiment, then try one of the Mistral 7B variants.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.