Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro· GoogleHugging Face LMArena Human Preference 140K

which are the best language models to run locally on your pc

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-5-sonnet-20241022’s answer.
  1. User

    which are the best language models to run locally on your pc

  2. Gemini Assistant

    Of course! Running language models locally is an exciting and rapidly evolving field. The "best" model depends heavily on your hardware and what you want to do with it.

    Here’s a breakdown of the best language models to run locally, categorized by their strengths, along with the tools you'll need to run them.

    First, a Crucial Concept: Quantization

    You'll see terms like Q4_K_M, 4-bit, GGUF, etc. This is quantization. It's a process that shrinks models to a fraction of their original size by reducing the precision of their numbers (weights).

    • Why it's essential: A 7-billion parameter model might be 14GB originally, but a quantized version can be under 5GB. This is the magic that makes it possible to run powerful models on consumer hardware.
    • The Trade-off: A tiny bit of performance/accuracy is lost, but for most uses, it's completely worth it.
    • Key Format: GGUF: This is the most popular format for running models on a mix of CPU and GPU. It's what tools like LM Studio and Ollama use.

    The Top Contenders: Best Local LLMs (as of mid-2024)

    The landscape changes fast, but these are the current champions.

    1. Meta's Llama 3 Family

    The current king of open models. Llama 3 is incredibly capable, follows instructions well, and is less "sanitized" or prone to refusing prompts than many other models.

    • Llama 3 8B: The Best All-Rounder. This is the model most people should start with. It's fast, incredibly smart for its size, and can run on a wide range of hardware (even just CPU with enough RAM, but a GPU is recommended). Excellent for chatting, summarization, and creative writing.
    • Llama 3 70B: The Performance King. This model rivals the performance of closed-source giants like GPT-3.5 and sometimes even GPT-4. However, it requires a beast of a machine, typically needing a high-end GPU with 24GB of VRAM (like an RTX 3090/4090) to run smoothly, even when quantized.

    2. Microsoft's Phi-3 Family

    These are "small language models" (SLMs) that punch way above their weight. They are trained on highly curated, "textbook quality" data, making them surprisingly coherent and capable for their tiny size.

    • Phi-3 Mini (3.8B): The Small-but-Mighty Champion. This is arguably the best model if you have limited hardware (e.g., a laptop without a dedicated GPU or a GPU with <6GB VRAM). It's shockingly good for its size and perfect for simple tasks, coding, and chat.
    • Phi-3 Medium (14B): A fantastic middle-ground model that challenges models twice its size. If you can't run a 70B model but want more power than an 8B, this is a top choice.

    3. Mistral AI Family

    Mistral was the previous champion before Llama 3 and is still incredibly relevant and powerful.

    • Mistral 7B: A former favorite, now slightly overshadowed by Llama 3 8B, but still an excellent, fast, and efficient model. It's particularly good because of its permissive license.
    • Mixtral 8x7B: The Balanced Expert. This is a "Mixture of Experts" (MoE) model. It's like having 8 smaller "specialist" models working together. It’s very powerful, often outperforming Llama 2 70B while being much faster and requiring less VRAM (it only uses about 13B parameters at a time). It's a great choice for those with a good GPU (e.g., 12GB+ VRAM).

    4. Specialized Models

    • For Coding:
      • CodeLlama: Meta's family of code-specialized models. The Llama 3 versions are still in development, but the Llama 2-based ones are excellent.
      • Phind-CodeLlama: A fine-tuned version of CodeLlama that many developers swear by. It often gives better, more direct coding answers.
      • Deepseek Coder: Another very strong family of coding models that often tops the leaderboards.

    How to Run These Models: The Software

    You don't need to be a programmer! These tools make it easy.

    The Easy Way (GUI - Recommended for most users)

    1. LM Studio: The most popular choice. It's a polished, user-friendly application for Windows, Mac, and Linux.

      • Pros: Built-in model browser to search and download models from Hugging Face, a chat UI, and a local server compatible with the OpenAI API. You can easily see resource usage and tweak settings.
      • Best for: Beginners and those who want a simple, all-in-one solution.
    2. Ollama: A close second, but more developer-focused. It runs as a background service with a command-line interface.

      • Pros: Incredibly simple to use from the terminal (ollama run llama3). It's lightweight and manages models cleanly. Many third-party GUIs are built on top of it.
      • Best for: Developers or users who are comfortable with the command line and want a clean, simple backend.
    3. Jan: A newer, open-source alternative to LM Studio that is gaining popularity. It offers a clean interface and local API server functionality.

    The Advanced Way (Command Line / Coding)

    1. llama.cpp: The core engine that powers much of the local LLM world. It’s a lightning-fast C++ implementation for running models.

      • Use Case: For developers who want maximum performance and control, or to integrate LLMs into their own C++ applications.
    2. Hugging Face Transformers: The standard Python library for anyone doing ML/AI development.

      • Use Case: For Python developers who want to fine-tune models or build complex applications that require programmatic control over the LLM.

    Which Model Should You Use? A Quick Guide

    If your PC is a...Your Goal is...Start with these Models
    Basic Laptop / PC (No dedicated GPU, 8-16GB RAM)Simple chat, summarization, light coding help.Phi-3 Mini (3.8B), Llama 3 8B (heavily quantized, e.g., Q2 or Q3)
    Decent Gaming PC (GPU with 8-12GB VRAM, e.g., RTX 3060/4060)High-quality chat, complex tasks, good coding.Llama 3 8B, Mixtral 8x7B (low-bit quantize), Phi-3 Medium
    High-End Enthusiast PC (GPU with 16-24GB+ VRAM, e.g., RTX 4090)The absolute best performance, rivaling cloud APIs.Llama 3 70B, other large models (Yi 34B, etc.)
    Any PCThe best possible coding assistant.Phind-CodeLlama, Deepseek Coder, Llama 3 8B Instruct

    My recommendation for a first-time user:

    1. Download LM Studio.
    2. In the search bar, type Llama 3 8B Instruct GGUF.
    3. Look for files uploaded by "QuantFactory" and download a version with Q4_K_M in the name. This is a great balance of size and quality.
    4. Load the model in the chat tab and start experimenting

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.