what is the best LLM for local use under 20b parameters?
- User
what is the best LLM for local use under 20b parameters?
- Assistant
Determining the "best" Local Large Language Model (LLM) under 20 billion parameters depends heavily on your specific needs (e.g., coding, general chat, reasoning) and your hardware capabilities (CPU, GPU VRAM). However, based on recent benchmarks and community discussions, several models consistently stand out:
Top Contenders (< 20B Parameters):
-
Llama 3 8B:
- Developer: Meta
- Strengths: Excellent all-around performer for its size, particularly strong in general text generation, instruction following, and increasingly, fine-tuned variants excel in specific tasks [4, 6, 8][1]. It often sets the benchmark for models in this size class [4, 18].
- Considerations: Base model needs fine-tuning for chat/instruction following (use "Instruct" versions for chat) [14].
-
Phi-3 Mini (3.8B) & Phi-3 Small (7B):
- Developer: Microsoft
- Strengths: Remarkably capable for their extremely small size, designed to run efficiently even on mobile devices [15, 19][2][3][4]. Phi-3 Mini, despite being only 3.8B parameters, competes well against 7B/8B models like Mistral 7B and Llama 3 8B on many benchmarks [15, 19]. Phi-3 Small (7B) offers a step up in performance[4]. Good at reasoning and coding relative to size [9, 15].
- Considerations: May lack the broad factual knowledge of larger models due to smaller size and specific training data focus [19][5]. Some benchmark results showing it beating much larger models have sparked discussion about potential benchmark contamination, though it performs well on private benchmarks too [18][3]. User experience compared to Llama 3 8B can vary [18].
-
Mistral 7B:
- Developer: Mistral AI
- Strengths: Very strong performance for a 7B model, often considered highly efficient [1, 8]. Known for good coding and content generation capabilities [1]. It has a vast number of excellent fine-tuned versions available for specific tasks [4, 16]. Supports a 32k context window [8].
- Considerations: While excellent, newer models like Llama 3 8B and Phi-3 sometimes edge it out in benchmarks [6, 19].
-
Qwen 1.5 / Qwen 2 / Qwen 2.5 (e.g., 7B, 14B):
- Developer: Alibaba Cloud[6]
- Strengths: Qwen models are known to perform very well, often punching above their weight class across various sizes [14][3][6]. They demonstrate strong multilingual capabilities and perform well on tasks like coding and math [5, 9][1][7]. The 14B parameter version fits within the <20B category and offers a significant step up from 7/8B models [7].
- Considerations: Might be less widely discussed than Llama or Mistral in some communities, but benchmark performance is strong [20].
-
Gemma 2 (9B):
- Developer: Google
- Strengths: Google's open model family built with similar technology to Gemini. Designed for responsible AI development and efficient deployment across hardware [5, 8]. The 9B model offers a balance between the smaller 7/8B models and larger ones, with strong performance for its size [9]. Good framework compatibility [5, 8].
- Considerations: Performance relative to competitors like Llama 3 8B or Qwen models depends on the specific benchmark.
-
GPT-NeoX (20B):
- Developer: EleutherAI[7][8][9]
- Strengths: A slightly larger model at the edge of your requirement. It offers strong performance in language understanding, math, and few-shot tasks [5, 10, 12][7].
- Considerations: Being a 20B model, it requires more resources than the 7/8B options [5]. May be outperformed by newer, more efficient models in some areas.
Factors for Local Use:
- Hardware: Running LLMs locally is heavily dependent on your RAM and, crucially, your GPU's VRAM if you have one [3, 14][3]. Smaller models (like Phi-3 Mini or Mistral/Llama 7/8B) are more feasible on consumer hardware, especially systems without powerful GPUs [3, 13][3].
- Quantization: To run these models efficiently, especially on resource-constrained hardware, you'll likely use quantized versions (e.g., 4-bit, 8-bit GGUF)[3]. Quantization significantly reduces the model's size and resource requirements, often with only a small impact on performance [3, 16][3]. A 4-bit quantized 7B model can sometimes outperform an 8-bit 3B model on CPU [3][10].
- Tools: Software like Ollama, LM Studio, llama.cpp, and Hugging Face's libraries simplify downloading and running these models locally [1, 2, 3, 5][3].
Conclusion:
For the best balance of performance and resource requirements under 20B parameters currently, Llama 3 8B Instruct and Mistral 7B Instruct (especially its fine-tunes) are very strong choices [4, 6]. If you need the absolute smallest model possible that is still highly capable, Phi-3 Mini is remarkable [15, 19]. If you have slightly more resources ( VRAM) and want a step up, Qwen 1.5/2.5 14B is a compelling option [7, 14].
Ultimately, the "best" choice is subjective. It's recommended to try a few different models (quantized versions are readily available) using tools like Ollama to see which performs best for your specific tasks and hardware setup [2, 3].
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.