My Computer has the following specs CPU: AMD 5900HX GPU: AMD RX6600 8GB RAM: 64GB What…
- User
My Computer has the following specs CPU: AMD 5900HX GPU: AMD RX6600 8GB RAM: 64GB
What local llm would you recommend? Also, what is the largest model I can run? How many tokens/second can I expect?
Assistant
Given your system specs (AMD Ryzen 9 5900HX, RX 6600 8GB, and 64GB RAM), you have a solid setup for running local LLMs. Here are my recommendations:
Recommended Local LLMs
-
Best Balance of Performance & Size:
- Mistral-7B (or Mistral-7B-Instruct)
- Excellent performance for its size, supports long contexts, and runs well on your hardware.
- Llama-2-13B (or Llama-2-13B-Chat)
- Slightly larger but still manageable, with strong reasoning capabilities.
- Mistral-7B (or Mistral-7B-Instruct)
-
Smaller & Faster (Good for Quick Responses):
- Phi-2 (2.7B) – Surprisingly capable for its size, runs very fast.
- TinyLlama (1.1B) – Lightweight but decent for basic tasks.
-
Larger Models (If You Want More Power):
- Llama-2-70B (with 4-bit or 5-bit quantization)
- Possible with your 64GB RAM, but expect slower speeds (~5-10 tokens/sec).
- Mistral-7B x2 (14B equivalent via model merging)
- Some tools allow merging two 7B models for better performance.
- Llama-2-70B (with 4-bit or 5-bit quantization)
Largest Model You Can Run
- Without Quantization:
- ~13B models (e.g., Llama-2-13B) should run smoothly in 16-bit (~26GB VRAM equivalent, but your GPU has 8GB, so you'll rely on CPU RAM).
- With 4-bit Quantization:
- Up to ~30B models (e.g., Llama-2-30B, but performance may suffer).
- 70B models (e.g., Llama-2-70B) are possible but will be very slow (~3-8 tokens/sec).
Expected Tokens/Second (Inference Speed)
Model Size Quantization Expected Speed (tokens/sec) 7B 4-bit 20-40 13B 4-bit 10-20 30B 4-bit 5-10 70B 4-bit 3-8 - Faster speeds are possible with GPU acceleration (ROCm for AMD GPUs).
- Slower speeds if running purely on CPU (but your 5900HX is decent for this).
Best Tools to Run LLMs Locally
- LM Studio (Easy GUI for Windows/Mac)
- Ollama (Simple CLI-based, supports many models)
- text-generation-webui (Advanced, supports quantization & fine-tuning)
- llama.cpp (Best for CPU/GPU optimization)
Final Recommendation
- Best overall choice: Mistral-7B (4-bit quantized) – Fast (~30 tokens/sec) and high-quality.
- If you need more power: Llama-2-13B (4-bit) – Slower (~15 tokens/sec) but stronger.
- For experimentation: Phi-2 (2.7B) – Very fast (~50+ tokens/sec) but smaller.
Would you like help setting up any of these?
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.