Skip to content
TheChatArchiveThe Chat Archive
Mistralmistral-medium-2505· Mistral AIHugging Face LMArena Human Preference 140K

My Computer has the following specs CPU: AMD 5900HX GPU: AMD RX6600 8GB RAM: 64GB What…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219’s answer.
  1. User

    My Computer has the following specs CPU: AMD 5900HX GPU: AMD RX6600 8GB RAM: 64GB

    What local llm would you recommend? Also, what is the largest model I can run? How many tokens/second can I expect?

  2. Mistral Assistant

    Given your system specs (AMD Ryzen 9 5900HX, RX 6600 8GB, and 64GB RAM), you have a solid setup for running local LLMs. Here are my recommendations:

    Recommended Local LLMs

    1. Best Balance of Performance & Size:

      • Mistral-7B (or Mistral-7B-Instruct)
        • Excellent performance for its size, supports long contexts, and runs well on your hardware.
      • Llama-2-13B (or Llama-2-13B-Chat)
        • Slightly larger but still manageable, with strong reasoning capabilities.
    2. Smaller & Faster (Good for Quick Responses):

      • Phi-2 (2.7B) – Surprisingly capable for its size, runs very fast.
      • TinyLlama (1.1B) – Lightweight but decent for basic tasks.
    3. Larger Models (If You Want More Power):

      • Llama-2-70B (with 4-bit or 5-bit quantization)
        • Possible with your 64GB RAM, but expect slower speeds (~5-10 tokens/sec).
      • Mistral-7B x2 (14B equivalent via model merging)
        • Some tools allow merging two 7B models for better performance.

    Largest Model You Can Run

    • Without Quantization:
      • ~13B models (e.g., Llama-2-13B) should run smoothly in 16-bit (~26GB VRAM equivalent, but your GPU has 8GB, so you'll rely on CPU RAM).
    • With 4-bit Quantization:
      • Up to ~30B models (e.g., Llama-2-30B, but performance may suffer).
      • 70B models (e.g., Llama-2-70B) are possible but will be very slow (~3-8 tokens/sec).

    Expected Tokens/Second (Inference Speed)

    Model SizeQuantizationExpected Speed (tokens/sec)
    7B4-bit20-40
    13B4-bit10-20
    30B4-bit5-10
    70B4-bit3-8
    • Faster speeds are possible with GPU acceleration (ROCm for AMD GPUs).
    • Slower speeds if running purely on CPU (but your 5900HX is decent for this).

    Best Tools to Run LLMs Locally

    • LM Studio (Easy GUI for Windows/Mac)
    • Ollama (Simple CLI-based, supports many models)
    • text-generation-webui (Advanced, supports quantization & fine-tuning)
    • llama.cpp (Best for CPU/GPU optimization)

    Final Recommendation

    • Best overall choice: Mistral-7B (4-bit quantized) – Fast (~30 tokens/sec) and high-quality.
    • If you need more power: Llama-2-13B (4-bit) – Slower (~15 tokens/sec) but stronger.
    • For experimentation: Phi-2 (2.7B) – Very fast (~50+ tokens/sec) but smaller.

    Would you like help setting up any of these?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.