My Computer has the following specs CPU: AMD 5900HX GPU: AMD RX6600 8GB RAM: 64GB What…
- User
My Computer has the following specs CPU: AMD 5900HX GPU: AMD RX6600 8GB RAM: 64GB
What local llm would you recommend? Also, what is the largest model I can run? How many tokens/second can I expect?
Assistant
For your system with an AMD 5900HX CPU, RX6600 8GB GPU, and 64GB RAM, I'd recommend Mistral 7B Instruct.
The largest model you could reasonably run would be a 13B parameter model like Llama 3 13B, which should fit within your 8GB VRAM with some quantization (like 4-bit).
For performance, you can expect roughly:
- 20-30 tokens/second with a 7B model using your GPU
- 10-15 tokens/second with a 13B model
Your 64GB of system RAM gives you plenty of headroom for context length, but the GPU VRAM is the main limiting factor. If you want to prioritize speed over size, stick with a well-optimized 7B model.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.