I'm planning on building a PC with 64GB VRAM (two RTX 5090s) and 512 DDR4 RAM with…
- User
I'm planning on building a PC with 64GB VRAM (two RTX 5090s) and 512 DDR4 RAM with 8x64GB sticks and a threadripper 3975wx. What llms can I run inference on and what performance should I expect ? Will the 64GB VRAM help with performance if the model is split between VRAM and RAM?
Assistant
The hardware setup you're describing is quite powerful, though I should note the RTX 5090 doesn't exist yet as of my last update (Oct 2024) - you might be referring to a future card or the RTX 4090 (24GB VRAM per card).
With 64GB total VRAM and 512GB RAM, you could run:
- 7B parameter models (Llama 3, Mistral, etc.): Extremely fast, 100+ tokens/second
- 13B-14B models: Very good performance, 50-70 tokens/second
- 34B-70B models: Usable with quantization (4-bit, 8-bit)
- Larger models (70B-100B+): Possible with CPU offloading
Regarding VRAM vs. RAM splitting: Yes, the 64GB VRAM helps significantly. Models perform best when they fit entirely in VRAM. When they don't, portions can be offloaded to RAM, which is much slower but still functional with your setup. The Threadripper CPU will help minimize the performance hit when accessing RAM-stored weights.
For best results, look into tools like llama.cpp, vLLM, or TGI with quantization techniques to maximize VRAM efficiency.
Would you like me to elaborate on specific optimization techniques for this hardware?
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.