Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-7-sonnet-20250219-thinking-32k· AnthropicHugging Face LMArena Human Preference 140K

I'm planning on building a PC with 64GB VRAM (two RTX 5090s) and 512 DDR4 RAM with…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-flash-lite-preview-06-17-thinking’s answer.
  1. User

    I'm planning on building a PC with 64GB VRAM (two RTX 5090s) and 512 DDR4 RAM with 8x64GB sticks and a threadripper 3975wx. What llms can I run inference on and what performance should I expect ? Will the 64GB VRAM help with performance if the model is split between VRAM and RAM?

  2. Claude Assistant

    The hardware setup you're describing is quite powerful, though I should note the RTX 5090 doesn't exist yet as of my last update (Oct 2024) - you might be referring to a future card or the RTX 4090 (24GB VRAM per card).

    With 64GB total VRAM and 512GB RAM, you could run:

    • 7B parameter models (Llama 3, Mistral, etc.): Extremely fast, 100+ tokens/second
    • 13B-14B models: Very good performance, 50-70 tokens/second
    • 34B-70B models: Usable with quantization (4-bit, 8-bit)
    • Larger models (70B-100B+): Possible with CPU offloading

    Regarding VRAM vs. RAM splitting: Yes, the 64GB VRAM helps significantly. Models perform best when they fit entirely in VRAM. When they don't, portions can be offloaded to RAM, which is much slower but still functional with your setup. The Threadripper CPU will help minimize the performance hit when accessing RAM-stored weights.

    For best results, look into tools like llama.cpp, vLLM, or TGI with quantization techniques to maximize VRAM efficiency.

    Would you like me to elaborate on specific optimization techniques for this hardware?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.