Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-7-sonnet-20250219· AnthropicHugging Face LMArena Human Preference 140K

If a load a 405b 4 bit quant into VRAM and RAM, what speed should I expect ?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Kimikimi-k2-0711-preview’s answer.
  1. User

    If a load a 405b 4 bit quant into VRAM and RAM, what speed should I expect ?

  2. Claude Assistant

    For a 405B parameter model at 4-bit quantization split between VRAM and RAM on your setup:

    You'd likely see around 1-3 tokens per second at best. The performance would be heavily bottlenecked by the constant data transfer between VRAM and system RAM, as only about 15-20% of the model would fit in your 64GB VRAM.

    The Threadripper CPU would help with the processing overhead, but the PCIe bandwidth limitations when constantly swapping weights between VRAM and RAM would create significant latency.

    This would be usable for non-time-sensitive applications, but definitely not for interactive conversations. You might improve performance slightly with optimized inference engines like vLLM with PagedAttention, but the fundamental memory bandwidth limitation would remain the primary constraint.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.