Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-7-sonnet-20250219· AnthropicHugging Face LMArena Human Preference 140K

How much memory do you need do run a large language model locally on your own system?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Metallama-3.3-70b-instruct’s answer.
  1. User

    How much memory do you need do run a large language model locally on your own system?

  2. Claude Assistant

    The memory requirements for running a large language model locally vary significantly based on the model size. For a modern LLM like GPT-4 or similar:

    • High-end models: 24-80GB+ of RAM and VRAM combined
    • Mid-size models: 8-16GB
    • Smaller models: 4-8GB

    Quantized versions (reduced precision) can lower these requirements substantially. For example, a quantized 7B parameter model might run on 8GB of RAM, while the full precision version would need much more.

    Your GPU's VRAM is particularly important, as it's typically faster than system RAM for model inference. Many enthusiasts use consumer GPUs with 8-24GB VRAM for running smaller or quantized models locally.

    Would you like me to explain more about model quantization or specific hardware recommendations?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.