Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-7-sonnet-20250219· AnthropicHugging Face LMArena Human Preference 140K

The more one gives token in input to an LLM, the more the LLM uses ram. Why is that?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-flash-preview-04-17’s answer.
  1. User

    The more one gives token in input to an LLM, the more the LLM uses ram. Why is that?

  2. Claude Assistant

    When you feed more tokens into an LLM, it needs to track more information in memory for several reasons:

    1. The model needs to maintain the entire context window in memory to generate coherent responses
    2. For each token, it stores attention patterns showing how it relates to all other tokens
    3. The attention mechanism creates matrices that grow quadratically with input length (if you double the input, memory usage roughly quadruples)

    This is why very long contexts are computationally expensive - the self-attention mechanism that helps LLMs understand relationships between words requires significant memory that scales dramatically with input length. This is also why many research efforts focus on developing more efficient attention mechanisms that can handle longer contexts without the same memory penalties.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.