The more one gives token in input to an LLM, the more the LLM uses ram. Why is that?
- User
The more one gives token in input to an LLM, the more the LLM uses ram. Why is that?
Assistant
When you feed more tokens into an LLM, it needs to track more information in memory for several reasons:
- The model needs to maintain the entire context window in memory to generate coherent responses
- For each token, it stores attention patterns showing how it relates to all other tokens
- The attention mechanism creates matrices that grow quadratically with input length (if you double the input, memory usage roughly quadruples)
This is why very long contexts are computationally expensive - the self-attention mechanism that helps LLMs understand relationships between words requires significant memory that scales dramatically with input length. This is also why many research efforts focus on developing more efficient attention mechanisms that can handle longer contexts without the same memory penalties.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.