What's Feed Forward Network in Transformer based architecture? How does it differ from…
- User
What's Feed Forward Network in Transformer based architecture? How does it differ from forward pass?
Assistant
Let me explain the Feed Forward Network (FFN) in Transformers and how it differs from a general forward pass.
A Feed Forward Network in Transformers is a specific component that:
- Comes after the self-attention layer in each Transformer block
- Consists of two linear transformations with a non-linear activation (typically ReLU) in between
- Processes each position's features independently and identically
The architecture looks like this:
FFN(x) = max(0, xW₁ + b₁)W₂ + b₂Where:
- W₁, W₂ are weight matrices
- b₁, b₂ are bias vectors
- max(0, x) represents the ReLU activation
Key characteristics:
- Position-wise: applies the same transformation to each position independently
- Usually has a larger intermediate dimension (often 4x the model dimension)
- Helps in capturing non-linear interactions
In contrast, a forward pass refers to the general process of data flowing through any neural network from input to output, computing activations layer by layer. It's a broader term that encompasses:
- The entire sequence of computations through all layers
- Any type of layer (not just feed-forward)
- The complete path from input to final output
So while a feed-forward network is a specific architectural component in Transformers, a forward pass is the general process of data flowing through any neural network architecture.
Would you like me to elaborate on any particular aspect of FFNs or forward passes?
- User
It occurs to me that KV cache is a crutial part during model inference. Suppose I'm having one multi-round talk with a LLM, in each round i, my prompt is p_i and LLM's reply is r_i. How does KV cache work in this scenario?
Assistant
Let me explain how KV cache works in a multi-round conversation scenario.
For your conversation with rounds (p₁,r₁), (p₂,r₂), ..., (pₙ,rₙ):
- First Round (p₁,r₁):
- When processing p₁, the model computes Key (K) and Value (V) vectors for each token in p₁
- These K,V pairs are stored in cache
- When generating r₁, each new token attends to:
- All K,V pairs from p₁
- K,V pairs from previously generated tokens in r₁
- Subsequent Rounds (pᵢ,rᵢ):
- The model maintains K,V pairs from all previous rounds
- For new prompt pᵢ:
- Computes new K,V pairs only for pᵢ
- Appends these to the cached K,V pairs
- When generating rᵢ, each new token attends to:
- Cached K,V pairs from (p₁,r₁) to (pᵢ₋₁,rᵢ₋₁)
- K,V pairs from pᵢ
- K,V pairs from previously generated tokens in rᵢ
Key benefits:
- Avoids recomputing K,V pairs for previous context
- Reduces computational cost significantly
- Enables faster inference while maintaining coherence across conversation
Limitations:
- Memory usage grows with conversation length
- May need to truncate older context when reaching model's context window
- Cache needs to be managed carefully for memory efficiency
This is why KV cache is crucial for making chat models practical and responsive in real-world applications.
- User
Will the total inference time become longer in further rounds of this multi-round talk, since the computation of QKV values grows linearly or sublinealy with the length of context, even with the help of KV cache?
Assistant
Yes, the total inference time will generally increase in later rounds, even with KV cache, though the increase is less severe than without caching. Here's why:
- For each new token generation:
- Q (Query) must be computed for the new token
- This Q needs to attend to ALL cached K,V pairs
- Attention computation complexity is O(n) where n is the context length
- Each new token needs to compute attention scores with all previous tokens
- Matrix multiplication for attention grows linearly with context length
- Growth pattern:
Round 1: length(p₁ + r₁) = n₁ Round 2: length(p₁ + r₁ + p₂ + r₂) = n₂ ... Round i: length(sum(p₁...pᵢ + r₁...rᵢ)) = nᵢ- Each token in rᵢ needs O(nᵢ) computations
- While KV cache saves recomputation of K,V values
- The attention computation still needs to process growing context
- Mitigating strategies:
- Context window truncation
- Sliding window attention
- Sparse attention mechanisms
- Structured state space models
- Other attention approximation methods
So while KV cache significantly reduces computation by avoiding K,V recalculation, the fundamental attention computation still causes inference time to grow with context length. This is one reason why many models implement context window limits or other optimization strategies.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.