Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-5-sonnet-20241022· AnthropicHugging Face LMArena Human Preference 140K

What's Feed Forward Network in Transformer based architecture? How does it differ from…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-opus-4-20250514’s answer.
  1. User

    What's Feed Forward Network in Transformer based architecture? How does it differ from forward pass?

  2. Claude Assistant

    Let me explain the Feed Forward Network (FFN) in Transformers and how it differs from a general forward pass.

    A Feed Forward Network in Transformers is a specific component that:

    1. Comes after the self-attention layer in each Transformer block
    2. Consists of two linear transformations with a non-linear activation (typically ReLU) in between
    3. Processes each position's features independently and identically

    The architecture looks like this:

    FFN(x) = max(0, xW₁ + b₁)W₂ + b₂
    

    Where:

    • W₁, W₂ are weight matrices
    • b₁, b₂ are bias vectors
    • max(0, x) represents the ReLU activation

    Key characteristics:

    • Position-wise: applies the same transformation to each position independently
    • Usually has a larger intermediate dimension (often 4x the model dimension)
    • Helps in capturing non-linear interactions

    In contrast, a forward pass refers to the general process of data flowing through any neural network from input to output, computing activations layer by layer. It's a broader term that encompasses:

    • The entire sequence of computations through all layers
    • Any type of layer (not just feed-forward)
    • The complete path from input to final output

    So while a feed-forward network is a specific architectural component in Transformers, a forward pass is the general process of data flowing through any neural network architecture.

    Would you like me to elaborate on any particular aspect of FFNs or forward passes?

  3. User

    It occurs to me that KV cache is a crutial part during model inference. Suppose I'm having one multi-round talk with a LLM, in each round i, my prompt is p_i and LLM's reply is r_i. How does KV cache work in this scenario?

  4. Claude Assistant

    Let me explain how KV cache works in a multi-round conversation scenario.

    For your conversation with rounds (p₁,r₁), (p₂,r₂), ..., (pₙ,rₙ):

    1. First Round (p₁,r₁):
    • When processing p₁, the model computes Key (K) and Value (V) vectors for each token in p₁
    • These K,V pairs are stored in cache
    • When generating r₁, each new token attends to:
      • All K,V pairs from p₁
      • K,V pairs from previously generated tokens in r₁
    1. Subsequent Rounds (pᵢ,rᵢ):
    • The model maintains K,V pairs from all previous rounds
    • For new prompt pᵢ:
      • Computes new K,V pairs only for pᵢ
      • Appends these to the cached K,V pairs
    • When generating rᵢ, each new token attends to:
      • Cached K,V pairs from (p₁,r₁) to (pᵢ₋₁,rᵢ₋₁)
      • K,V pairs from pᵢ
      • K,V pairs from previously generated tokens in rᵢ

    Key benefits:

    • Avoids recomputing K,V pairs for previous context
    • Reduces computational cost significantly
    • Enables faster inference while maintaining coherence across conversation

    Limitations:

    • Memory usage grows with conversation length
    • May need to truncate older context when reaching model's context window
    • Cache needs to be managed carefully for memory efficiency

    This is why KV cache is crucial for making chat models practical and responsive in real-world applications.

  5. User

    Will the total inference time become longer in further rounds of this multi-round talk, since the computation of QKV values grows linearly or sublinealy with the length of context, even with the help of KV cache?

  6. Claude Assistant

    Yes, the total inference time will generally increase in later rounds, even with KV cache, though the increase is less severe than without caching. Here's why:

    1. For each new token generation:
    • Q (Query) must be computed for the new token
    • This Q needs to attend to ALL cached K,V pairs
    • Attention computation complexity is O(n) where n is the context length
      • Each new token needs to compute attention scores with all previous tokens
      • Matrix multiplication for attention grows linearly with context length
    1. Growth pattern:
    Round 1: length(p₁ + r₁) = n₁
    Round 2: length(p₁ + r₁ + p₂ + r₂) = n₂
    ...
    Round i: length(sum(p₁...pᵢ + r₁...rᵢ)) = nᵢ
    
    • Each token in rᵢ needs O(nᵢ) computations
    • While KV cache saves recomputation of K,V values
    • The attention computation still needs to process growing context
    1. Mitigating strategies:
    • Context window truncation
    • Sliding window attention
    • Sparse attention mechanisms
    • Structured state space models
    • Other attention approximation methods

    So while KV cache significantly reduces computation by avoiding K,V recalculation, the fundamental attention computation still causes inference time to grow with context length. This is one reason why many models implement context window limits or other optimization strategies.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.