What is the KV cache size for a batch size of 5 given this config? { "dim": 5120,…
- User
What is the KV cache size for a batch size of 5 given this config?
{ "dim": 5120, "n_layers": 40, "head_dim": 128, "hidden_dim": 14336, "n_heads": 32, "n_kv_heads": 8, "rope_theta": 1000000000.0, "norm_eps": 1e-05, "vocab_size": 131072, "vision_encoder": { "hidden_size": 1024, "num_channels": 3, "image_size": 1024, "patch_size": 16, "rope_theta": 10000.0, "intermediate_size": 4096, "num_hidden_layers": 24, "num_attention_heads": 16, "image_token_id": 10 } }
- Assistant
To determine the KV (Key-Value) cache size for the given model configuration with a batch size of 5, we need to calculate the memory required to store the keys and values generated during the model's self-attention mechanism across all layers and sequences.
Given Model Configuration Parameters:
- Batch Size (batch_size): 5
- Number of Layers (n_layers): 40
- Sequence Length (sequence_length): Since the maximum sequence length isn't explicitly provided, we'll assume a common large context length used in models like Llama-2, which is 16384 tokens.
- Number of Key-Value Heads (n_kv_heads): 8
- Head Dimension (head_dim): 128
- Bytes per Element (bytes_per_element): Assuming float16 precision, which is 2 bytes per element.
Calculation Steps:
-
Compute the Size per Layer per Sample: [ \text{Size per Layer per Sample} = \text{sequence_length} \times \text{n_kv_heads} \times \text{head_dim} \times 2 \times \text{bytes_per_element} ]
- The factor of 2 accounts for both keys and values.
- Plugging in the numbers: [ 16384 \times 8 \times 128 \times 2 \times 2 = 67,108,864 \text{ bytes} ]
-
Compute the Total Size for All Layers and All Samples in the Batch: [ \text{Total KV Cache Size} = \text{batch_size} \times \text{n_layers} \times \text{Size per Layer per Sample} ]
- Plugging in the numbers: [ 5 \times 40 \times 67,108,864 = 13,421,772,800 \text{ bytes} ]
-
Convert Bytes to Gigabytes: [ \text{Total KV Cache Size in GB} = \frac{13,421,772,800}{1024^3} \approx 12.5 \text{ GB} ]
Answer:
The KV cache size is 12.5 GB for a batch size of 5 with the given model configuration.