What are the values of nheads nctx dmodel and dhead for vit and swin
- User
What are the values of n_heads n_ctx d_model and d_head for vit and swin
Assistant
The parameters n_heads, n_ctx, d_model, and d_head are commonly used in transformer architectures, including vision transformers (ViT) and Swin Transformer. Here's what these parameters typically mean:
- n_heads: Number of attention heads in the multi-head attention mechanism.
- n_ctx: Context length — often refers to the maximum sequence length the model can handle (e.g., number of patches in an image).
- d_model: Hidden dimension / embedding size.
- d_head: Dimension per attention head = d_model / n_heads.
These parameters vary depending on the specific model variant (e.g., ViT-Base, ViT-Large, Swin-Tiny, Swin-Base). Let's go over typical values for ViT and Swin Transformers:
🔷 Vision Transformer (ViT)
Model Variant n_heads n_ctx (patches) d_model d_head (= d_model / n_heads) ViT-Tiny 12 196 192 16 ViT-Small 12 196 384 32 ViT-Base 12 16,384 (224x224, patch=16) 768 64 ViT-Large 16 16,384 1024 64 ViT-Huge 32 16,384 1280 40 ℹ️ Note: For ViT, n_ctx is calculated as
(image_size // patch_size)^2. For example, with a 224x224 image and 16x16 patches → 14x14 = 196 patches for ViT-Tiny (some small variants), but standard ViT-Base uses 224x224 with 16x16 → 16,384 patches.Some ViT-Base implementations use:
- n_heads = 12, d_model = 768 → d_head = 64 (this contradicts earlier 16-head ViT-Base; Google's original ViT-Base indeed uses 12 heads)
✅ Clarification: The originally released ViT-Base by Google has:
- n_heads = 12
- d_model = 768
- n_ctx = 16,384
- d_head = 64
🔶 Swin Transformer
Swin Transformer uses shifted windows and hierarchical representation. The number of tokens (sequence length) depends on the image size and window/patch configuration.
Model Variant n_heads n_ctx (approx., per stage) d_model d_head Swin-Tiny 3 ~4,900 (after patch embed) 96 32 Swin-Small 6 ~4,900 96 16 Swin-Base 12 ~4,900 128 16 Swin-Large 24 ~4,900 192 8 ⚠️ Caveat: The n_ctx above is only approximate and varies by stage. Swin uses patch embeddings and divides images into non-overlapping patches (e.g., 4x4 for Swin-Tiny with 224x224 input → 56x56 patches), then groups them into windows (e.g., 7x7 windows → 49 tokens per window × 56x56 windows across layers).
More precisely:
Swin-Tiny
- Patch size: 4×4 → 56 × 56 patches
- Window size: 7×7 → 8 × 8 windows = 64 tokens per layer (per stage)
- But total sequence length varies by stage due to downsampling (Stage 1: 56×56, Stage 2: 28×28, etc.)
- n_heads: [3, 6, 12, 24] across 4 stages
- d_model: 96
- d_head: 32
Wait — this contradicts earlier simplified table. Let's clarify per-stage configs:
Accurate Configs:
✅ Swin-Tiny
- Embedding dim (d_model): 96
- Depths: [2, 2, 6, 2]
- Heads: [3, 6, 12, 24]
- Window size: 7
- Patch size: 4
- So:
- Stage 1: 56x56 → 3,136 tokens → 3 heads → d_head = 96 / 3 = 32
- Stage 2: 28x28 → 784 tokens → 6 heads → d_head = 16
- Stage 3: 14x14 → 196 tokens → 12 heads → d_head = 8
- Stage 4: 7x7 → 49 tokens → 24 heads → d_head = 4
But you asked for overall n_ctx — total context length varies. For Swin-Tiny, max tokens across all stages is not usually summed; each stage processes its own spatial resolution.
If we're talking about the maximum sequence length in any single stage, it's:
- Stage 1: 3,136
- Stage 2: 784
- Stage 3: 196
- Stage 4: 49
So there isn't a single "n_ctx" — but if one must pick, the first stage is often considered representative.
✅ Swin-Base
- d_model = 128
- Depths = [2, 2, 18, 2]
- Heads = [4, 8, 16, 32]
- Patch size = 4
- Window size = 7
- Max tokens (stage 1): (224//4)^2 = 56x56 = 3,136
- d_head = 128 / 4 = 32 (Stage 1), but actually:
- Stage 1: 4 heads → d_head = 32
- Stage 2: 8 heads → d_head = 16
- Stage 3: 16 heads → d_head = 8
- Stage 4: 32 heads → d_head = 4
Wait — this also shows that earlier generalizations were off.
✅ Corrected Summary for Swin:
Model d_model Heads (per stage) d_head Max Tokens (Stage 1) Swin-T 96 [3, 6, 12, 24] 32,16,8,4 3,136 Swin-S 96 [3, 6, 12, 24] same 3,136 Swin-B 128 [4, 8, 16, 32] 32,16,8,4 3,136 Swin-L 192 [6,12,24,48] 32,16,8,4 3,136 So in Swin, d_head is consistent per stage, and n_ctx per stage is fixed based on image size and patching (e.g., 3,136 for 224x224 input).
✅ Final Answer (Typical Values):
ViT-Base (as in original paper):
- n_heads: 12
- n_ctx: 16,384 (for 224x224 image with 16x16 patches)
- d_model: 768
- d_head: 64
Swin-Tiny:
- n_heads: [3, 6, 12, 24] (per stage)
- n_ctx (stage 1): 3,136 (for 224x224 image with 4x4 patches)
- d_model: 96
- d_head: 32 (Stage 1), varies by stage
🧠 Note: Swin uses different numbers of heads at each stage, unlike ViT which typically uses the same number of heads throughout.
Let me know if you want configs for a specific model variant like ViT-Large or Swin-Base!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.