Skip to content
TheChatArchiveThe Chat Archive
Hunyuanhunyuan-turbos-20250416· TencentHugging Face LMArena Human Preference 140K

What are the values of nheads nctx dmodel and dhead for vit and swin

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-flash’s answer.
  1. User

    What are the values of n_heads n_ctx d_model and d_head for vit and swin

  2. Hunyuan Assistant

    The parameters n_heads, n_ctx, d_model, and d_head are commonly used in transformer architectures, including vision transformers (ViT) and Swin Transformer. Here's what these parameters typically mean:

    • n_heads: Number of attention heads in the multi-head attention mechanism.
    • n_ctx: Context length — often refers to the maximum sequence length the model can handle (e.g., number of patches in an image).
    • d_model: Hidden dimension / embedding size.
    • d_head: Dimension per attention head = d_model / n_heads.

    These parameters vary depending on the specific model variant (e.g., ViT-Base, ViT-Large, Swin-Tiny, Swin-Base). Let's go over typical values for ViT and Swin Transformers:


    🔷 Vision Transformer (ViT)

    Model Variantn_headsn_ctx (patches)d_modeld_head (= d_model / n_heads)
    ViT-Tiny1219619216
    ViT-Small1219638432
    ViT-Base1216,384 (224x224, patch=16)76864
    ViT-Large1616,384102464
    ViT-Huge3216,384128040

    ℹ️ Note: For ViT, n_ctx is calculated as (image_size // patch_size)^2. For example, with a 224x224 image and 16x16 patches → 14x14 = 196 patches for ViT-Tiny (some small variants), but standard ViT-Base uses 224x224 with 16x16 → 16,384 patches.

    Some ViT-Base implementations use:

    • n_heads = 12, d_model = 768 → d_head = 64 (this contradicts earlier 16-head ViT-Base; Google's original ViT-Base indeed uses 12 heads)

    ✅ Clarification: The originally released ViT-Base by Google has:

    • n_heads = 12
    • d_model = 768
    • n_ctx = 16,384
    • d_head = 64

    🔶 Swin Transformer

    Swin Transformer uses shifted windows and hierarchical representation. The number of tokens (sequence length) depends on the image size and window/patch configuration.

    Model Variantn_headsn_ctx (approx., per stage)d_modeld_head
    Swin-Tiny3~4,900 (after patch embed)9632
    Swin-Small6~4,9009616
    Swin-Base12~4,90012816
    Swin-Large24~4,9001928

    ⚠️ Caveat: The n_ctx above is only approximate and varies by stage. Swin uses patch embeddings and divides images into non-overlapping patches (e.g., 4x4 for Swin-Tiny with 224x224 input → 56x56 patches), then groups them into windows (e.g., 7x7 windows → 49 tokens per window × 56x56 windows across layers).

    More precisely:

    Swin-Tiny

    • Patch size: 4×4 → 56 × 56 patches
    • Window size: 7×7 → 8 × 8 windows = 64 tokens per layer (per stage)
    • But total sequence length varies by stage due to downsampling (Stage 1: 56×56, Stage 2: 28×28, etc.)
    • n_heads: [3, 6, 12, 24] across 4 stages
    • d_model: 96
    • d_head: 32

    Wait — this contradicts earlier simplified table. Let's clarify per-stage configs:

    Accurate Configs:

    ✅ Swin-Tiny

    • Embedding dim (d_model): 96
    • Depths: [2, 2, 6, 2]
    • Heads: [3, 6, 12, 24]
    • Window size: 7
    • Patch size: 4
    • So:
      • Stage 1: 56x56 → 3,136 tokens → 3 heads → d_head = 96 / 3 = 32
      • Stage 2: 28x28 → 784 tokens → 6 heads → d_head = 16
      • Stage 3: 14x14 → 196 tokens → 12 heads → d_head = 8
      • Stage 4: 7x7 → 49 tokens → 24 heads → d_head = 4

    But you asked for overall n_ctx — total context length varies. For Swin-Tiny, max tokens across all stages is not usually summed; each stage processes its own spatial resolution.

    If we're talking about the maximum sequence length in any single stage, it's:

    • Stage 1: 3,136
    • Stage 2: 784
    • Stage 3: 196
    • Stage 4: 49

    So there isn't a single "n_ctx" — but if one must pick, the first stage is often considered representative.

    ✅ Swin-Base

    • d_model = 128
    • Depths = [2, 2, 18, 2]
    • Heads = [4, 8, 16, 32]
    • Patch size = 4
    • Window size = 7
    • Max tokens (stage 1): (224//4)^2 = 56x56 = 3,136
    • d_head = 128 / 4 = 32 (Stage 1), but actually:
      • Stage 1: 4 heads → d_head = 32
      • Stage 2: 8 heads → d_head = 16
      • Stage 3: 16 heads → d_head = 8
      • Stage 4: 32 heads → d_head = 4

    Wait — this also shows that earlier generalizations were off.

    ✅ Corrected Summary for Swin:

    Modeld_modelHeads (per stage)d_headMax Tokens (Stage 1)
    Swin-T96[3, 6, 12, 24]32,16,8,43,136
    Swin-S96[3, 6, 12, 24]same3,136
    Swin-B128[4, 8, 16, 32]32,16,8,43,136
    Swin-L192[6,12,24,48]32,16,8,43,136

    So in Swin, d_head is consistent per stage, and n_ctx per stage is fixed based on image size and patching (e.g., 3,136 for 224x224 input).


    ✅ Final Answer (Typical Values):

    ViT-Base (as in original paper):

    • n_heads: 12
    • n_ctx: 16,384 (for 224x224 image with 16x16 patches)
    • d_model: 768
    • d_head: 64

    Swin-Tiny:

    • n_heads: [3, 6, 12, 24] (per stage)
    • n_ctx (stage 1): 3,136 (for 224x224 image with 4x4 patches)
    • d_model: 96
    • d_head: 32 (Stage 1), varies by stage

    🧠 Note: Swin uses different numbers of heads at each stage, unlike ViT which typically uses the same number of heads throughout.

    Let me know if you want configs for a specific model variant like ViT-Large or Swin-Base!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.