Skip to content
The Chat Archive
qwq-32b LMArena Human Preference 140K

[code] What is the activated parameters count and total parameters count for the model?

  1. User
    {
      "architectures": [
        "Qwen3MoeForCausalLM"
      ],
      "attention_bias": false,
      "attention_dropout": 0.0,
      "bos_token_id": 151643,
      "decoder_sparse_step": 1,
      "eos_token_id": 151645,
      "head_dim": 128,
      "hidden_act": "silu",
      "hidden_size": 4096,
      "initializer_range": 0.02,
      "intermediate_size": 12288,
      "max_position_embeddings": 40960,
      "max_window_layers": 94,
      "mlp_only_layers": [],
      "model_type": "qwen3_moe",
      "moe_intermediate_size": 1536,
      "norm_topk_prob": true,
      "num_attention_heads": 64,
      "num_experts": 128,
      "num_experts_per_tok": 8,
      "num_hidden_layers": 94,
      "num_key_value_heads": 4,
      "output_router_logits": false,
      "rms_norm_eps": 1e-06,
      "rope_scaling": null,
      "rope_theta": 1000000.0,
      "router_aux_loss_coef": 0.001,
      "sliding_window": null,
      "tie_word_embeddings": false,
      "torch_dtype": "bfloat16",
      "transformers_version": "4.51.0",
      "use_cache": true,
      "use_sliding_window": false,
      "vocab_size": 151936
    }  
    

    What is the activated parameters count and total parameters count for the model?

  2. Assistant

    The activated and total parameters for the Qwen3-Moe model can be calculated based on the provided configuration:

    Total Parameters (Approximate):

    The model's total parameters are dominated by the MoE (Mixture of Experts) layers. Each MoE layer has 128 experts, and each expert's feed-forward network (FFN) has parameters calculated as follows:

    • Per Expert Parameters:
      ( 2 \times \text{hidden_size} \times \text{moe_intermediate_size} )
      ( = 2 \times 4096 \times 1536 \approx 12.58 \text{ million} ).

    • Total per Layer (MoE FFN):
      ( 128 \text{ experts} \times 12.58 \text{ million} \approx 1.61 \text{ billion} ).

    • Total for All Layers:
      ( 94 \text{ layers} \times 1.61 \text{ billion} \approx \boxed{151.4 \text{ billion parameters}} ).

    Activated Parameters per Token (Approximate):

    When a token is processed, it uses 8 experts per layer (as specified by num_experts_per_tok). Thus:

    • Activated per Layer:
      ( 8 \text{ experts} \times 12.58 \text{ million} \approx 100.66 \text{ million} ).

    • Total Activated for All Layers:
      ( 94 \text{ layers} \times 100.66 \text{ million} \approx \boxed{9.46 \text{ billion parameters}} ).

    Summary

    • Total Parameters: ~151.4 billion (primarily from MoE layers).
    • Activated Parameters per Token: ~9.46 billion (only the selected experts are "activated" for computation).

    These values assume all layers are MoE-based, as indicated by the configuration's focus on MoE parameters (e.g., moe_intermediate_size, num_experts, etc.). Other components like embeddings and attention layers contribute negligibly to the total count compared to the MoE parameters.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.