[code] What is the activated parameters count and total parameters count for the model?
- User
{ "architectures": [ "Qwen3MoeForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 151643, "decoder_sparse_step": 1, "eos_token_id": 151645, "head_dim": 128, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 12288, "max_position_embeddings": 40960, "max_window_layers": 94, "mlp_only_layers": [], "model_type": "qwen3_moe", "moe_intermediate_size": 1536, "norm_topk_prob": true, "num_attention_heads": 64, "num_experts": 128, "num_experts_per_tok": 8, "num_hidden_layers": 94, "num_key_value_heads": 4, "output_router_logits": false, "rms_norm_eps": 1e-06, "rope_scaling": null, "rope_theta": 1000000.0, "router_aux_loss_coef": 0.001, "sliding_window": null, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.51.0", "use_cache": true, "use_sliding_window": false, "vocab_size": 151936 }What is the activated parameters count and total parameters count for the model?
- Assistant
The activated and total parameters for the Qwen3-Moe model can be calculated based on the provided configuration:
Total Parameters (Approximate):
The model's total parameters are dominated by the MoE (Mixture of Experts) layers. Each MoE layer has 128 experts, and each expert's feed-forward network (FFN) has parameters calculated as follows:
-
Per Expert Parameters:
( 2 \times \text{hidden_size} \times \text{moe_intermediate_size} )
( = 2 \times 4096 \times 1536 \approx 12.58 \text{ million} ). -
Total per Layer (MoE FFN):
( 128 \text{ experts} \times 12.58 \text{ million} \approx 1.61 \text{ billion} ). -
Total for All Layers:
( 94 \text{ layers} \times 1.61 \text{ billion} \approx \boxed{151.4 \text{ billion parameters}} ).
Activated Parameters per Token (Approximate):
When a token is processed, it uses 8 experts per layer (as specified by
num_experts_per_tok). Thus:-
Activated per Layer:
( 8 \text{ experts} \times 12.58 \text{ million} \approx 100.66 \text{ million} ). -
Total Activated for All Layers:
( 94 \text{ layers} \times 100.66 \text{ million} \approx \boxed{9.46 \text{ billion parameters}} ).
Summary
- Total Parameters: ~151.4 billion (primarily from MoE layers).
- Activated Parameters per Token: ~9.46 billion (only the selected experts are "activated" for computation).
These values assume all layers are MoE-based, as indicated by the configuration's focus on MoE parameters (e.g.,
moe_intermediate_size,num_experts, etc.). Other components like embeddings and attention layers contribute negligibly to the total count compared to the MoE parameters. -
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.