Skip to content
The Chat Archive
qwen3-235b-a22b· Alibaba (Qwen) LMArena Human Preference 140K

what would be the training time for training an llm similar to deepseek r1 on 2 cerebras…

  1. User

    what would be the training time for training an llm similar to deepseek r1 on 2 cerebras csc3 systems

  2. Assistant

    Estimating the training time for a Large Language Model (LLM) similar to DeepSeek R1 on two Cerebras CSC3 systems involves significant uncertainty due to undisclosed technical details (e.g., exact parameter count, dataset size, hardware specifications). Below is a structured analysis:


    Key Assumptions

    1. Model Size:

      • Assume DeepSeek R1 has ~100–120 billion parameters (based on public benchmarks of similar models).
    2. Training Dataset:

      • Typically, large LLMs are trained on 1–5 trillion tokens of text data.
    3. Compute Requirements:

      • Training compute is approximated as: Total FLOPS≈6×Parameters×Tokens\text{Total FLOPS} \approx 6 \times \text{Parameters} \times \text{Tokens} For 100B parameters and 1T tokens: 6×1011×1012=6×1023 FLOPS.6 \times 10^{11} \times 10^{12} = 6 \times 10^{23} \text{ FLOPS}.
    4. CSC3 Hardware:

      • Cerebras CSC3 systems utilize wafer-scale AI accelerators with high memory bandwidth and model-parallelism capabilities.
      • Estimated performance (based on Cerebras' marketing claims and comparisons to competitors):
        • Single CSC3: ~1–10 PFLOPS (10¹⁵–10¹⁶ FLOPS) of effective AI compute.
        • Two CSC3s: ~2–20 PFLOPS (assuming near-linear scaling).
    5. Efficiency Factors:

      • Real-world utilization rates (e.g., memory bottlenecks, communication overhead) reduce theoretical peak performance. Assume 50–80% efficiency.

    Estimated Training Time

    Using the formula:

    Time (seconds)=Total FLOPSEffective System FLOPS\text{Time (seconds)} = \frac{\text{Total FLOPS}}{\text{Effective System FLOPS}}
    ScenarioEffective System FLOPSTime for 6e23 FLOPS
    Conservative Estimate2 PFLOPS (2 × 10¹⁵)~3.47 million seconds (~40 days)
    Moderate Estimate10 PFLOPS (1 × 10¹⁶)~600,000 seconds (~7 days)
    Optimistic Estimate20 PFLOPS (2 × 10¹⁶)~300,000 seconds (~3.5 days)

    Critical Considerations

    1. Memory Constraints:

      • Training a 100B+ model requires significant on-chip memory for weights, gradients, and activations. Cerebras systems excel here with wafer-scale memory (potentially >100GB per core), but batch sizes may need tuning to fit.
    2. Dataset Size:

      • If DeepSeek R1 was trained on 5T tokens, the time scales linearly (e.g., 5× longer than the estimates above).
    3. Software Optimization:

      • Cerebras’ proprietary software stack (e.g., weight streaming, ZeRO optimization) improves scaling efficiency compared to traditional GPU clusters.
    4. Model Parallelism:

      • Cerebras’ architecture is designed for trillion-parameter models, so two CSC3 systems could handle 100B models with minimal inter-system communication overhead.

    Final Answer

    Training an LLM similar to DeepSeek R1 on two Cerebras CSC3 systems would likely take between 3.5 days and 40 days, depending on:

    • Actual model size (parameters),
    • Dataset size (tokens),
    • Hardware performance (FLOPS and memory),
    • Software efficiency.

    For tighter estimates, precise details about DeepSeek R1’s training configuration and CSC3 specifications would be required. Cerebras’ systems are uniquely suited for such workloads, potentially outperforming traditional GPU clusters by orders of magnitude.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.