Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Develop a numerical reproducibility testbed for ML pipelines focusing on floating-point…

  1. User

    Develop a numerical reproducibility testbed for ML pipelines focusing on floating-point drift across hardware: instrumentation, tolerance policy, and CI integration.

  2. GLM Assistant

    1. Analyze the Request: * Topic: Numerical reproducibility testbed for Machine Learning (ML) pipelines. * Focus: Floating-point drift across different hardware (e.g., CPU vs. GPU, Vendor A GPU vs. Vendor B GPU, or even different SIMD instruction sets). * Key Components Required: * Instrumentation (How to capture numerical values). * Tolerance Policy (How to determine if a difference is acceptable). * CI Integration (How to automate this in a pipeline).

    1. Deconstruct the Problem:

      • Why does this happen? Non-associativity of floating-point arithmetic, different reductions orders (parallel vs. serial), fused-multiply-add (FMA) instructions, different library implementations (cuDNN vs. MKL).
      • What needs to be tested? Model weights, gradients, loss values, intermediate activations, final inference outputs.
      • How to structure the solution? I need a modular system. A Python library approach seems best, given the dominance of Python in ML.
    2. Drafting the Components:

      • Component 1: Instrumentation (The "Hooks")

        • Need a way to grab tensors without changing the core model code too much.
        • Mechanism: Hooks in PyTorch/TensorFlow are standard.
        • Serialization: Save tensors to disk. Need a format that preserves precision (Numpy arrays, HDF5). Don't save as images or compressed formats!
        • Metadata: Need to know where the data came from (hardware ID, seed, library versions).
      • Component 2: Comparison Engine (The "Math")

        • Direct equality (==) is useless.
        • Standard metrics: Absolute difference, Relative difference.
        • Advanced metrics: ULP (Units in the Last Place) – this is the gold standard for floating point, but can be tricky for very large/small numbers.
        • Statistical checks: Kullback-Leibler divergence (for probability distributions), Cosine similarity.
      • Component 3: Tolerance Policy (The "Rules")

        • One size does not fit all. A difference in logits is different from a difference in gradient norms.
        • Need a configuration file (YAML/JSON).
        • Strictness: rtol/atol like numpy.allclose.
        • Bit-level: Allow N bits of flip.
      • Component 4: CI Integration (The "Automation")

        • Needs to run on PRs.
        • Workflow:
          1. Generate "Golden Master" on a reference hardware (e.g., a specific CPU runner).
          2. Upload artifact.
          3. Run on target hardware (e.g., a GPU runner).
          4. Download artifact.
          5. Compare.
        • Reporting: JUnit XML for GitHub Actions/GitLab CI visualization.
    3. Developing the Code Structure (Mental Draft -> Concrete):

      • Class Structure:

        • Snapshot: Holds the tensor and metadata.
        • Comparator: Does the math.
        • Policy: Holds the rules.
        • Reporter: Generates output.
      • Refining the Instrumentation:

        • Let's use a context manager or a decorator.
        • Example: with NumericsMonitor(model) as monitor: ...
    4. Refining the "Floating-Point Drift" aspect:

      • Must mention deterministic flags (e.g., CUBLAS_WORKSPACE_CONFIG).
      • Must explain why we expect drift (reduction order).
    5. Writing the Content (Iterative refinement):

      • Introduction: Define the scope. It's about statistical reproducibility vs bitwise reproducibility.
      • Architecture: Divide into the three requested pillars.
      • Code Examples: Use PyTorch as the default example (most common for this type of research).
      • Advanced Topic: Mention "Golden Master" vs "Runtime Shadowing".
    6. Self-Correction/Review during drafting:

      • Initial thought: Just compare final accuracy.
      • Correction: No, that hides numerical instability. We need to compare tensors (gradients/activations).
      • Initial thought: Use numpy.isclose.
      • Correction: That's too simple. We need a custom comparator that handles NaNs and Infs gracefully and supports ULP.
    7. Finalizing the Output Structure:

      • Phase 1: The Architecture.
      • Phase 2: Implementation (Python/PyTorch code).
        • The NumericsProfiler class.
        • The Comparator logic (ULP + Relative).
        • The TolerancePolicy config.
      • Phase 3: CI/CD Workflow.
        • The "Golden Master" pattern.
        • GitHub Actions YAML example.
      • Phase 4: Best Practices.
        • Seeds, Environment variables.
    8. Adding Polish: Ensure code snippets are clean, comments are helpful, and the tone is technical and engineering-focused. (e.g., use terms like "non-determinism", "fused-multiply-add", "hermeticity").

    (Self-Correction during Code Generation): I need to make sure the ULP calculation handles edge cases (zero, infinity). I will implement a simplified ULP check.

    1. Final Output Generation (Proceeding to generate the response based on these steps). This design outlines a production-grade Numerical Reproducibility Testbed (NRT). This system acts as a "CI/CD for Math," ensuring that code changes, dependency updates, or hardware migrations do not silently alter model behavior due to floating-point intricacies.

    The architecture is divided into three layers: Instrumentation (Data Capture), Policy (Evaluation Logic), and Integration (Automation).


    1. Architecture Overview

    The testbed relies on the Golden Master pattern.

    1. Reference Run: Execute the pipeline on a "stable" environment (e.g., CPU or a specific GPU driver) to generate baseline tensors.
    2. Target Run: Execute the pipeline on the new environment (e.g., a different GPU architecture or after a library bump).
    3. Comparator: Apply tolerance policies to the divergences.

    2. Layer 1: Instrumentation (The Hooks)

    We need to capture tensor states without cluttering the model code. We use a decorator/context-manager approach to snapshot tensors at critical points (Forward pass, Loss calc, Gradients).

    The NumericSpy Tool

    This utility wraps PyTorch modules (or TensorFlow/Keras layers) to dump tensor snapshots to disk or memory.

    # instrumentation.py
    import torch
    import json
    from pathlib import Path
    from typing import Dict, List
    
    class NumericSpy:
        def __init__(self, log_dir: str = "./numeric_logs"):
            self.log_dir = Path(log_dir)
            self.log_dir.mkdir(exist_ok=True)
            self.manifest = {}
    
        def snapshot(self, tensor: torch.Tensor, name: str, step: int):
            """
            Saves a tensor to disk with high precision.
            For large tensors, statistical sampling (mean/std/histogram) 
            is preferred over full dumps to save IO.
            """
            if tensor is None: return
            
            # Convert to CPU numpy for consistent storage format
            data = tensor.detach().cpu().numpy()
            
            # File naming convention: step_name.npy
            fname = f"{step:04d}_{name}.npy"
            path = self.log_dir / fname
            
            # Save raw data (ensure float64 preservation if casted)
            np.save(path, data.astype(np.float64))
            
            # Log metadata
            self.manifest[name] = {
                "shape": list(data.shape),
                "dtype": str(tensor.dtype),
                "file": fname
            }
    
        def save_manifest(self):
            with open(self.log_dir / "manifest.json", "w") as f:
                json.dump(self.manifest, f, indent=2)
    
    def instrument_model(model: torch.nn.Module, spy: NumericSpy, step: int):
        """
        Registers forward hooks to capture layer outputs.
        """
        hooks = []
        def make_hook(name):
            def hook(module, input, output):
                # Capture Output Mean and Std Dev (Lightweight monitoring)
                # Or capture full tensor (Deep debugging)
                if isinstance(output, torch.Tensor):
                    spy.snapshot(output, f"{name}_output", step)
            return hook
    
        for name, layer in model.named_modules():
            if name:  # Avoid root hook
                hooks.append(layer.register_forward_hook(make_hook(name)))
        
        return hooks
    

    3. Layer 2: Tolerance Policy (The Logic)

    Comparing floating points requires nuance. A strict == fails due to non-associativity of floating-point arithmetic (e.g., (A+B)+C != A+(B+C) on GPUs). We implement a hierarchical policy engine.

    The Policy Engine

    # policy_engine.py
    import numpy as np
    from dataclasses import dataclass
    from enum import Enum
    
    class AlertLevel(Enum):
        PASS = 0
        WARNING = 1    # Within tolerance, but trending up
        FAILURE = 2    # Exceeded tolerance
    
    @dataclass
    class ToleranceSpec:
        rtol: float = 1e-5  # Relative tolerance
        atol: float = 1e-8  # Absolute tolerance
        ulp_limit: int = 4  # Units in the Last Place (strict bit-level check)
        
        # Context-aware rules
        # Gradients usually noisier than weights
        tensor_type_modifiers = {
            "gradient": {"rtol": 1e-4},
            "weight": {"rtol": 1e-5},
            "activation": {"rtol": 1e-5}
        }
    
    class DriftComparator:
        def __init__(self, policy: ToleranceSpec):
            self.policy = policy
    
        def compute_ulp(self, a, b):
            """Computes Units in the Last Place difference."""
            # Simplification: View bits as integers and subtract
            # Note: Complex implementation needed for handling signs/NaNs
            if a == b: return 0
            return abs(np.view(np.float64(a), np.int64) - np.view(np.float64(b), np.int64))
    
        def compare(self, tensor_a: np.ndarray, tensor_b: np.ndarray, name: str):
            # 1. Shape Check
            if tensor_a.shape != tensor_b.shape:
                return AlertLevel.FAILURE, "Shape mismatch"
    
            # 2. NaN/Inf Check
            if np.any(np.isnan(tensor_b)) and not np.any(np.isnan(tensor_a)):
                return AlertLevel.FAILURE, "Target generated NaNs"
    
            # 3. Statistical Check (Fast fail)
            if not np.allclose(tensor_a, tensor_b, rtol=self.policy.rtol, atol=self.policy.atol):
                # 4. Deep Dive: Error Distribution
                diff = np.abs(tensor_a - tensor_b)
                max_diff = np.max(diff)
                mean_diff = np.mean(diff)
                
                # Check if errors are uniformly distributed (random noise) 
                # vs biased (algorithmic error).
                # If mean_diff is close to 0, it's likely just precision drift.
                # If mean_diff is high, logic is broken.
                
                details = {
                    "max_diff": max_diff,
                    "mean_diff": mean_diff,
                    "status": "FAIL"
                }
                return AlertLevel.FAILURE, details
    
            return AlertLevel.PASS, {"status": "PASS"}
    

    4. Layer 3: CI Integration (The Automation)

    We need a workflow that runs on every Pull Request (PR). We use GitHub Actions (conceptually) to demonstrate.

    The Workflow Strategy

    1. Matrix Job: Run the same test script on different hardware (e.g., runner: ubuntu-latest vs runner: [self-hosted-gpu-a100]).
    2. Artifact Upload: Upload the NumericSpy logs from the reference run.
    3. Comparison Job: Download artifacts and run DriftComparator.

    GitHub Actions Workflow (.github/workflows/numeric_ci.yml)

    name: Numeric Reproducibility Check
    
    on: [pull_request]
    
    jobs:
      generate-golden:
        runs-on: ubuntu-latest  # CPU Reference
        steps:
          - uses: actions/checkout@v3
          - name: Setup Python
            uses: actions/setup-python@v4
            with:
              python-version: '3.10'
          - name: Install Deps
            run: pip install -r requirements.txt
          - name: Run Training Step (Generate Golden)
            run: python train.py --mode snapshot --output_dir ./golden_logs
          - name: Upload Golden Artifact
            uses: actions/upload-artifact@v3
            with:
              name: golden-logs
              path: ./golden_logs
    
      check-target:
        runs-on: [self-hosted, gpu] # Target Hardware
        needs: generate-golden
        steps:
          - uses: actions/checkout@v3
          - name: Download Golden
            uses: actions/download-artifact@v3
            with:
              name: golden-logs
              path: ./golden_logs
          
          - name: Run Training Step (Target)
            run: python train.py --mode snapshot --output_dir ./target_logs
          
          - name: Compare Results
            id: comparison
            run: |
              python ci_compare.py \
                --ref ./golden_logs \
                --target ./target_logs \
                --report ./drift_report.json
          
          - name: Fail if Drift Detected
            run: |
              if grep -q '"status": "FAILURE"' ./drift_report.json; then
                echo "::error::Numerical drift detected exceeding tolerance."
                cat ./drift_report.json
                exit 1
              fi
    

    5. Implementation: The Comparison Script (ci_compare.py)

    This script ties the instrumentation to the policy.

    import argparse
    import numpy as np
    import json
    from policy_engine import DriftComparator, ToleranceSpec
    
    def main(ref_dir, target_dir, report_path):
        policy = ToleranceSpec()
        comparator = DriftComparator(policy)
        
        # Load Manifests
        with open(f"{ref_dir}/manifest.json") as f:
            ref_manifest = json.load(f)
        
        results = []
        
        for tensor_name, meta in ref_manifest.items():
            ref_file = f"{ref_dir}/{meta['file']}"
            tgt_file = f"{target_dir}/{meta['file']}"
            
            try:
                ref_tensor = np.load(ref_file)
                tgt_tensor = np.load(tgt_file)
                
                level, msg = comparator.compare(ref_tensor, tgt_tensor, tensor_name)
                
                results.append({
                    "tensor": tensor_name,
                    "status": level.name,
                    "details": msg
                })
                
                if level.name == "FAILURE":
                    print(f"[FAIL] {tensor_name}: {msg}")
                else:
                    print(f"[PASS] {tensor_name}")
                    
            except FileNotFoundError:
                results.append({
                    "tensor": tensor_name,
                    "status": "ERROR",
                    "details": "File missing in target run"
                })
    
        with open(report_path, "w") as f:
            json.dump(results, f, indent=2)
    
    if __name__ == "__main__":
        parser = argparse.ArgumentParser()
        parser.add_argument("--ref", required=True)
        parser.add_argument("--target", required=True)
        parser.add_argument("--report", required=True)
        args = parser.parse_args()
        main(args.ref, args.target, args.report)
    

    Summary of Features

    1. Hardware Agnostic: By casting to float64 during snapshotting, we normalize the bit-representation differences between hardware (e.g., TF32 on Ampere GPUs vs FP32 on Volta) to standard Double Precision for comparison.
    2. Non-Intrusive: The instrument_model function allows dropping the testbed into existing codebases with minimal refactoring.
    3. Tunable: Adjusting ToleranceSpec allows teams to define strictness for specific layers (e.g., allowing more drift in Softmax outputs than in Weight matrices).
    4. CI-Ready: Exits with code 1 on drift, blocking merges that introduce numerical instability.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.