Develop a numerical reproducibility testbed for ML pipelines focusing on floating-point…
- User
Develop a numerical reproducibility testbed for ML pipelines focusing on floating-point drift across hardware: instrumentation, tolerance policy, and CI integration.
Assistant
1. Analyze the Request: * Topic: Numerical reproducibility testbed for Machine Learning (ML) pipelines. * Focus: Floating-point drift across different hardware (e.g., CPU vs. GPU, Vendor A GPU vs. Vendor B GPU, or even different SIMD instruction sets). * Key Components Required: * Instrumentation (How to capture numerical values). * Tolerance Policy (How to determine if a difference is acceptable). * CI Integration (How to automate this in a pipeline).
-
Deconstruct the Problem:
- Why does this happen? Non-associativity of floating-point arithmetic, different reductions orders (parallel vs. serial), fused-multiply-add (FMA) instructions, different library implementations (cuDNN vs. MKL).
- What needs to be tested? Model weights, gradients, loss values, intermediate activations, final inference outputs.
- How to structure the solution? I need a modular system. A Python library approach seems best, given the dominance of Python in ML.
-
Drafting the Components:
-
Component 1: Instrumentation (The "Hooks")
- Need a way to grab tensors without changing the core model code too much.
- Mechanism: Hooks in PyTorch/TensorFlow are standard.
- Serialization: Save tensors to disk. Need a format that preserves precision (Numpy arrays, HDF5). Don't save as images or compressed formats!
- Metadata: Need to know where the data came from (hardware ID, seed, library versions).
-
Component 2: Comparison Engine (The "Math")
- Direct equality (
==) is useless. - Standard metrics: Absolute difference, Relative difference.
- Advanced metrics: ULP (Units in the Last Place) – this is the gold standard for floating point, but can be tricky for very large/small numbers.
- Statistical checks: Kullback-Leibler divergence (for probability distributions), Cosine similarity.
- Direct equality (
-
Component 3: Tolerance Policy (The "Rules")
- One size does not fit all. A difference in logits is different from a difference in gradient norms.
- Need a configuration file (YAML/JSON).
- Strictness:
rtol/atollikenumpy.allclose. - Bit-level: Allow N bits of flip.
-
Component 4: CI Integration (The "Automation")
- Needs to run on PRs.
- Workflow:
- Generate "Golden Master" on a reference hardware (e.g., a specific CPU runner).
- Upload artifact.
- Run on target hardware (e.g., a GPU runner).
- Download artifact.
- Compare.
- Reporting: JUnit XML for GitHub Actions/GitLab CI visualization.
-
-
Developing the Code Structure (Mental Draft -> Concrete):
-
Class Structure:
Snapshot: Holds the tensor and metadata.Comparator: Does the math.Policy: Holds the rules.Reporter: Generates output.
-
Refining the Instrumentation:
- Let's use a context manager or a decorator.
- Example:
with NumericsMonitor(model) as monitor: ...
-
-
Refining the "Floating-Point Drift" aspect:
- Must mention deterministic flags (e.g.,
CUBLAS_WORKSPACE_CONFIG). - Must explain why we expect drift (reduction order).
- Must mention deterministic flags (e.g.,
-
Writing the Content (Iterative refinement):
- Introduction: Define the scope. It's about statistical reproducibility vs bitwise reproducibility.
- Architecture: Divide into the three requested pillars.
- Code Examples: Use PyTorch as the default example (most common for this type of research).
- Advanced Topic: Mention "Golden Master" vs "Runtime Shadowing".
-
Self-Correction/Review during drafting:
- Initial thought: Just compare final accuracy.
- Correction: No, that hides numerical instability. We need to compare tensors (gradients/activations).
- Initial thought: Use
numpy.isclose. - Correction: That's too simple. We need a custom comparator that handles
NaNs andInfs gracefully and supports ULP.
-
Finalizing the Output Structure:
- Phase 1: The Architecture.
- Phase 2: Implementation (Python/PyTorch code).
- The
NumericsProfilerclass. - The
Comparatorlogic (ULP + Relative). - The
TolerancePolicyconfig.
- The
- Phase 3: CI/CD Workflow.
- The "Golden Master" pattern.
- GitHub Actions YAML example.
- Phase 4: Best Practices.
- Seeds, Environment variables.
-
Adding Polish: Ensure code snippets are clean, comments are helpful, and the tone is technical and engineering-focused. (e.g., use terms like "non-determinism", "fused-multiply-add", "hermeticity").
(Self-Correction during Code Generation): I need to make sure the ULP calculation handles edge cases (zero, infinity). I will implement a simplified ULP check.
- Final Output Generation (Proceeding to generate the response based on these steps). This design outlines a production-grade Numerical Reproducibility Testbed (NRT). This system acts as a "CI/CD for Math," ensuring that code changes, dependency updates, or hardware migrations do not silently alter model behavior due to floating-point intricacies.
The architecture is divided into three layers: Instrumentation (Data Capture), Policy (Evaluation Logic), and Integration (Automation).
1. Architecture Overview
The testbed relies on the Golden Master pattern.
- Reference Run: Execute the pipeline on a "stable" environment (e.g., CPU or a specific GPU driver) to generate baseline tensors.
- Target Run: Execute the pipeline on the new environment (e.g., a different GPU architecture or after a library bump).
- Comparator: Apply tolerance policies to the divergences.
2. Layer 1: Instrumentation (The Hooks)
We need to capture tensor states without cluttering the model code. We use a decorator/context-manager approach to snapshot tensors at critical points (Forward pass, Loss calc, Gradients).
The
NumericSpyToolThis utility wraps PyTorch modules (or TensorFlow/Keras layers) to dump tensor snapshots to disk or memory.
# instrumentation.py import torch import json from pathlib import Path from typing import Dict, List class NumericSpy: def __init__(self, log_dir: str = "./numeric_logs"): self.log_dir = Path(log_dir) self.log_dir.mkdir(exist_ok=True) self.manifest = {} def snapshot(self, tensor: torch.Tensor, name: str, step: int): """ Saves a tensor to disk with high precision. For large tensors, statistical sampling (mean/std/histogram) is preferred over full dumps to save IO. """ if tensor is None: return # Convert to CPU numpy for consistent storage format data = tensor.detach().cpu().numpy() # File naming convention: step_name.npy fname = f"{step:04d}_{name}.npy" path = self.log_dir / fname # Save raw data (ensure float64 preservation if casted) np.save(path, data.astype(np.float64)) # Log metadata self.manifest[name] = { "shape": list(data.shape), "dtype": str(tensor.dtype), "file": fname } def save_manifest(self): with open(self.log_dir / "manifest.json", "w") as f: json.dump(self.manifest, f, indent=2) def instrument_model(model: torch.nn.Module, spy: NumericSpy, step: int): """ Registers forward hooks to capture layer outputs. """ hooks = [] def make_hook(name): def hook(module, input, output): # Capture Output Mean and Std Dev (Lightweight monitoring) # Or capture full tensor (Deep debugging) if isinstance(output, torch.Tensor): spy.snapshot(output, f"{name}_output", step) return hook for name, layer in model.named_modules(): if name: # Avoid root hook hooks.append(layer.register_forward_hook(make_hook(name))) return hooks
3. Layer 2: Tolerance Policy (The Logic)
Comparing floating points requires nuance. A strict
==fails due to non-associativity of floating-point arithmetic (e.g.,(A+B)+C != A+(B+C)on GPUs). We implement a hierarchical policy engine.The Policy Engine
# policy_engine.py import numpy as np from dataclasses import dataclass from enum import Enum class AlertLevel(Enum): PASS = 0 WARNING = 1 # Within tolerance, but trending up FAILURE = 2 # Exceeded tolerance @dataclass class ToleranceSpec: rtol: float = 1e-5 # Relative tolerance atol: float = 1e-8 # Absolute tolerance ulp_limit: int = 4 # Units in the Last Place (strict bit-level check) # Context-aware rules # Gradients usually noisier than weights tensor_type_modifiers = { "gradient": {"rtol": 1e-4}, "weight": {"rtol": 1e-5}, "activation": {"rtol": 1e-5} } class DriftComparator: def __init__(self, policy: ToleranceSpec): self.policy = policy def compute_ulp(self, a, b): """Computes Units in the Last Place difference.""" # Simplification: View bits as integers and subtract # Note: Complex implementation needed for handling signs/NaNs if a == b: return 0 return abs(np.view(np.float64(a), np.int64) - np.view(np.float64(b), np.int64)) def compare(self, tensor_a: np.ndarray, tensor_b: np.ndarray, name: str): # 1. Shape Check if tensor_a.shape != tensor_b.shape: return AlertLevel.FAILURE, "Shape mismatch" # 2. NaN/Inf Check if np.any(np.isnan(tensor_b)) and not np.any(np.isnan(tensor_a)): return AlertLevel.FAILURE, "Target generated NaNs" # 3. Statistical Check (Fast fail) if not np.allclose(tensor_a, tensor_b, rtol=self.policy.rtol, atol=self.policy.atol): # 4. Deep Dive: Error Distribution diff = np.abs(tensor_a - tensor_b) max_diff = np.max(diff) mean_diff = np.mean(diff) # Check if errors are uniformly distributed (random noise) # vs biased (algorithmic error). # If mean_diff is close to 0, it's likely just precision drift. # If mean_diff is high, logic is broken. details = { "max_diff": max_diff, "mean_diff": mean_diff, "status": "FAIL" } return AlertLevel.FAILURE, details return AlertLevel.PASS, {"status": "PASS"}
4. Layer 3: CI Integration (The Automation)
We need a workflow that runs on every Pull Request (PR). We use GitHub Actions (conceptually) to demonstrate.
The Workflow Strategy
- Matrix Job: Run the same test script on different hardware (e.g.,
runner: ubuntu-latestvsrunner: [self-hosted-gpu-a100]). - Artifact Upload: Upload the
NumericSpylogs from the reference run. - Comparison Job: Download artifacts and run
DriftComparator.
GitHub Actions Workflow (
.github/workflows/numeric_ci.yml)name: Numeric Reproducibility Check on: [pull_request] jobs: generate-golden: runs-on: ubuntu-latest # CPU Reference steps: - uses: actions/checkout@v3 - name: Setup Python uses: actions/setup-python@v4 with: python-version: '3.10' - name: Install Deps run: pip install -r requirements.txt - name: Run Training Step (Generate Golden) run: python train.py --mode snapshot --output_dir ./golden_logs - name: Upload Golden Artifact uses: actions/upload-artifact@v3 with: name: golden-logs path: ./golden_logs check-target: runs-on: [self-hosted, gpu] # Target Hardware needs: generate-golden steps: - uses: actions/checkout@v3 - name: Download Golden uses: actions/download-artifact@v3 with: name: golden-logs path: ./golden_logs - name: Run Training Step (Target) run: python train.py --mode snapshot --output_dir ./target_logs - name: Compare Results id: comparison run: | python ci_compare.py \ --ref ./golden_logs \ --target ./target_logs \ --report ./drift_report.json - name: Fail if Drift Detected run: | if grep -q '"status": "FAILURE"' ./drift_report.json; then echo "::error::Numerical drift detected exceeding tolerance." cat ./drift_report.json exit 1 fi
5. Implementation: The Comparison Script (
ci_compare.py)This script ties the instrumentation to the policy.
import argparse import numpy as np import json from policy_engine import DriftComparator, ToleranceSpec def main(ref_dir, target_dir, report_path): policy = ToleranceSpec() comparator = DriftComparator(policy) # Load Manifests with open(f"{ref_dir}/manifest.json") as f: ref_manifest = json.load(f) results = [] for tensor_name, meta in ref_manifest.items(): ref_file = f"{ref_dir}/{meta['file']}" tgt_file = f"{target_dir}/{meta['file']}" try: ref_tensor = np.load(ref_file) tgt_tensor = np.load(tgt_file) level, msg = comparator.compare(ref_tensor, tgt_tensor, tensor_name) results.append({ "tensor": tensor_name, "status": level.name, "details": msg }) if level.name == "FAILURE": print(f"[FAIL] {tensor_name}: {msg}") else: print(f"[PASS] {tensor_name}") except FileNotFoundError: results.append({ "tensor": tensor_name, "status": "ERROR", "details": "File missing in target run" }) with open(report_path, "w") as f: json.dump(results, f, indent=2) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--ref", required=True) parser.add_argument("--target", required=True) parser.add_argument("--report", required=True) args = parser.parse_args() main(args.ref, args.target, args.report)Summary of Features
- Hardware Agnostic: By casting to
float64during snapshotting, we normalize the bit-representation differences between hardware (e.g., TF32 on Ampere GPUs vs FP32 on Volta) to standard Double Precision for comparison. - Non-Intrusive: The
instrument_modelfunction allows dropping the testbed into existing codebases with minimal refactoring. - Tunable: Adjusting
ToleranceSpecallows teams to define strictness for specific layers (e.g., allowing more drift in Softmax outputs than in Weight matrices). - CI-Ready: Exits with code 1 on drift, blocking merges that introduce numerical instability.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.