Design a computational reproducibility rubric: environment capture, seeds, provenance,…
- User
Design a computational reproducibility rubric: environment capture, seeds, provenance, and review artifacts.
- Assistant
Computational Reproducibility Rubric
Purpose
This rubric evaluates the reproducibility of computational experiments (e.g., in machine learning, simulations, data analysis). It assesses four core pillars: Environment Capture (software/hardware setup), Seeds (stochastic reproducibility), Provenance (data and computation lineage), and Review Artifacts (verification materials).
Scoring Scale (per criterion):
- 4 (Excellent): Fully meets/exceeds best practices; enables bit-for-bit reproduction.
- 3 (Good): Meets standards; reproduction likely with minor effort.
- 2 (Fair): Partial compliance; reproduction possible but error-prone.
- 1 (Poor): Minimal compliance; reproduction unlikely.
- 0 (None): Absent or misleading.
Total Score: Sum of all criteria (max 100 points).
Thresholds: ≥90 = Excellent; 75-89 = Good; 50-74 = Fair; <50 = Poor.Use this rubric for self-assessment, peer review, or publication checklists. Provide evidence (e.g., links/files) for scores.
1. Environment Capture (25 points max)
Ensures the exact runtime environment is replicable, minimizing "it works on my machine" issues.
Criterion Description Score (0-4) Evidence Dependency Specification All software deps (e.g., Python/R version, libraries) listed precisely (e.g., requirements.txt,environment.yml). No version ranges.Containerization/Virtualization Environment fully containerized (Docker/Podman) or virtualized (Conda/Venv) with a shareable image/file. Includes hardware specs (CPU/GPU, RAM). OS & System-Level Capture OS version, kernel, and system tools (e.g., via Dockerfileorsingularity.def) specified. Reproducible on multiple platforms tested.Reproducibility Test Instructions + script to rebuild env and run a smoke test (e.g., docker run --rm myimage python -c "import numpy; print(numpy.__version__)").Subtotal /16 Notes: Bonus +9 points if env auto-builds from a Git repo (e.g., GitHub Actions badge).
2. Seeds (25 points max)
Controls randomness for deterministic outputs in stochastic code (e.g., ML training, sampling).
Criterion Description Score (0-4) Evidence Seed Declaration All random sources seeded explicitly (e.g., numpy.random.seed(42),torch.manual_seed(42),random.seed(42)). Covers NumPy, PyTorch/TensorFlow, Python random, OS RNG.Global/Per-Module Seeds Seeds set globally at script start and per-module (e.g., CUDA, multiprocessing). Environment vars used for overrides. Multi-Run Verification Code runs ≥3 times with same seed yields identical outputs (e.g., hashes match). Tested and logged. Seed Documentation Seeds listed in README with rationale (e.g., "42 for universality"). Instructions to set PYTHONHASHSEED=0.Subtotal /16 Notes: Bonus +9 points if seeds are configurable via CLI/args with defaults.
3. Provenance (25 points max)
Tracks data origins, transformations, and computation history for auditability.
Criterion Description Score (0-4) Evidence Data Provenance All inputs (datasets, models) cited with URLs/hashes (SHA256). Raw data accessible; no "email me for data." Computation Lineage Workflow graph (e.g., via MLflow, DVC, or Jupyter provenance) logs steps, params, intermediates. Git commit pinned. Version Control Integration Code/data versioned in Git (or DVC). pip freezeor lockfile included. Repro instructions use exact commit.Output Provenance Final results include metadata (e.g., JSON with hashes, params). Tamper-evident (e.g., signed artifacts). Subtotal /16 Notes: Bonus +9 points if uses standards like RO-Crate or W3C PROV.
4. Review Artifacts (25 points max)
Provides verifiable materials for independent review/reproduction.
Criterion Description Score (0-4) Evidence Code & Instructions Self-contained repo with README.md: one-command repro (e.g.,make repro). No external depedencies beyond env.Executable Notebooks/Scripts Interactive artifacts (e.g., nbconvert to HTML/PDF, Binder/Colab link) showing inputs → outputs. Expected Outputs Checksums/hashes for all outputs (e.g., md5sum results.csv). Diffable baselines.Review Script Automated checker (e.g., repro-test.py) that fails on mismatch. Runtime <1hr on standard hardware.Subtotal /16 Notes: Bonus +9 points if hosted reproducibly (e.g., Zenodo DOI, Binder badge).
Usage Guidelines
- Assessor Instructions: Score based on provided materials. Require hyperlinks to artifacts.
- Improvement Tips:
Score Range Action 90-100 Publish as-is. 75-89 Fix partial items. 50-74 Major env/provenance gaps; rebuild. <50 Start over with tools like Cookiecutter or ReproZip. - Tools Recommendations: Docker, DVC, MLflow, Weights & Biases, Papers with Code reproducibility stamps.
This rubric aligns with standards from ACM, NeurIPS, and ReScience. Total: /100. Comments: ________________
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.