Engineer a machine learning experiment tracking system with reversible lineage: dataset…
- User
Engineer a machine learning experiment tracking system with reversible lineage: dataset snapshots, environment capture, dependency graphs, and compliance exports.
Assistant
The user wants a comprehensive Machine Learning (ML) experiment tracking system. The key features requested are:
- Reversible Lineage: The ability to trace back from a model/result to everything that created it (data, code, config) and forward from a change to see what it affects (though "reversible" usually emphasizes the backward trace in this context, i.e., full reproducibility).
- Dataset Snapshots: Versioning data, not just referencing it.
- Environment Capture: Capturing the software stack (libraries, versions, possibly OS/hardware info).
- Dependency Graphs: Visualizing or storing the relationships between code, data, parameters, and models.
- Compliance Exports: Generating reports or data packages suitable for audits or regulatory bodies.
This is a complex software engineering task. It requires defining an architecture, choosing a stack, and detailing specific components.
1. Architecture & Core Concepts:
- Entity-Centric: Everything is an object (Experiment, Run, Dataset, Model, Artifact).
- Graph-Based: The relationships form a Directed Acyclic Graph (DAG).
- Storage: Need a database for metadata (SQL or Graph DB) and an object store (S3, MinIO, local) for artifacts.
2. Component Breakdown:
-
Lineage & Dependency Graphs:
- How do we track this? We need a context manager or a decorator that wraps the training function.
- Inside the wrapper, we log inputs (params, data refs) and outputs (model files, metrics).
- We need a schema to store
Parent -> Childrelationships.
-
Dataset Snapshots:
- We can't just store file paths. We need content-addressable storage (hashing) or versioning (like DVC or Delta Lake).
- For a custom solution, let's implement a
SnapshotManagerthat calculates a hash of the data (or a sample) and copies it to a versioned store.
-
Environment Capture:
- Python:
pip freezeorconda env export. - Container: Docker image IDs.
- Hardware:
gpuinfo,cpuinfo.
- Python:
-
Compliance Exports:
- JSON-LD or a structured PDF/Data package.
- Must include: Who ran it, when, with what data, and the result.
3. Implementation Strategy (Python): I will design a modular Python library called
ReversiML(placeholder name) or justMLTracker.- Module 1: Core Store (Metadata): SQLite for simplicity in a standalone example, but designed so it could be swapped for Postgres/Neo4j.
- Module 2: Artifact Store: Local file system abstraction.
- Module 3: Lineage Tracker: The API the user interacts with.
- Module 4: Compliance: Reporting tools.
4. Drafting the Code Structure:
-
Imports:
os,json,hashlib,datetime,subprocess,sqlite3,networkx(for graph logic),pandas(for data handling). -
Class
LineageNode: Represents a node in the graph (Run, Dataset, Param). -
Class
ExperimentTracker:start_run(): Captures environment, creates ID.log_dataset(): Hashes input data, saves snapshot/metadata.log_params(): Inputs.log_model(): Outputs.end_run(): Finalizes.- Reversible logic: Store
input_hashandoutput_hash.
-
Compliance: A function
export_compliance_report(run_id).
5. Refining specific requirements:
- Reversible Lineage: This implies we need to query the graph. "Give me everything that used Dataset V1". "Give me the exact code version used for Model V2".
- Environment: I'll add a helper to dump
pip freeze. - Dependency Graph: I'll use
networkxto build the graph and provide a method to visualize or traverse it.
6. Writing the Code (Mental Draft & Refinement):
-
Database Schema:
runs: id, timestamp, status, env_hash, code_hash.artifacts: id, type (dataset/model), path, hash.lineage: parent_id, child_id, relation_type.environment: run_id, package_list (JSON).
-
Dataset Snapshotting: Instead of copying massive files (which makes the example hard to run), I will implement a "Copy-on-Write" simulation or just strict metadata hashing + copying to a
.mltrack/artifactsfolder. -
Compliance: Generate a JSON manifest.
7. Adding "Reversibility": The key is the query capability.
- Forward: Run -> Models.
- Backward: Model -> Run -> Dataset -> Raw Data.
8. Final Polish: Make the code self-contained and runnable. Use type hinting. Add comments explaining why specific choices were made (e.g., why SHA256 for lineage).
Self-Correction during drafting: A full UI is too much code. I will provide a CLI interface or just Python API with a printout of the graph structure. I'll stick to a Python API with a
visualizemethod that prints ASCII or generates a Graphviz dot file (conceptually).Let's structure the solution into:
- Architecture Overview
- Core Implementation (The Python Library)
- Usage Example (Demonstrating the workflow)
- Compliance Export Logic
Specific detail on Environment: Ensure we capture library versions. Specific detail on Snapshots: Calculate a checksum.
Let's write the code.
Plan:
- Define
ArtifactStore: handles file versioning/hashing. - Define
MetadataStore: SQLite backend for graph nodes/edges. - Define
EnvironmentCapture: System introspection. - Define
Tracker: The main API. - Define
LineageGraph: Logic to traverse dependencies. - Define
ComplianceExporter: Generate the audit trail.
Let's refine the "Snapshot" part. If the user provides a Pandas DataFrame, hash it and save to parquet. If a file path, hash and copy.
Let's refine "Reversible". If I have a Model A, I need to find the Run, then the Params, then the Dataset.
Okay, ready to generate the response. This solution engineers a complete, standalone Machine Learning Experiment Tracking system called "Memento". It is designed to run locally using Python, SQLite (for metadata), and the filesystem (for artifact storage), replacing the need for external servers like MLflow or Weight & Biases for this specific context.
System Architecture
- Metadata Store (SQLite): Stores the relational graph of Experiments, Runs, Parameters, Metrics, and Lineage pointers.
- Artifact Store (Filesystem): A content-addressable store (CAS) where files (datasets, models) are saved by their SHA-256 hash to ensure immutability and deduplication.
- Lineage Engine: Tracks the DAG (Directed Acyclic Graph) connecting Datasets Runs Models.
- Compliance Module: Exports a self-contained, human-readable bundle (JSON + Binaries) for auditors.
1. The Implementation
import os import json import sqlite3 import hashlib import shutil import subprocess import sys import pandas as pd from datetime import datetime from typing import Optional, Dict, List, Any, Tuple from pathlib import Path import pickle # ========================================== # 1. Core Storage & Database Schema # ========================================== class MementoCore: def __init__(self, root_dir: str = "./memento_store"): self.root_path = Path(root_dir) self.db_path = self.root_path / "metadata.db" self.artifact_path = self.root_path / "artifacts" if not self.root_path.exists(): self._initialize_store() def _initialize_store(self): """Setup directory structure and SQL Schema.""" self.root_path.mkdir(parents=True) self.artifact_path.mkdir(parents=True) conn = sqlite3.connect(self.db_path) cursor = conn.cursor() # Schema designed for Lineage and Compliance cursor.execute(""" CREATE TABLE experiments ( experiment_id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT UNIQUE, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ) """) cursor.execute(""" CREATE TABLE runs ( run_uuid TEXT PRIMARY KEY, experiment_id INTEGER, status TEXT DEFAULT 'running', start_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP, end_time TIMESTAMP, environment_json TEXT, FOREIGN KEY(experiment_id) REFERENCES experiments(experiment_id) ) """) cursor.execute(""" CREATE TABLE lineage_edges ( child_uuid TEXT, parent_uuid TEXT, relation_type TEXT, -- 'DATASET_INPUT', 'MODEL_OUTPUT', 'RUN_DERIVATION' PRIMARY KEY (child_uuid, parent_uuid, relation_type) ) """) cursor.execute(""" CREATE TABLE artifacts ( artifact_hash TEXT PRIMARY KEY, original_filename TEXT, stored_path TEXT, size_bytes INTEGER, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ) """) # Key-Value tables for flexible params/metrics cursor.execute("CREATE TABLE params (run_uuid TEXT, key TEXT, value TEXT, PRIMARY KEY (run_uuid, key))") cursor.execute("CREATE TABLE metrics (run_uuid TEXT, key TEXT, value REAL, step INTEGER, PRIMARY KEY (run_uuid, key, step))") cursor.execute("CREATE TABLE tags (run_uuid TEXT, key TEXT, value TEXT, PRIMARY KEY (run_uuid, key))") conn.commit() conn.close() def get_conn(self): return sqlite3.connect(self.db_path) # ========================================== # 2. Environment & Dependency Capture # ========================================== class EnvironmentCapture: @staticmethod def capture_snapshot() -> Dict[str, Any]: """ Captures the OS, Python version, and installed packages (pip). In a production system, this would also capture Docker image IDs or Conda specs. """ snapshot = { "python_version": sys.version, "platform": sys.platform, "dependencies": [] } try: # Attempt to capture pip list result = subprocess.run( [sys.executable, "-m", "pip", "list", "--format=json"], capture_output=True, text=True ) if result.returncode == 0: snapshot["dependencies"] = json.loads(result.stdout) except Exception as e: snapshot["capture_error"] = str(e) return snapshot # ========================================== # 3. Artifact & Hashing System # ========================================== class ArtifactManager: def __init__(self, core: MementoCore): self.core = core self.artifact_dir = core.artifact_path def _compute_hash(self, file_path: Path) -> str: sha256 = hashlib.sha256() with open(file_path, "rb") as f: for chunk in iter(lambda: f.read(4096), b""): sha256.update(chunk) return sha256.hexdigest() def log_artifact(self, local_path: str, run_uuid: str, artifact_type: str) -> str: """ Ingests a file, saves it by hash (deduplication), and updates lineage. """ src_path = Path(local_path) if not src_path.exists(): raise FileNotFoundError(f"Artifact not found: {local_path}") file_hash = self._compute_hash(src_path) dest_path = self.artifact_dir / file_hash # Copy only if not exists (Content Addressable Storage) if not dest_path.exists(): shutil.copy2(src_path, dest_path) # Update Metadata conn = self.core.get_conn() cursor = conn.cursor() # Register artifact cursor.execute("INSERT OR IGNORE INTO artifacts (artifact_hash, original_filename, stored_path, size_bytes) VALUES (?, ?, ?, ?)", (file_hash, src_path.name, str(dest_path), src_path.stat().st_size)) # Create Lineage Edge # If this is an input dataset: Parent=Hash, Child=Run # If this is an output model: Parent=Run, Child=Hash if artifact_type == "INPUT": cursor.execute("INSERT INTO lineage_edges (parent_uuid, child_uuid, relation_type) VALUES (?, ?, ?)", (file_hash, run_uuid, 'DATASET_INPUT')) elif artifact_type == "OUTPUT": cursor.execute("INSERT INTO lineage_edges (parent_uuid, child_uuid, relation_type) VALUES (?, ?, ?)", (run_uuid, file_hash, 'MODEL_OUTPUT')) conn.commit() conn.close() return file_hash def retrieve_artifact(self, artifact_hash: str) -> Path: """Retrieves the physical path of an artifact by its hash.""" return self.artifact_dir / artifact_hash # ========================================== # 4. The Experiment Tracker (Main Interface) # ========================================== class ExperimentTracker: def __init__(self, root_dir: str = "./memento_store"): self.core = MementoCore(root_dir) self.artifacts = ArtifactManager(self.core) self.current_run_id = None def create_experiment(self, name: str) -> int: conn = self.core.get_conn() cursor = conn.cursor() try: cursor.execute("INSERT INTO experiments (name) VALUES (?)", (name,)) conn.commit() return cursor.lastrowid except sqlite3.IntegrityError: cursor.execute("SELECT experiment_id FROM experiments WHERE name=?", (name,)) return cursor.fetchone()[0] finally: conn.close() def start_run(self, experiment_name: str) -> str: import uuid run_uuid = str(uuid.uuid4()) exp_id = self.create_experiment(experiment_name) # Capture Environment env_snap = EnvironmentCapture.capture_snapshot() conn = self.core.get_conn() cursor = conn.cursor() cursor.execute("INSERT INTO runs (run_uuid, experiment_id, environment_json) VALUES (?, ?, ?)", (run_uuid, exp_id, json.dumps(env_snap))) conn.commit() conn.close() self.current_run_id = run_uuid return run_uuid def log_params(self, params: Dict[str, Any]): conn = self.core.get_conn() cursor = conn.cursor() for k, v in params.items(): cursor.execute("INSERT INTO params (run_uuid, key, value) VALUES (?, ?, ?)", (self.current_run_id, k, str(v))) conn.commit() conn.close() def log_metric(self, key: str, value: float, step: int = 0): conn = self.core.get_conn() cursor = conn.cursor() cursor.execute("INSERT INTO metrics (run_uuid, key, value, step) VALUES (?, ?, ?, ?)", (self.current_run_id, key, value, step)) conn.commit() conn.close() def log_dataset(self, file_path: str, alias: str = None): """Logs an input dataset snapshot.""" h = self.artifacts.log_artifact(file_path, self.current_run_id, "INPUT") if alias: self._tag_run(f"dataset_alias:{alias}", h) def log_model(self, file_path: str, alias: str = "final_model"): """Logs an output model.""" h = self.artifacts.log_artifact(file_path, self.current_run_id, "OUTPUT") if alias: self._tag_run(f"model_alias:{alias}", h) def _tag_run(self, key, value): conn = self.core.get_conn() cursor = conn.cursor() cursor.execute("INSERT INTO tags (run_uuid, key, value) VALUES (?, ?, ?)", (self.current_run_id, key, value)) conn.commit() conn.close() def end_run(self, status: str = "FINISHED"): conn = self.core.get_conn() cursor = conn.cursor() cursor.execute("UPDATE runs SET status=?, end_time=CURRENT_TIMESTAMP WHERE run_uuid=?", (status, self.current_run_id)) conn.commit() conn.close() # ========================================== # 5. Reversible Lineage & Compliance # ========================================== def get_lineage_graph(self, run_uuid: str) -> Dict: """ Reconstructs the full graph for a specific run. """ conn = self.core.get_conn() cursor = conn.cursor() nodes = [] edges = [] # 1. Get Run Metadata cursor.execute("SELECT * FROM runs WHERE run_uuid=?", (run_uuid,)) run_data = cursor.fetchone() nodes.append({"id": run_uuid, "type": "RUN", "data": run_data}) # 2. Get Inputs (Parents where relation is DATASET_INPUT) cursor.execute("SELECT parent_uuid FROM lineage_edges WHERE child_uuid=? AND relation_type='DATASET_INPUT'", (run_uuid,)) inputs = cursor.fetchall() for row in inputs: artifact_hash = row[0] cursor.execute("SELECT original_filename, created_at FROM artifacts WHERE artifact_hash=?", (artifact_hash,)) meta = cursor.fetchone() nodes.append({"id": artifact_hash, "type": "INPUT_DATASET", "name": meta[0] if meta else "Unknown"}) edges.append({"source": artifact_hash, "target": run_uuid}) # 3. Get Outputs (Children where relation is MODEL_OUTPUT) cursor.execute("SELECT child_uuid FROM lineage_edges WHERE parent_uuid=? AND relation_type='MODEL_OUTPUT'", (run_uuid,)) outputs = cursor.fetchall() for row in outputs: artifact_hash = row[0] cursor.execute("SELECT original_filename, created_at FROM artifacts WHERE artifact_hash=?", (artifact_hash,)) meta = cursor.fetchone() nodes.append({"id": artifact_hash, "type": "OUTPUT_MODEL", "name": meta[0] if meta else "Unknown"}) edges.append({"source": run_uuid, "target": artifact_hash}) conn.close() return {"nodes": nodes, "edges": edges} def export_compliance_bundle(self, run_uuid: str, export_dir: str): """ Generates a directory with all necessary artifacts and a manifest.json for regulatory audits. """ exp_path = Path(export_dir) / f"audit_bundle_{run_uuid}" exp_path.mkdir(parents=True, exist_ok=True) lineage = self.get_lineage_graph(run_uuid) manifest = { "run_id": run_uuid, "export_time": datetime.now().isoformat(), "lineage_graph": lineage, "environment": None, "parameters": [], "metrics": [], "files": {} } conn = self.core.get_conn() cursor = conn.cursor() # Get Env cursor.execute("SELECT environment_json FROM runs WHERE run_uuid=?", (run_uuid,)) res = cursor.fetchone() if res: manifest["environment"] = json.loads(res[0]) # Get Params/Metrics cursor.execute("SELECT key, value FROM params WHERE run_uuid=?", (run_uuid,)) manifest["parameters"] = [{"key": r[0], "value": r[1]} for r in cursor.fetchall()] cursor.execute("SELECT key, value, step FROM metrics WHERE run_uuid=?", (run_uuid,)) manifest["metrics"] = [{"key": r[0], "value": r[1], "step": r[2]} for r in cursor.fetchall()] # Copy Artifacts for node in lineage['nodes']: if node['type'] in ['INPUT_DATASET', 'OUTPUT_MODEL']: a_hash = node['id'] src_path = self.artifacts.retrieve_artifact(a_hash) dst_path = exp_path / f"{node['type']}_{node['name']}_{a_hash[:8]}" shutil.copy2(src_path, dst_path) manifest["files"][node['id']] = str(dst_path.name) # Generate Checksum file for verification with open(str(dst_path) + ".sha256", "w") as f: f.write(f"{a_hash} {dst_path.name}") conn.close() # Write Manifest with open(exp_path / "manifest.json", "w") as f: json.dump(manifest, f, indent=4) print(f"Compliance bundle exported to: {exp_path}") return exp_path2. Usage Simulation
Here we simulate a data science workflow. We create a "dataset," run an experiment, log the model, and then export the compliance bundle.
# 1. Setup tracker = ExperimentTracker(root_dir="./ml_ops_db") # 2. Create Dummy Data (Simulating external data generation) data = pd.DataFrame({'feature_a': [1, 2, 3], 'feature_b': [4, 5, 6], 'target': [0, 1, 0]}) data_path = "dataset_v1.csv" data.to_csv(data_path, index=False) # 3. Start Run run_id = tracker.start_run("Classifier_Exp_01") # 4. Log Inputs, Params, Env tracker.log_dataset(data_path, alias="raw_training_data") tracker.log_params({"learning_rate": 0.01, "epochs": 10, "model_type": "RandomForest"}) # 5. Simulate Training print(f"Training model for run {run_id}...") # ... training logic here ... metrics = {"accuracy": 0.95, "f1_score": 0.92} # 6. Log Outputs tracker.log_metric("accuracy", metrics["accuracy"]) tracker.log_metric("f1_score", metrics["f1_score"]) # Save dummy model import joblib class DummyModel: def predict(self, x): return [0] model_path = "model.joblib" joblib.dump(DummyModel(), model_path) tracker.log_model(model_path, alias="production_candidate") tracker.end_run()3. Demonstrating Reversible Lineage & Compliance
Now we utilize the unique features: tracing back from the result and exporting the audit bundle.
print("\n--- Lineage Trace ---") lineage_data = tracker.get_lineage_graph(run_id) print(json.dumps(lineage_data, indent=2)) print("\n--- Compliance Export ---") audit_path = tracker.export_compliance_bundle(run_id, "./compliance_audits") # Verification print("\n--- Verification ---") print("Bundle Contents:", os.listdir(audit_path))Key Engineering Decisions
- Content Addressable Storage (CAS): By renaming files to their hash, we guarantee immutability. If
dataset_v1.csvchanges but the filename remains the same, the hash changes, preventing "silent" data corruption in the lineage. - Separation of Metadata and Artifacts: The database stays lightweight (JSON/SQL) while heavy binaries are managed by the file system (or S3 in a cloud variant).
- Lineage Schema: The
lineage_edgestable is generic. It supportsDATASET -> RUN(Input) andRUN -> MODEL(Output). This forms a Directed Acyclic Graph (DAG) enabling the "Reversible" requirement (tracing a model back to the exact data and parameters). - Compliance Bundle: The export function creates a "Frozen" directory. It includes the artifacts, the
manifest.json(the logic), and.sha256files (the verification), satisfying audit requirements for reproducibility.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.