Build a large-file artifact store: CAS, GC, compaction, and retention.
- User
Build a large-file artifact store: CAS, GC, compaction, and retention.
Assistant
The user wants a "large-file artifact store." This is a systems design and implementation task. It's a broad request, so I need to narrow down the scope while ensuring I cover the key requirements mentioned: CAS (Content Addressable Storage), GC (Garbage Collection), Compaction, and Retention.
1. Understanding the Requirements:
- Large-file: The system needs to handle files that are potentially gigabytes in size. This rules out keeping everything in memory. It implies streaming, chunking, or efficient disk I/O.
- Artifact Store: A place to store build artifacts (binaries, logs, docker images, etc.).
- CAS (Content Addressable Storage): Files are stored and retrieved by their hash (usually SHA-256) rather than a filename. This provides deduplication (same content = same hash = stored only once) and integrity.
- GC (Garbage Collection): Reclaiming space used by artifacts that are no longer needed.
- Compaction: This is interesting. In a CAS context, compaction usually relates to:
- If we use chunking, maybe compacting small chunks into larger files.
- If we use a log-structured approach, rewriting files to remove "holes" left by deleted data (though CAS usually implies immutable blobs, so "deletion" is the tricky part).
- Or, it might refer to deduplication savings. Let's assume the user means "reclaiming disk space efficiently" or "merging fragments."
- Retention: Policies for how long to keep things (e.g., "keep last 30 days" or "keep last 5 builds").
2. High-Level Architecture:
I'll design a Python-based implementation for clarity, but structure it like a real system.
- Storage Layer: The physical filesystem.
- Index/Metadata Layer: A database (SQLite for simplicity in this example) to map hashes to files and track references/usage.
- API Layer: Methods to
put,get, anddelete(ref count decrement).
3. Addressing Specific Components:
-
CAS Implementation:
- Input: File stream.
- Action: Calculate SHA-256.
- Storage: Create a directory structure based on the hash (e.g.,
ab/cdef...) to avoid too many files in one directory. - Deduplication: Check if hash exists before writing.
-
Retention:
- We need to associate a "timestamp" or "tag" with an artifact. Let's assume we store "blobs" (the content) and "manifests" (metadata about the artifact).
- Retention policies usually apply to the manifests (e.g., "delete build X"), which triggers the deletion of the underlying blobs if they aren't used elsewhere.
-
Garbage Collection (GC):
- Direct deletion is dangerous in CAS because multiple artifacts might point to the same blob (deduplication).
- Approach: Reference Counting or Mark-and-Sweep.
- Ref Counting: Fast, but can be tricky with crashes.
- Mark-and-Sweep: Safer. Traverse the "live" manifests, mark all referenced blobs, delete unmarked blobs.
- Hybrid: Reference counting in the DB for quick checks, and a periodic sweep for orphans.
-
Compaction:
- If the store is just a collection of loose files in directories, "compaction" isn't really a filesystem term (the OS handles blocks).
- However, if we want efficient storage, maybe we pack files into large append-only archives (like git packfiles or tarballs) and index into them.
- Decision: For a "large-file" store, packing might hurt performance (read requires seeking inside a huge archive). Let's stick to a "loose object" format for the core implementation, but mention that "Compaction" in this context usually means "Defragmentation of free space" or "Rewriting the DB".
- Correction: In many artifact stores (like Artifactory/Nexus), "compaction" isn't a standard term. The user might mean Deduplication (which CAS handles) or Sparse File reclamation. I will implement a "Vacuum" or "GC" process that effectively compacts the storage by removing dead objects.
4. Refining the "Compaction" Requirement: Actually, looking at high-performance object stores (like CASFS or IPFS implementations), they often pack small files. But the prompt says "Large-file". If the files are huge, packing them is bad. Interpretation: I will interpret "Compaction" as the process of the GC actually deleting files from disk to free up space, or potentially merging "shards" if I went that route (which I won't for simplicity). I'll stick to GC = Deletion.
5. Designing the Class Structure:
ArtifactStore: Main entry point.BlobManager: Handles the raw bytes (read/write files by hash).MetadataManager: Handles the DB (SQLite). Tracks:blobstable: hash, size, ref_count.artifactstable: name, version, timestamp, retention_policy.artifact_blobsmap: links artifacts to blobs.
6. Implementation Steps (Mental Draft):
- Init: Create dirs, init DB.
- Put:
- Calculate hash.
- Store blob (if new).
- Increment ref count in DB (or create entry).
- Create artifact entry linking to blob.
- Get: Read blob.
- Retention Job:
- Find artifacts older than policy.
- Unlink them from blobs (decrement ref counts).
- Delete artifact metadata.
- GC/Compaction:
- Find blobs with ref_count <= 0 (or non-existent in metadata).
- Delete files from disk.
- Clean up DB.
7. Coding Details (Python):
- Use
hashlibfor SHA256. - Use
shutilfor moving files. - Use
sqlite3for metadata. - Need to handle concurrency? I'll stick to a single-process model for the code snippet but mention locking strategies.
8. Refinement on "Large File" Handling:
- Don't read the whole file into RAM.
- Use chunks:
for chunk in iter(lambda: f.read(4096), b''):
9. Constructing the Response:
- Introduction: Define the architecture (CAS + Ref Counting).
- The Code: A single Python script (or organized classes) implementing the core logic.
- Class
CASStore: The heavy lifter. - Method
ingest: Streaming hash and storage. - Method
apply_retention: The logic to expire old artifacts. - Method
collect_garbage: The logic to delete unreferenced blobs.
- Class
- Explanation: Walk through how CAS prevents duplication, how Retention works logically, and how GC ensures physical disk cleanup.
Self-Correction during drafting:
- Issue: Simple ref counting in a distributed system is hard.
- Fix: I will assume a local file store for this demo but design the GC as "Mark and Sweep" style because it's more robust for retention policies. Marking "live" set based on retention, sweeping the rest.
- Actually: Ref counting is better for immediate feedback. Let's do ref counting in the DB.
Revised Plan for "Compaction": Often, "Compaction" in database terms means rewriting the storage file to remove gaps. In a file-based CAS, files are immutable. The only "gap" is deleted files. So, the GC is the compactor.
Let's write the Python solution. Focus on:
- Hashing: SHA-256.
- Storage:
data/ab/cdef... - Database:
artifacts.db. - Streaming: 64KB chunks.
Drafting the Retention Logic: Retention is usually "Keep last N" or "Keep newer than T". I'll implement a simple "TTL" (Time To Live).
Drafting the GC: Two-step process.
- Expiration: Soft delete. Update DB to say "this artifact is gone", decrement blob refs.
- Purge (GC): Hard delete. Look for blob refs == 0, delete file.
Let's go. This is a design and implementation for a Content-Addressable Storage (CAS) system optimized for large files. It includes a metadata index for fast lookups, reference counting for garbage collection, and a compaction strategy to manage disk usage.
Architecture Overview
- Storage Layer (CAS): Files are stored based on their SHA-256 hash. This deduplicates content automatically (uploading the same file twice consumes no extra space).
- Metadata Layer: A lightweight database (SQLite in this example) tracks:
- Blobs: The physical files (hash size, location).
- Objects: The logical files (user-defined path hash, retention policy, timestamp).
- Retention & GC: A background process that identifies expired objects and orphaned blobs.
Python Implementation
You will need
tqdmfor progress bars (optional, but helpful) and standard libraries.import os import hashlib import sqlite3 import shutil import time from pathlib import Path from datetime import datetime, timedelta from typing import Optional, Tuple, List # Constants BLOCK_SIZE = 65536 # 64KB chunks for hashing large files DB_NAME = "metadata.db" class ArtifactStore: def __init__(self, root_dir: str): self.root_path = Path(root_dir).absolute() self.data_path = self.root_path / "blobs" self.tmp_path = self.root_path / "tmp" self.db_path = self.root_path / DB_NAME # Initialize directory structure self.root_path.mkdir(parents=True, exist_ok=True) self.data_path.mkdir(exist_ok=True) self.tmp_path.mkdir(exist_ok=True) self._init_db() def _get_db(self) -> sqlite3.Connection: """Returns a thread-local DB connection.""" conn = sqlite3.connect(str(self.db_path)) conn.row_factory = sqlite3.Row return conn def _init_db(self): """Initialize database schema.""" with self._get_db() as conn: # Physical storage tracking conn.execute(""" CREATE TABLE IF NOT EXISTS blobs ( hash TEXT PRIMARY KEY, size INTEGER NOT NULL, ref_count INTEGER DEFAULT 0, last_access INTEGER NOT NULL ) """) # Logical object tracking (The "Artifact") conn.execute(""" CREATE TABLE IF NOT EXISTS objects ( path TEXT PRIMARY KEY, hash TEXT NOT NULL, created_at INTEGER NOT NULL, expires_at INTEGER, FOREIGN KEY(hash) REFERENCES blobs(hash) ) """) conn.commit() # --- Core CAS Operations --- def _calculate_hash(self, file_path: Path) -> str: """Streaming SHA-256 hash calculation.""" sha256 = hashlib.sha256() with open(file_path, 'rb') as f: while chunk := f.read(BLOCK_SIZE): sha256.update(chunk) return sha256.hexdigest() def _blob_path(self, file_hash: str) -> Path: """Sharding strategy: store in subdirs to avoid FS limits: /blobs/ab/cdef...""" subdir = self.data_path / file_hash[:2] subdir.mkdir(exist_ok=True) return subdir / file_hash def ingest(self, source_path: str, dest_path: str, retention_days: Optional[int] = None): """ Ingests a file into the store. 1. Hash file. 2. Copy to blob storage if not exists (CAS). 3. Create/Update logical object mapping. """ source = Path(source_path) if not source.exists(): raise FileNotFoundError(f"Source file not found: {source_path}") # 1. Hash print(f"Hashing {source_path}...") file_hash = self._calculate_hash(source) file_size = source.stat().st_size target_blob = self._blob_path(file_hash) # 2. Physical Storage (CAS Check) with self._get_db() as conn: # Check if blob exists blob_row = conn.execute("SELECT * FROM blobs WHERE hash = ?", (file_hash,)).fetchone() if not blob_row: # Physical copy print(f"New blob {file_hash[:8]}... copying to store.") shutil.copy2(source, target_blob) # Insert blob record conn.execute( "INSERT INTO blobs (hash, size, ref_count, last_access) VALUES (?, ?, 0, ?)", (file_hash, file_size, int(time.time())) ) # 3. Logical Mapping # Calculate expiry expires_ts = None if retention_days: expires_ts = int(time.time()) + (retention_days * 86400) # Upsert object (update if exists, insert if new) # If path already exists pointing to a DIFFERENT hash, we must decrement old blob ref old_obj = conn.execute("SELECT hash FROM objects WHERE path = ?", (dest_path,)).fetchone() if old_obj and old_obj['hash'] != file_hash: self._decrement_ref(conn, old_obj['hash']) conn.execute(""" INSERT INTO objects (path, hash, created_at, expires_at) VALUES (?, ?, ?, ?) ON CONFLICT(path) DO UPDATE SET hash=excluded.hash, created_at=excluded.created_at, expires_at=excluded.expires_at """, (dest_path, file_hash, int(time.time()), expires_ts)) # Increment ref count for the new/current blob conn.execute("UPDATE blobs SET ref_count = ref_count + 1, last_access = ? WHERE hash = ?", (int(time.time()), file_hash)) conn.commit() print(f"Artifact '{dest_path}' stored successfully. Hash: {file_hash[:12]}...") def retrieve(self, dest_path: str) -> Tuple[Path, dict]: """Retrieves the physical path of an artifact. Returns file system path.""" with self._get_db() as conn: row = conn.execute("SELECT hash FROM objects WHERE path = ?", (dest_path,)).fetchone() if not row: raise KeyError(f"Artifact not found: {dest_path}") # Update access time conn.execute("UPDATE blobs SET last_access = ? WHERE hash = ?", (int(time.time()), row['hash'])) conn.commit() blob_path = self._blob_path(row['hash']) if not blob_path.exists(): raise FileNotFoundError(f"Data loss: Blob {row['hash']} missing for {dest_path}") return blob_path # --- Maintenance: Retention, GC, Compaction --- def _decrement_ref(self, conn: sqlite3.Connection, file_hash: str): """Decrements ref count, does NOT commit transaction.""" conn.execute("UPDATE blobs SET ref_count = ref_count - 1 WHERE hash = ?", (file_hash,)) def run_retention(self): """Deletes logical objects past their expiration date.""" print("Running Retention Policy...") now = int(time.time()) count = 0 with self._get_db() as conn: # Find expired objects expired = conn.execute("SELECT path, hash FROM objects WHERE expires_at IS NOT NULL AND expires_at < ?", (now,)).fetchall() for obj in expired: # Remove logical reference conn.execute("DELETE FROM objects WHERE path = ?", (obj['path'],)) # Decrement physical reference self._decrement_ref(conn, obj['hash']) count += 1 conn.commit() print(f"Retention complete. Removed {count} expired artifacts.") def run_gc(self): """ Garbage Collection. Deletes blobs where ref_count <= 0. """ print("Running Garbage Collection...") cleaned_size = 0 count = 0 with self._get_db() as conn: # Find orphans orphans = conn.execute("SELECT hash, size FROM blobs WHERE ref_count <= 0").fetchall() for blob in orphans: blob_path = self._blob_path(blob['hash']) try: if blob_path.exists(): blob_path.unlink() # Delete file cleaned_size += blob['size'] count += 1 # Remove metadata conn.execute("DELETE FROM blobs WHERE hash = ?", (blob['hash'],)) except OSError as e: print(f"Error deleting blob {blob['hash']}: {e}") conn.commit() print(f"GC complete. Reclaimed {count} blobs ({cleaned_size / (1024*1024):.2f} MB).") def run_compaction(self): """ Compaction. In this context, compaction implies checking for inconsistencies and optionally moving 'hot' data or defragging (conceptually). Here we implement 'Verification' and 'Directory Trimming'. """ print("Running Compaction & Verification...") # 1. DB Verification: Clean up objects pointing to missing blobs # (This handles data loss scenarios where files were manually deleted) with self._get_db() as conn: # This is a complex query in real SQL, keeping it simple here: # We iterate blobs, check if file exists. If not, delete blob record and dangling refs. all_blobs = conn.execute("SELECT hash FROM blobs").fetchall() for b in all_blobs: if not self._blob_path(b['hash']).exists(): print(f"Corruption detected: Missing blob {b['hash']}. Cleaning metadata.") conn.execute("DELETE FROM objects WHERE hash = ?", (b['hash'],)) conn.execute("DELETE FROM blobs WHERE hash = ?", (b['hash'],)) conn.commit() # 2. Directory Pruning: Remove empty hash subdirectories # (Compacting directory structure) for subdir in self.data_path.iterdir(): if subdir.is_dir() and not any(subdir.iterdir()): subdir.rmdir() print("Compaction complete.") def status(self): with self._get_db() as conn: blobs = conn.execute("SELECT COUNT(*), SUM(size) FROM blobs").fetchone() objs = conn.execute("SELECT COUNT(*) FROM objects").fetchone() orphans = conn.execute("SELECT COUNT(*) FROM blobs WHERE ref_count <= 0").fetchone() print("\n--- Store Status ---") print(f"Total Logical Objects: {objs[0]}") print(f"Total Physical Blobs: {blobs[0]}") print(f"Physical Size: {blobs[1] / (1024*1024):.2f} MB") print(f"Orphaned Blobs: {orphans[0]} (Run GC to clean)") print("--------------------\n") # --- Usage Example --- if __name__ == "__main__": # Setup store = ArtifactStore("./my_artifact_store") # Create a dummy large file dummy_file = "large_test_file.bin" with open(dummy_file, "wb") as f: f.write(os.urandom(1024 * 1024 * 10)) # 10MB random data # 1. Ingest with 1-day retention (TTL) print("\n[1] Ingesting file...") store.ingest(dummy_file, "backups/data_v1.bin", retention_days=1) # 2. Ingest same file again (Deduplication test) print("\n[2] Ingesting identical file with different name...") store.ingest(dummy_file, "archives/data_copy.bin", retention_days=1) store.status() # 3. Test Retrieval print("\n[3] Retrieving file...") path = store.retrieve("backups/data_v1.bin") print(f"File retrieved at: {path}") # 4. Test GC (Without expiration) print("\n[4] Testing GC (Nothing should be deleted)...") store.run_gc() # 5. Simulate Deletion (Manual delete of object) # In a real system, you might have a `store.delete(path)` method. # Here we manually manipulate DB to simulate a delete operation that leaves an orphan. print("\n[5] Simulating manual deletion of object...") with store._get_db() as conn: conn.execute("DELETE FROM objects WHERE path = 'backups/data_v1.bin'") conn.execute("UPDATE blobs SET ref_count = ref_count - 1 WHERE hash = (SELECT hash FROM blobs LIMIT 1)") conn.commit() store.status() # Should show 1 orphan # 6. Run GC (Should clean up) print("\n[6] Running GC (Should reclaim space)...") store.run_gc() store.status() # Cleanup local dummy file os.remove(dummy_file)How to design this for Production Scale
This code provides a functional local prototype. For a production system handling terabytes or petabytes, you would make the following architectural adjustments:
-
Storage Backend (The "CAS"):
- Local FS: As shown above, but use ZFS or Btrfs to handle compression and bit-rot detection natively.
- Cloud: Use S3 with Smart Tiering. The logic remains the same (key = hash), but
_blob_pathreturns an S3 URI.
-
The Database (The "Index"):
- SQLite works for single-node deployments.
- PostgreSQL or CockroachDB is required for distributed systems. You need strong transactionality (ACID) to ensure
ref_countnever drifts from reality. - Schema Optimization: Add columns for
mime_type,custom_metadata, andproject_idto allow filtering.
-
Garbage Collection (GC) Strategies:
- Reference Counting: Used in the example. Pro: Instant availability of space for reuse. Con: Can be slow if many references change at once.
- Mark-and-Sweep: Better for massive stores.
- Mark: Iterate all valid Objects and set a
visible_at = now()flag on the blobs. - Sweep: Delete any blob where
visible_at < threshold.
- Mark: Iterate all valid Objects and set a
- Tombstones: Never delete immediately. Move deleted blobs to a "Trash" folder/table. A separate lazy process cleans the trash after 24h, allowing for "Undo Delete" functionality.
-
Compaction:
- Deduplication Ratio: The code above deduplicates perfectly at the file level. For large files that change slightly (e.g., VM images or Docker containers), use Chunking (e.g., FastCDC). Split files into variable-sized blocks (avg 64KB - 4MB) and hash those individually. This allows storing a 1GB file that had 10MB of changes while only consuming 10MB of new space.
-
Retention Enforcement:
- Do not run
SELECT *on the database to find expired items. Use a Time-Travel architecture (like Snowflake or TiDB) if possible, or index theexpires_atcolumn heavily. - Use a Priority Queue for deletions to smooth out I/O load (don't delete 10TB instantly).
- Do not run
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.