Design a content-addressed artifact store: dedup, integrity, and retention.
- User
Design a content-addressed artifact store: dedup, integrity, and retention.
Assistant
The user wants me to design a "content-addressed artifact store." The key requirements are:
- Content-addressed: This implies using hashes (like SHA-256) as the primary identifiers for data. This naturally leads to deduplication and integrity.
- Deduplication: Storing unique chunks of data only once.
- Integrity: Ensuring data hasn't been tampered with or corrupted.
- Retention: Policies for keeping or deleting data (e.g., garbage collection, legal holds, TTL).
This is a system design problem. I need to structure the answer logically, covering the core concepts, architecture, components, and workflows.
1. High-Level Concept:
- What is it? A storage system where the key is the hash of the content.
- Why? Deduplication is automatic (same content = same hash). Integrity is built-in (hash mismatch = corruption).
2. Core Components:
- The Hashing Algorithm: Needs to be secure and fast enough. SHA-256 is the standard. BLAKE3 is a faster alternative.
- Chunking Strategy: How do we break files down?
- Fixed-size: Simple, but insertion/deletion shifts everything (the "dedup" problem).
- Content-Defined Chunking (CDC): Rabin fingerprinting or similar. Finds boundaries based on content. Better for shifting data.
- Merkle Trees: Hierarchical structure. Good for verifying sub-trees.
- Storage Layer (CAS - Content Addressable Storage):
- Blob Store: Where the raw data lives. Could be S3, local disk, etc.
- Metadata Store: Mapping hashes to locations, sizes, timestamps, and reference counts.
- Index/Manifest Layer: How do we reconstruct a "file" or "artifact" from a list of hashes? This is usually a manifest file (JSON/XML/Protobuf) which itself is stored in the CAS.
3. Detailed Feature Breakdown:
- Deduplication:
- Process: Client chunks data -> Hashes chunks -> Checks if hash exists -> Uploads only missing ones.
- Granularity: File-level vs. Block-level vs. Byte-level. Block-level is the sweet spot for artifact stores.
- Integrity:
- Write Path: Hash is calculated client-side and verified server-side.
- Read Path: Re-calculate hash on read.
- Storage: "Write-once" semantics. You never update a hash; you only add new ones. If a hash exists, you don't overwrite.
- Retention (The tricky part):
- Since chunks are shared, you can't just delete a chunk when an artifact is deleted (other artifacts might use it).
- Reference Counting: Every chunk has a count of how many artifacts reference it.
- Garbage Collection (GC):
- Mark-and-Sweep: Traverse live manifests, mark chunks. Delete unmarked.
- Reference Counting: Decrement on artifact deletion. Delete when 0. (Risk: distributed race conditions).
- Policies: TTL (Time to Live), Legal Hold.
4. Architecture Diagram (Mental Draft):
- Client -> API Gateway -> Upload Service -> Chunker -> Storage Backend (S3/Posix) -> Metadata DB (Cassandra/Postgres/DynamoDB)
5. Refinement & Trade-offs:
- Encryption: Should it be convergent encryption? (Deterministic encryption allows dedup of encrypted data but has security trade-offs).
- Performance: CAS can be chatty. High IOPS.
- Global vs. Local: Is this a git repo or a cloud storage bucket?
6. Structuring the Output: I will organize this as a technical design document.
- Introduction: Define CAS.
- Core Architecture:
- Ingestion (Chunking, Hashing).
- Storage (Blob, Metadata).
- Implementing Deduplication: Chunking strategies (CDC vs Fixed).
- Ensuring Integrity: Merkle Trees, Verification.
- Managing Retention: The DAG model, Reference Counting, Garbage Collection.
- Data Structures: Manifests.
- API Design: Basic endpoints.
Self-Correction/Review during drafting:
- Initial thought: Just use simple reference counting.
- Correction: Reference counting is dangerous in distributed systems if an increment/decrement is lost. A periodic "Mark and Sweep" GC is safer for long-term retention, though ref counts are good for speed. I should mention both.
- Retention: Need to mention that retention applies to Artifacts (the logical file), not just raw chunks. Deleting an artifact triggers the GC process for chunks.
Key Technologies to mention:
- SHA-256 / BLAKE3
- Rabin Fingerprinting (for CDC)
- Merkle DAGs (like IPFS/Git)
- S3 / MinIO
- Cassandra / ScyllaDB (for metadata scale)
Let's build the response.
- Step 1: API/Interface. Users interact with "Files/Artifacts", not "Hashes".
- Step 2: Write Path. File -> Chunk -> Hash -> Check Existence -> Write.
- Step 3: Read Path. Manifest -> List of Hashes -> Fetch Chunks -> Reassemble.
- Step 4: Retention. Reachability graph.
This looks solid. I will present this as a system design specification.
Plan:
- Executive Summary: What is it and why do we care?
- Core Data Structures: The Merkle DAG.
- Ingestion Pipeline (Dedup): Chunking strategies.
- Storage Layer: Separation of metadata and data.
- Integrity Mechanisms: Verification steps.
- Retention & Garbage Collection: The lifecycle logic.
- Security Considerations: Encryption and dedup.
Let's write it out. This design outlines a robust Content-Addressable Storage (CAS) system. In this architecture, data is not stored by a filename (location-addressed) but by a cryptographic hash of its content (content-addressed).
1. Core Concepts & Data Model
The system relies on a Merkle DAG (Directed Acyclic Graph) structure, similar to Git or IPFS.
The Addressing Scheme
- Algorithm: SHA-256 (or BLAKE3 for performance).
- Identifier: The
Hashserves as the unique ID. - Immutability: Once written, an object with a specific hash cannot be modified. To change data, you write a new object with a new hash.
Data Types
- Blobs: Raw binary data (the file chunks).
- Trees (Nodes): Lists of hashes pointing to Blobs or other Trees. This allows for directory structures.
- Manifests: The user-facing entry point. It contains the Root Tree hash, metadata (tags, timestamps), and cryptographic signatures.
2. Ingestion Pipeline (Write Path)
The goal here is to maximize deduplication and throughput.
Step 1: Chunking (The Dedup Engine)
We do not hash whole files; we chunk them. If a single byte changes in a 1GB file, we only want to store the changed chunk, not the whole file.
- Strategy: Content-Defined Chunking (CDC).
- Instead of splitting every 4MB (fixed size), we analyze the byte stream to find "natural" boundaries (e.g., using Rabin fingerprinting).
- Why? If you insert text at the start of a file, fixed-size chunking shifts all boundaries, changing all hashes. CDC keeps subsequent boundaries the same, ensuring maximum deduplication for shifted data.
- Process: File Stream Chunker
Chunk[].
Step 2: Hashing & Dedup Check
- Calculate
Hash = SHA-256(Chunk). - Existence Check: Query the Metadata Store:
Exists?(Hash).- Yes: Discard chunk (it is already stored).
- No: Keep chunk for writing.
Step 3: Storage
- Write the unique chunks to the Blob Store.
- Construct the Tree objects linking these chunks.
- Construct the Manifest pointing to the Root Tree.
3. Storage Architecture
We separate "what we have" (Metadata) from "the data itself" (Blob Storage).
A. The Blob Store (The Data Plane)
This is optimized for dense storage and high throughput.
- Layout: Use a directory structure based on the hash prefix to avoid directory limits (e.g.,
/data/ab/cdef...). - Tiering:
- Hot: NVMe SSDs (for recent builds/artifacts).
- Cold: S3/Glacier (for long-term retention).
- Compression: Apply Zstd compression to blobs before writing (dedup works better on uncompressed data, but storage is cheaper compressed).
B. The Metadata Store (The Control Plane)
This is optimized for lookups and graph traversal. A Key-Value store (RocksDB, Cassandra, DynamoDB) is ideal.
- Key:
Hash - Value:
Size,Type(Blob/Tree),References,CreationDate.
C. The Reference Tracker (The Retention Engine)
This maps logical names to physical hashes.
- Table:
Artifacts - Columns:
Name(e.g.,project-alpha/build-101),RootHash,RetentionPolicy,ExpireAt,ReferenceCount.
4. Integrity Verification
Integrity is native to CAS, but requires implementation rigor.
The "Trust On First Use" (TOFU) Model
- Write Path: The client calculates the hash. The server calculates the hash on receipt. If they mismatch, the write is rejected.
- Read Path: When a client requests a hash, they can re-calculate the hash of the delivered stream. It must match the requested ID.
Self-Healing (Scrubbing)
A background process (The "Janitor") performs random sampling or linear scans:
- Read a Blob.
- Hash it.
- Does
CalculatedHash == Filename?- Yes: Update
LastVerifiedTimestamp. - No: Data Corruption Detected.
- If Replication is enabled: Restore from replica.
- If no replica: Delete entry and mark as lost (reduce reference count).
- Yes: Update
5. Retention & Garbage Collection (GC)
You cannot simply delete an artifact because its chunks might be shared by another artifact. We need an Indirect Reference Counting system.
Reference Counting
Every Chunk stores a
ref_count.- Ingest: If Chunk is new,
ref_count = 1. If existing,ref_count++. - Delete Manifest:
ref_count--for all chunks in that tree.
Garbage Collection Strategies
Do not delete immediately. Use a "Mark and Sweep" or "Tombstone" approach.
- Tombstones: When a user deletes an artifact, we do not touch the blobs. We simply mark the Manifest as
Deleted = truein the Metadata store. - Sweep Phase (GC):
- Runs during low-traffic windows.
- Traverses the Metadata Store.
- Identifies Chunks where
ref_count == 0(Orphaned). - Safety Check: If
ref_count == 0butCreationDate < 24 hours ago, keep it (grace period for partial uploads). - Deletes the Blob from the Blob Store.
Retention Policies
- TTL (Time To Live): Artifacts automatically move to "Tombstone" status after
ExpireAt. - Legal Hold: Overrides
ExpireAt. Theref_countis artificially locked/incremented so GC ignores it.
6. API Design Summary
Method Endpoint Description PUT /artifacts/{name}Upload file. Headers: X-Retention: 30d. ReturnsManifestID.GET /artifacts/{name}Download file. Server reassembles chunks. HEAD /artifacts/{name}Check existence/integrity (returns hash). MOUNT /cas/{hash}Direct raw access to a specific chunk (for caching/proxies). GC /admin/gcTrigger garbage collection manually. 7. Scalability & Security Considerations
Deduplication vs. Encryption
- Problem: Standard encryption (AES-GCM) randomizes data, killing deduplication (identical files produce different ciphertext).
- Solution: Convergent Encryption.
- Key =
SHA-256(File Content). - Encrypt(Content, Key).
- This ensures identical content produces identical ciphertext, allowing deduplication even on encrypted data. Note: This is susceptible to "Confirmation of File" attacks.
- Key =
Distributed Caching
- Place a CDN or caching proxy (Varnish/Nginx) in front of the
GETendpoints. - Since content is immutable, you can cache aggressively (Cache-Control:
public, max-age=31536000, immutable) based solely on the URL (which contains the hash).
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.