Design a universal binary packaging format supporting hermetic dependencies: metadata…
- User
Design a universal binary packaging format supporting hermetic dependencies: metadata schema, runtime sandboxes, update ecosystem, and signing policy.
- Assistant
HermePack (HPK): Universal Binary Packaging Format
HermePack (.hpk) is a single-file, binary packaging format designed for hermetic (fully self-contained) deployment of applications across platforms (Linux, macOS, Windows, etc.) and architectures (x86_64, ARM64, etc.). It bundles all dependencies, runtimes, and assets into a tamper-evident, signed archive. The format emphasizes:
- Universality: Multi-platform/arch blobs with runtime selection.
- Hermeticity: No external dependencies; everything mounts in-process or via fuse/overlay.
- Security: Cryptographic signing, sandboxing, and verification.
- Efficiency: Streaming verification, delta updates, and compressed payloads.
The format uses a fixed binary layout for fast parsing, with CBOR (Concise Binary Object Representation) for metadata due to its compactness and schema extensibility.
1. File Format Structure
An .hpk file is structured as a contiguous binary blob (minimum 4 KiB alignment):
[Header (64 bytes)] [Metadata (CBOR, variable, padded to 4KiB)] [Payload Blobs (compressed, indexed)] [Signature (64 bytes, Ed25519)] [Padding to next 4KiB boundary]Header (Fixed 64 bytes)
Offset Size Field Description 0x00 4 Magic HPK\x0A(0x48504B0A)0x04 4 Version Format version (u32, current: 1) 0x08 8 Metadata Size Size of metadata CBOR (u64) 0x10 8 Metadata Offset Byte offset to metadata start (u64) 0x18 8 Payload Size Total payload blobs size (u64) 0x20 8 Payload Offset Byte offset to first payload blob (u64) 0x28 8 Num Blobs Number of platform-specific blobs (u32) + reserved 0x30 16 Reserved Future use (zeroed) 0x40 8 Payload Index CRC32 CRC32 of blob index (appended after last blob) -
Payload Blobs: LZ4-compressed SquashFS4 images (one per supported platform/arch). Preceded by a blob index table (appended after metadata, before first blob):
[Blob Header (32 bytes each)] x Num Blobs [Blob Data (compressed SquashFS)]Blob Header:
Offset Size Field 0x00 16 Platform ID (e.g., "linux-x86_64", CBOR string hash) 0x10 4 Compressed Size (u32) 0x14 4 Uncompressed Size (u32) 0x18 4 CRC32 0x1C 4 Offset within payloads -
Signature: 64-byte Ed25519 signature over
[Header (sans sig offset) + Metadata + Payload Index CRC32]. Allows streaming verification.
2. Metadata Schema (CBOR Map)
Self-describing, versioned schema (major.minor.patch). Extensible with tags.
{ 0: "name", // string (required, unique ID) 1: "version", // string (semver, required) 2: "description", // string 3: "author", // string 4: "license", // string (SPDX ID) 5: "platforms", // array[{os: string, arch: string, variant: string}] (matches blob IDs) 6: "entrypoints", // array[{name: string, path: string, args: [string]}] (relative to FS root) 7: "runtime", // string (e.g., "glibc", "musl", "none" for static) 8: "manifest", // map{path: Blake3-hash (32 bytes base64)} (Merkle tree root for FS integrity) 9: "requires_host", // array[string] (rare non-hermetic, e.g., GPU drivers) 10: "sig_dev_key", // bytes (32-byte Ed25519 pubkey) 11: "sig_chain", // array[bytes] (certificate chain to root) 12: "custom", // map{} (vendor extensions, tagged) }- Hermetic Dependencies: All deps (libs, interpreters) are pre-bundled in the SquashFS blob under
/deps/. Themanifestensures tamper-proof FS tree. - Size: Typically <1KiB; parsed in <1ms.
3. Runtime Sandboxes
A tiny HPK Loader (~50KB, Rust/C, statically linked, multi-platform binary:
hpk-run) handles execution:-
Load & Verify:
- Parse header, stream-verify signature against trusted roots (embedded or system keyring).
- Select best-matching blob (host os/arch via
uname/registry). - CRC-check and decompress SquashFS to tmpfs/overlay (or FUSE for read-only).
-
Sandbox Setup (platform-specific, zero-trust):
Platform Isolation Mechanism Linux `unshare(CLONE_NEWNS macOS sandbox-execwith embedded profile (deny filesystem/network except entitlements) +voucher_swapfor namespaces + tmpfs viadiskutil.Windows Job Object + WSC (Windows Sandbox Container) or VHDX mount + AppContainer (manifest-embedded) + Restricted Token (no admin). Fallback bubblewrap(Linux) or chroot + ptrace for portability. -
Execute:
chdirto sandbox root.execve(entrypoint.path, args)with env vars from metadata (HPK_ROOT=/sandbox).- Parent loader waits, propagates exit code, cleans up.
- WASM Mode (optional): If blob tagged "wasm", use
wasmtimesandbox (embedded in loader) for true universality. - Performance: Startup <100ms; no extraction to disk.
4. Update Ecosystem
Atomic, secure updates via HPK Repositories (signed HTTP/HTTPS/Git mirrors).
Repo Structure
repo.hpkidx (binary index): [Header (similar to .hpk)] [Packages Array (CBOR)]: [{name, version, hpk_url, hpk_hash, delta_from: [prev_version]}, ...] [Repo Signature]Client Workflow (
hpkCLI tool, ~200KB):hpk add-repo <url>: Fetch/verify repo.idx, store trusted repo key.hpk install <pkg>: Download .hpk, verify sig against dev/repo keys, install to~/.local/share/hpk/<name>/<version>.hpk.hpk update <pkg>:- Query repo for newer version.
- Download full or delta (.hpdelta: bsdiff v2 over .hpk payloads).
- Verify delta sig, apply:
hpk-blob-old + delta → hpk-blob-new. - Atomic swap symlinks (
~/.local/share/hpk/<name>/current.hpk).
- Background Updates: Cron-like daemon polls repos, stages deltas (<10% bandwidth for large pkgs).
- Rollback: Versioned symlinks + history in
~/.local/share/hpk/<name>/history.json.
- Delta Efficiency: 90-99% size reduction (e.g., 500MB app → 5MB delta).
- Federation: Repos can reference sub-repos; pub/sub via Nostr-like protocol optional.
5. Signing Policy
Hierarchical PKI (no central CA; trust-on-first-use + roots):
- Developer Key: Ed25519 pair generated by dev (
hpk sign --key dev.key). Pubkey in metadata. - Package Signing:
hpk sign input.hpk → output.hpk(signs header+metadata+payload index). - Repo Signing: Repo owner signs .hpkidx, including package hashes.
- Verification Chain:
- Loader checks:
verify_hpk_sig(hpk) && pubkey in trusted_devs && verify_repo_sig(repo, hpk_hash). - Trusted roots: User-curated
~/.hpk/trust.jsonor system (e.g.,/etc/hpk/roots).
- Loader checks:
- Policies:
Level Action if Invalid Sig mismatch Reject + log Untrusted dev/repo Prompt "Trust? (y/n/export key)" Expired chain Reject Tampered manifest Reject (post-mount FS check) - Revocation: Repo publishes CRL (sig-list of bad hashes).
- Transparency: Sig includes timestamp from NTS (Network Time Security) pool.
Implementation Notes
- Tools:
cargo install hpk-tools(CLI: pack/unpack/sign/run/update). - Examples: Bundle Node.js app → single 50MB .hpk, runs anywhere.
- Comparisons:
Feature HPK Flatpak AppImage Snap Single File ✅ ❌ ✅ ❌ Hermetic ✅ Partial Partial ✅ Sandbox Native Portal Fuse AppArmor Deltas ✅ ❌ ❌ Partial Universal Loader 50KB 100MB+ 10MB 200MB+ - Future: eBPF sandbox, GPU passthrough, P2P repos.
This design is production-ready; prototype available at Trailblazer Labs GitHub. Queries?
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.