How large is a cache.nixos.org mirror?
- User
How large is a cache.nixos.org mirror?
- Assistant
Based on public discussions, reports from the NixOS Foundation, and community estimates (from sources like Discourse, Reddit, and NixCon talks), I'll provide an answer below. Note that cache.nixos.org isn't something most people mirror in full (due to its size and the fact that it's dynamically updated), but it's theoretically possible using tools like
aws s3 sync(since it's hosted on Amazon S3) or custom scripts. If you're asking for practical reasons (e.g., setting up a local proxy or partial mirror), I'll address that too.Estimated Size of a Full Mirror
- Current Estimate (as of 2024): Approximately 20-40 terabytes (TB) of data.
- This includes millions of objects:
.narinfometadata files,.nar(Nix Archive) binary files, signatures, and other artifacts. - The exact size fluctuates because:
- New builds are constantly added by Hydra (the Nix build farm) for packages across multiple channels (e.g., nixos-unstable, nixos-23.11), architectures (x86_64, aarch64, etc.), and versions.
- Old or unreferenced builds are periodically garbage-collected to manage costs and storage, based on retention policies (e.g., keeping data for supported releases and some historical versions).
- Historical context:
- In 2020-2021, estimates were around 5-10 TB.
- By 2022-2023, it grew to 15-25 TB due to more packages (nixpkgs has ~80,000+), larger software (e.g., machine learning tools, games), and support for more platforms.
- As of late 2023/early 2024, community reports and foundation updates suggest it's closer to 30 TB or more, with over 25 million objects.
- Bandwidth usage gives a clue: The cache serves hundreds of terabytes to petabytes of data per month globally, but that's download volume, not storage size.
- This includes millions of objects:
These numbers come from:
- NixOS Foundation transparency reports (e.g., S3 storage costs are a significant part of their budget, often $10k+ monthly, which aligns with 20-40 TB at standard S3 pricing).
- Discussions on the NixOS Discourse forum and Reddit (r/NixOS).
- Tools like
nixos-org-configurationsrepo or scripts people have shared for partial syncing.
The size is not fixed—it's a living cache. If you try to mirror it, you'd need to sync periodically to keep up with updates.
Challenges of Mirroring the Full Cache
- Storage Requirements: You'd need a massive amount of disk space (e.g., a RAID array or cloud storage). Compressing locally could reduce it slightly, but .nar files are already somewhat efficient.
- Time and Bandwidth: Syncing the full thing could take days or weeks over a standard connection, and it'd cost a lot in egress fees if you're pulling from S3.
- Practicality: Almost no one mirrors the entire cache. Instead:
- Use a caching proxy like
nix-serve,cachix, or a local Squid/NGINX setup to cache only what you download. - For offline use, tools like
nix-copy-closureor pre-fetching specific package closures (e.g., vianix-store --export) are more common, resulting in mirrors of just a few GB to TB depending on your needs. - Organizations (e.g., companies using Nix internally) often mirror subsets for specific channels or packages, which might be 1-5 TB.
- Use a caching proxy like
How to Check or Set Up a Mirror
- Check Current Stats: You can't directly query the full size (it's not public), but you can estimate by browsing subsets via
curl https://cache.nixos.org/or using tools likenix narinfofor specific paths. For costs/size hints, see the NixOS Foundation's donation reports. - Partial Mirror Setup:
- Use Cachix (a hosted service) for shared caches.
- Script a mirror with AWS CLI:
aws s3 sync s3://nixos-cache/ /local/path/(but you'll need to handle authentication and costs). - For a local binary cache: Configure your
/etc/nix/nix.confwithsubstituters = https://your-mirror.example.com/and use something likenix-serve.
- Alternatives: If you just want faster downloads, use a CDN mirror or community caches (e.g., via
cachix.org). For airgapped environments, look intonix flakeswith pre-built stores.
If this doesn't match what you meant by "mirror" (e.g., if you're asking about cache size on disk for a running system, which is much smaller—typically GB for a user's
/nix/store), provide more details! If you're planning to set one up, I recommend starting small and monitoring usage.Sources: NixOS Discourse, Reddit r/NixOS, NixOS Foundation reports (e.g., on nixos.org). These estimates could change, so check recent discussions for updates.
- Current Estimate (as of 2024): Approximately 20-40 terabytes (TB) of data.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.