Cache replicas
Development preview
This page describes a pinned development revision, not a published release. Its package version may still identify the preceding release while the next version is being prepared.
The cache can remember an output endorsement after evicting its NAR, then find that output on another server. Storage location does not grant build authority: the resolver preserves original Nix signatures, and the consumer still checks those signatures and the full NAR hash. This is independent of the CI coordinator. See provider trust for identity pins, recursive delegation, withdrawal and the 24-hour relay-outage policy.
Provider behavior
Enabling the managed cache's public listener also enables historical metadata. Only locally endorsed outputs enter that provider's history. Retaining an imported peer-signed output does not create a new local endorsement. Existing local outputs are indexed incrementally, in batches of 128, without delaying worker readiness. History failures or capacity limits do not fail the private job handoff.
Live /nix/<store-hash>.narinfo and /nix/<store-hash>.json describe retained, currently authorized outputs. The JSON form is a local diagnostic representation. Both disappear when the record is removed. Hashtree nodes have their own retention; their links record publication, not availability. Content-addressed narinfo is removed alongside archive availability. Evicting a NAR does not remove that endorsement. Neither kind of metadata proves an independent build or reproducibility.
Provider-owned NARs and immutable metadata use the existing hash-addressed GET/HEAD/range surface at /<sha256> (optional extension allowed). No NAR is copied into another local archive store, and nothing is uploaded to an external Blossom server. The public endpoint exposes no private handoff, resolver capability or publication operation.
Managed workers expose a preparation-scoped local-first gateway. Guests use it as their only substituter: a local hit makes no external cache HTTP requests, including nix-cache-info. Only a confirmed local miss invokes the historical replica resolver and ordinary operator-configured upstream caches concurrently. The first successful metadata response wins; ordinary upstream metadata must have a signature authorized by the preparation's pinned trust keys. Nix still checks the downloaded archive and output hashes. A slow provider therefore does not delay a hit from cache.nixos.org. All configured ordinary caches are eligible immediately, including caches after the fourth entry. Each has a reserved share of 64 metadata request slots across the worker (rounded down equally), so a slow cache cannot consume a fast cache's share. With the default single upstream, cache.nixos.org can use all 64 slots. At most 256 lookups are admitted; admitted requests wait FIFO for their cache's slots. Each HTTP exchange gets five seconds after acquiring a slot, within a 30-second total lookup budget. The first trusted hit cancels losing metadata lookups and releases their slots.
Ordinary archive downloads have a separate pool of 16 streams with 128 waiting slots and a 120-second queue deadline. A stream holds its permit until its body is consumed or dropped. This absorbs parallel Nix substitution bursts without requiring the client to retry immediate capacity errors. Slow peer transfers use a different pool and cannot occupy ordinary upstream slots. Ordinary archive URLs select one cache; their bytes are not raced against a different encoding. Local IO failures remain errors rather than triggering remote lookup. Runtime uses the same read capability, without publication authority.
The NixOS module supplies cache.nixos.org and cacheSubstituters to the worker. Manually configured workers use managed.upstream_caches (default: ["https://cache.nixos.org"]). Ordinary upstreams must serve archive URLs under the configured cache's nar/ path, on the same origin; redirects and credentials are rejected. Explicit operator URLs may use private networks. These are distinct from discovered replica URLs, which remain restricted to public HTTPS addresses. Ordinary archive bytes retain their compression, hashes and signatures. The guest's Nix client verifies them; the gateway never signs, decompresses or retains them. All offerings sharing the cache must configure the same ordinary upstreams.
No workflow or flake changes are required. Its resolver URL is scoped to the admitted handoff's read capability and trust snapshot. Later withdrawal excludes new preparations; already admitted preparation leases retain their policy until they finish or expire. Runtime jobs receive no publication authority.
Remote bytes stream through the resolver without archive staging or unpacking on the host. Preparation may retain them through the ordinary cache publication and disk-admission policy, preserving upstream signatures. The resolver itself retains only metadata and lookup results. Add optional storage-only origins with:
nix
services.ngit-ci-nix-provider.managedCache.replicaUrls = [
"https://replica.example.org"
];Approved provider origins are included automatically. These additional origins grant no signing trust. They must serve hash-addressed downloads; metadata is accepted only through a verified hash/catalog and an authorized Nix signature. All offerings sharing a cache use the same replica and trust configuration.
Wire format
See the normative cache protocol. The provider publishes one Narwal-compatible root event, alongside the independent trust-list event. The root references a MessagePack Hashtree containing standard narinfo links. One selected realization is retained per store hash. A newer admission replaces the previous selection without invalidating existing private job handoffs.
Lookup and failure behavior
On a local miss, the resolver races live signed narinfo and historical catalog lookups across approved storage origins. Hashtree nodes are fetched on demand, verified by hash and cached with bounded memory. Narinfo blobs are fetched live; a hit prefers that storage server for the archive, without a preliminary HEAD. Readers support arbitrary nested Narwal directories; our prefix layout makes likely paths searchable first. Traversal and HTTP work have bounded deadlines.
Every candidate must carry an authorized Nix signature. First successful metadata wins; different publishers' selections are not automatically a conflict. Exact archive locators remain bound to the compressed FileHash, including after gateway restart. The gateway preserves compression and original signatures, checks the streamed FileHash/size, and leaves uncompressed NAR verification to Nix.
Archive GET prefers the metadata source. If its first bytes have not arrived after 250 ms, the resolver probes a second replica of the same compressed FileHash. It probes at most two at once, immediately replaces failed probes, and selects the first acceptable response with bytes. The primary request keeps its place in a serving cache's queue for up to 120 seconds, rather than cancelling and repeatedly rejoining. Alternative probes get five seconds each; total selection is bounded to 120 seconds. A fast source therefore needs only one download. Losing probes are cancelled; no loser bytes reach Nix. The first chunk is buffered only until selection, then participates in the same size/hash verification as the rest of the stream. A fast first chunk is not a bandwidth guarantee: truncated, stalled, oversized or corrupt streams abort. Once bytes are sent, a transfer cannot switch replicas or encodings in place. Preparation enables Nix fallback; no metadata is an ordinary cache miss. A cached narinfo can outlive its archive, so clients must handle unavailable downloads even though our own storage pairs their eviction. Lookup hits last 15 seconds, completed misses 30 seconds; changes to the admitted catalog or trust snapshot produce different lookup keys. Live peer failures are best-effort misses, but are not memoized as definitive absence; a recovered live origin can serve the next request. Directory traversal is demand-driven and shared by immutable catalog root. Its cursor and filename index survive request cancellation and directory-node eviction, so repeated or concurrent misses do not restart whole-tree scans. Expected radix children are still searched first; arbitrary Narwal directory names remain supported. An incomplete traversal never proves absence. Index entries locate narinfo; each use still downloads and verifies the narinfo against the current publisher's authorized signing keys.
Bounds and retention
The public cache reserves separate budgets for 16 archive transfers and 64 metadata/HEAD requests. Bulk downloads cannot occupy the metadata budget; the constant nix-cache-info response does not consume either budget. Metadata has 192 waiting slots and a 30-second queue deadline; archives have 128 waiting slots and a 120-second queue deadline. Archive waiters hold an open inode but neither the cache-state lock nor a metadata permit. Public archive trust is rechecked after waiting. Full queues or exhausted deadlines return 503 with Retry-After, rather than accumulating an unbounded queue or reporting a false cache miss. The public pre-header request deadline is 150 seconds.
- History: at most 131,072 selected filenames and 128 MiB of current directory nodes. Leaves split at 128 entries using stable store-hash prefixes. Current and two publication roots retain their directory nodes, never their narinfo or NAR blobs. Updates reserve 128 MiB above the filesystem floor; pending publication is bounded to 16 MiB. Capacity exhaustion defers publication.
- Peer lookup: 32 authorized publishers and eight extra storage origins; 64 concurrent lookups and 192 waiting slots. The 30-second lookup budget includes waiting. Live and historical candidates are polled concurrently, with 64 actual metadata requests across the resolver and at most eight per origin. Candidates share one immutable origin inventory; historical attempts borrow it rather than duplicating URL strings per future. Origin admission happens before global admission, so requests waiting on one origin do not occupy the entire network pool. Origin bookkeeping is bounded to 512 simultaneously referenced entries. Each HTTP exchange gets five seconds after acquiring network capacity. Ordinary upstreams have independent budgets.
- At most four catalog indexes are retained, each with at most 131,072 fixed-size filename/digest/size entries and 4,096 discovered directory hashes, depth 32. Inactive roots are evicted least-recently-used; active roots are never evicted. Index capacity exhaustion is transient, not cached absence. A new immutable root gets a separate traversal; incomplete progress is retained in memory only.
- Immutable metadata, lookup results and archive locators each have a 16 MiB encoded-data budget, plus bounded bookkeeping. Result and locator counts are also capped at 1,024 and 2,048. Sixteen peer archive streams are allowed, with 128 waiting slots and a 120-second queue deadline. Hedging temporarily permits up to 32 first-byte probes; only 16 selected streams persist. Header/selection handling has a 270-second outer deadline and each selected stream a one-hour deadline, with a 30-second idle-read limit. Dropping a consumer cancels a stalled stream and releases its permit promptly; streamed bytes are not retained by the resolver.
- Upstream URLs must be credential-free HTTPS, at most 2,048 characters, with no query or fragment. Redirects and environment proxies are disabled. DNS results must be public addresses; private/special literal IPs are rejected too.
Current historical metadata remains on the provider's disk; this implementation has no external metadata/NAR upload or automatic bulk replication. Remote immutable metadata is an in-memory cache. A future storage service can retain those hash-addressed objects independently without acquiring signing authority. Catalog version high-water marks are durable and bounded to 32 publisher identities, like the trust graph's bounded history.
Saturation and latency
Waiting absorbs bursts; it cannot make an unavailable server or sustained load above capacity succeed indefinitely. When an ordinary upstream returns a normal metadata miss, an optional peer's 502/503 is treated as a request-local miss with Cache-Control: no-store, so Nix can build that path without disabling ordinary substitution for other dependencies. Resolver-routed guests set narinfo-cache-negative-ttl = 0 to prevent Nix retaining that temporary absence. A peer hit still wins over an ordinary miss or error. Failures without a healthy ordinary miss, and archive download failures, remain errors. Gateway 502/503 responses carry Retry-After: 2 and Cache-Control: no-store, including resolver-generated errors. Correctness does not rely on the client honoring Retry-After. Normal fast hits do not wait for slower candidates; a miss may require the bounded search to finish. Long transfers can delay queued archives, so provider allocation admission must reflect the shared worker's capacity. Increase job concurrency only with measured headroom.
The capability and its pinned trust policy are checked before lookup. After waiting, a finished or expired handoff is checked again before returning headers; it receives no queued response bytes. Already streaming responses retain the existing admission-time authorization semantics. Metadata races, disconnected consumers and cancelled waiters release their capacity rather than leaving background requests occupying slots. Resolver status exposes active/waiting peer lookups and ordinary metadata admission/activity and archive active/waiting counts. These are instantaneous diagnostics, not bandwidth estimates.
Standalone gateway
The same resolver is available without a coordinator, builder, Nix daemon or signing credential:
sh
ngit-ci-cache serve --config resolver.json
# Also: NGIT_CI_CACHE_RESOLVER_CONFIG, including .env.json
{
"listen": "127.0.0.1:17849",
"state_dir": "/var/lib/nix-replica-resolver",
"trust_lock": "/etc/nix/cache-trust.json",
"relays": ["wss://relay.example.org"],
"replica_urls": ["https://replica.example.org"]
}Install the approved effective Nix keys in the consuming daemon using the trust tooling, then use http://127.0.0.1:17849/nix as a substituter. The gateway cannot grant daemon trust. A remote gateway should sit behind TLS and request limits. Signed identity, trust and catalog announcements arrive through persistent relay subscriptions shared by the worker and standalone gateway. New catalog heads are persisted immediately; their immutable HTTP metadata is downloaded only when a local miss needs it. Reconnects replay stored heads, and a five-minute subscription renewal reconciles missed updates and refreshes the existing 24-hour trust window. Signed rollback/equivocation checks still apply; direct pins remain operator-owned. Restart after changing direct roots or service configuration.
The configured one to eight relays bootstrap discovery. Signed NIP-65 relay lists can add up to two write/unmarked WSS relays per watched provider, with eight extra relays overall. URLs with credentials are rejected and DNS must resolve to public addresses before admission. This precheck is not DNS pinning. Recursive watch expansion retains the existing 32-provider/eight-edge bounds; removal closes unneeded subscriptions. No provider signing key is needed to subscribe.
GET /status exposes request/cache-hit counters and metadata-index counts. The provider's private listener exposes equivalent diagnostics at /build-cache/resolver-status, including ordinary-upstream HTTP request counters; neither endpoint exposes capabilities or keys. ngit-ci-nar-cache::{catalog, replica} is the reusable library surface. Its explicit HTTP-client injection is for embedders choosing their own transport policy; the supplied gateway uses the restricted default client.
The rationale and future independent-build evidence are in the design record. Matching signatures or hashes must not be presented as proof that multiple providers independently rebuilt an output.