Skip to content

Cache sharing ​

Development preview

This page describes a pinned development revision, not a published release. Its package version may still identify the preceding release while the next version is being prepared.

Select storage and discovery endpoints using the service considerations, including Narwal event-kind acceptance and rate limits.

A provider can expose its controlled-preparation cache for ordinary Nix clients. Publication is optional. Consumers explicitly approve the provider's Nostr identity, using a signed announcement to bootstrap its current keys and catalog. An announcement is evidence of publisher identity, not proof of an independent rebuild.

The reusable ngit-ci-nar-cache library owns archive validation, Nix metadata, announcement verification and trust locks. ngit-ci-cache is a standalone tool in the Nix provider package; it needs no coordinator, workflow or repository to discover, pin or publish a cache. Native Nix supports multiple provider-specific substituters; the optional replica resolver adds demand lookup across their stored copies. Existing public C1 workflow and job events do not change.

Storage and publication ​

The cache worker keeps one archive per uncompressed NAR content hash. New uploads are validated and compressed with single-threaded zstd level 1 in one streaming pass, with bounded buffering and one ingress encoder at a time. No uncompressed staging copy is written. The worker inherits its service CPU priority; compression does not wait for an idle-CPU maintenance window. Existing raw archives remain readable until ordinary eviction. Cache size and admission accounting use unique stored archive bytes, rather than NarSize.

Compression preserves the signed uncompressed NAR identity. Narinfo advertises Compression: zstd and the compressed archive's FileHash and FileSize; NarHash, NarSize, references and original signatures remain unchanged. Catalog narinfo identifies the compressed archive by its own hash. Historical and live lookups both transfer the advertised representation. Raw-hash downloads remain available through bounded streaming decompression for existing consumers.

The persistent index now records archive encoding. An older binary cannot read that index; rollback requires a compatible binary or restoration of the pre-upgrade index after draining jobs. Never restore an index over a running worker or reinterpret compressed files as raw NARs.

Its optional public listener serves only persistent admitted records; temporary handoffs and upload capabilities are inaccessible. The private provider IPC still authorizes preparation ingress and seals it after the preparation VM exits. Runtime jobs receive no publication authority. See local preparation for that trust boundary.

The public listener serves standard /nix/nix-cache-info and /nix/<store-hash>.narinfo. Archive URLs resolve to /<sha256>, with optional extensions. Hash-addressed GET/HEAD, byte ranges, conditional reads and CORS form the read surface of Blossom BUD-01. This is a dedicated read-only cache service, not a general BUD-02 upload server. Its archive and eviction lifecycle is independent of repository blobs retained by ngit-grasp/ditto-blossom. No archive is copied into the latter service.

Public reads are bounded to sixteen concurrent file streams; metadata processing has a thirty-second deadline; archive streams have a one-hour deadline. Configure TLS, connection limits and timeouts at the reverse proxy. The private listener must never be exposed. Open file readers survive unlinking; admission and eviction do not wait for a slow public reader. The published Hashtree catalog records selections, not a live inventory. Live narinfo and archive requests use the same retained-record index; eviction removes availability from both routes. A narinfo read refreshes recent-use protection, and an open archive stream can finish after unlinking. A cache miss remains a normal Nix fallback to another approved cache or a build. Historical endorsements and replica lookup are separate from this live availability index; historical metadata never pins NARs.

Publication preserves valid signatures from configured upstream keys. A consumer must trust the actual signing key in a returned narinfo, including when that output is served by another provider. Merely trusting a server's Nostr identity is insufficient. Locally built unsigned outputs acquire the local cache signature only through controlled preparation. Imported outputs with unrecognized signatures are not promoted; their private job handoff remains available. Existing local endorsements from before enabling peer sharing remain local endorsements.

Retention under pressure ​

Items remain until size or free-space pressure requires eviction. Active handoffs and records read within five minutes are protected. Eviction groups aliases of the same NAR so it only counts space that can actually be reclaimed; the strongest reference determines the group's value.

The worker keeps a separate durable relevance.json index of preparation consumers. A consumer is scoped to coordinator-authorized logical repository, PR root or pushed branch, workflow path, job and Nix system. Offering identity and commit are excluded, so a new revision or a different machine size updates the same consumer. References point to NAR content hashes; multiple repositories share bytes and the strongest current reference wins, without multiplying value by the number of consumers.

Only successful, publication-enabled controlled preparation replaces a consumer's reference set. Sealing a partial handoff after failure does not replace it. A newer observed automatic allocation fences older completions, and repeated completion is idempotent. Manual historical reruns, tags, releases and issue jobs refresh ordinary reads but do not replace automatic branch references. Inputs and outputs present in the preparation export are covered, including optional build dependencies; guest-baseline paths not exported are excluded.

Current references start with equal preference for PRs and pushed branches. PR activity decays over seven days; branch activity decays over thirty days, using horizon / (horizon + age). Superseded archives lose that reference's preference; another repository or PR can still keep them current. Their fallback weight is 0.25 versus 8 for a fresh current reference. This weight is multiplied by last-read recency (one-day horizon), a bounded replacement preference (4 for locally endorsed builds, 1 for signed downloads), and divided by stored archive bytes with a 64 KiB minimum bucket. Active/recent read protections still apply. Incoming tracked preparations receive the fresh-reference preference during admission, so a new PR revision can replace its old dependencies under pressure.

Untracked entries retain the earlier heuristic: normal/PR preference 4:1 and one-day/one-hour last-use horizons. These are pressure preferences, never expiry deadlines or guarantees about rebuild expense. Per-derivation build times are not measured. No NAR is deleted just because its last reference disappears.

The index is provider-local and bounded to 1,024 consumers, 100,000 reference memberships and 64 MiB serialized data. When full, a new consumer can replace ordering fences inactive for a year; otherwise its hint is deferred without failing the job. Reference updates use atomic durable file replacement and do not grant publication or signature trust. A failed write leaves the previous reference set in force. Existing archives are not scanned or rebuilt to infer missing repository history.

This is an observation of automatic allocations seen by this provider, not an authoritative view of Git branch tips. The existing allocation does not carry source-event chronology, default-branch identity or PR closure updates. Allocation timestamps and IDs order observations; delayed scheduling of an older revision can still make it the latest observed allocation. All pushed branches receive the slower decay; closed or abandoned PRs decay until new evidence is available. Other providers' observations are not synchronized. These limits affect eviction preference only, never which bytes or signing keys are trusted.

Admission compares the incoming unique archives' aggregate value per byte with eligible eviction candidates. It refuses the incoming publication without first evicting a partial set if enough lower-value space cannot be made available. The private handoff stays pinned, so refusal never makes the current job rebuild. Ordinary pressure maintenance can still reclaim more valuable objects when the filesystem needs the space.

Signed downloads from ordinary upstreams, including cache.nixos.org, already join the controlled preparation export when publication is enabled. They retain their original signatures and now receive the download retention preference. Runtime-only downloads and gateway proxy streams do not independently populate this cache.

Announcement format ​

The cache protocol defines Narwal-compatible announcements, Hashtree nodes, compressed narinfo and the separate trust-list extension. A single announcement binds the Nostr identity to current Nix keys, a catalog root and independent Blossom storage hints. There is no separate identity announcement. The publisher checks its public root object before announcing it and refreshes at least hourly. Its Nostr credential stays in the publisher; the Nix signing key stays in the worker.

Explicit consumer trust ​

sh
ngit-ci-cache fetch --provider <provider-hex-or-npub> \
  --relay wss://relay.example.org --output announcement.json
# Review the provider, Nix key, URLs and intended operator policy.
ngit-ci-cache pin --provider <provider-hex-or-npub> \
  --event announcement.json --lock cache-trust.json
ngit-ci-cache config --lock cache-trust.json

fetch waits up to fifteen seconds for relay connections, queries the configured relays and selects the newest verified announcement. Configure more than one relay: successful publication to one does not imply every relay retained it. fetch does not trust anything. pin requires an explicit identity, verifies the Nostr signature and supported root-announcement shape, and rejects newly installed announcements older than 24 hours or more than five minutes in the future. An existing pin changes only with --replace; replacement must have a strictly newer timestamp. Same-time equivocation and rollback fail closed. The pin approves the Nostr identity and selected channel. Subsequent verified announcements update its Nix keys and catalog without repinning. Direct roots remain durable operator policy. Explicitly opted-in delegation has the separate freshness policy below. Conflicting Nix key names and duplicate identities are rejected.

The lock is versioned JSON containing the complete signed events and expected provider pubkeys. It is public data, suitable for review and version control. config prints trusted-key configuration; use the standalone gateway as the substituter because storage hints need not be Nix cache endpoints. It it does not modify a daemon. Standalone consumers install those substituters and trusted keys using their normal privileged Nix configuration. Native providers use --cache-trust-lock-file, NGIT_CI_PROVIDER_CACHE_TRUST_LOCK_FILE (also read from .env), or NixOS cacheTrustLockFile. Empty/unset disables this additional source of trust. Compose consumers can set the same variable and mount the lock read-only in a Compose override. A repository flake pin alone does not authenticate a cache announcement or grant permission to alter daemon trust.

NixOS publisher ​

nix
services.ngit-ci-nix-provider = {
  # Existing provider, adapter and managed-cache configuration remains required.
  managedCacheUrl = "http://10.231.0.1:17845";
  managedCache.publicListen = "127.0.0.1:17846";
  managedCache.publicUrl = "https://cache.example.org";
  cacheTrustLockFile = ./cache-trust.json; # optional approved peers
};

Reverse proxy that HTTPS origin to the public listener, including its root hash paths and /nix/ metadata. Never proxy the private listener at port 17845. Do not share a generic upload endpoint or lifecycle with repository-retention storage. All offerings using one provider credential share one cache and must use the same upstream key/lock policy. Only the cache-owning offering starts a publisher. The module does not open firewall ports or create DNS/TLS routing.

For an independent publisher, ngit-ci-cache publish --config publisher.json (or NGIT_CI_CACHE_PUBLICATION_CONFIG, also .env) accepts this JSON:

json
{
  "source_url": "http://10.231.0.1:17845/build-cache/publication",
  "public_url": "https://cache.example.org",
  "identity_file": "/run/credentials/cache-publisher/provider-key",
  "state_dir": "/var/lib/cache-publisher",
  "relays": ["wss://relay.example.org"],
  "interval_seconds": 60
}

identity_file contains an nsec or hex secret. Pass the path, never the secret on argv or in a Nix expression. The interval is 30–3600 seconds; --once performs one verified publication for diagnostics. Use a restricted service account with network access, its identity credential and a writable announcement state directory. Give long-running publishers a low-priority resource slice.

Scope and limits ​

Providers remain trusted to execute Nix correctly and protect their signing keys. The current preparation mode uses Nix's build sandbox inside one disposable VM; it does not promise protection against a sandbox escape between builds in that VM. Neither cache signatures nor mutual provider trust prove reproducibility. Coordinator-managed trust, quorum rebuilds, cache-aware scheduling and arbitrary on-demand aggregation gateways are separate policies. This implementation deliberately keeps discovery and trust independent of the scheduler, so those can be added without changing cache storage.

Signed delegation and withdrawal ​

A trust lock can opt selected directly pinned identities into recursive upstream trust. Its optional delegates array contains provider hex pubkeys; every entry must also occur in providers. ngit-ci-cache pin --delegate enables delegation for that root. Omitting the array preserves direct-only trust. This is an operator policy, not something a repository workflow can enable.

Providers publish the separate trust-list event. The publisher reads its configured direct roots without flattening the resolved graph. Trust-list publication precedes the public-root reachability check, so a storage outage does not suppress withdrawal.

The managed worker uses the cache owner's provider relay list to resolve opted-in roots and their upstream lists. It verifies signatures, identity/key bindings and monotonically advancing event versions, and persists the graph and last successful checks in its state directory. Same-time equivocation and rollback do not refresh authorization. The graph is bounded to 32 recorded identities and eight delegation edges in a traversed path; exceeding a bound is an explicit error, not a silently complete partial graph. Disconnected cycles cannot sustain trust. Retaining signed versions allows another direct or delegated route to preserve a key after one route is removed. Direct root identities remain explicitly pinned.

Refresh runs every five minutes with a bounded attempt. Missing relay responses preserve previous signed data and check timestamps; they are not removals. Each route may remain effective for 24 hours after its necessary verified checks. After that boundary delegated keys stop authorizing new preparations and persistent cache reads, even between refresh attempts. Cached bytes remain eligible for ordinary retention, and successful verification can restore their availability. Direct pins and already-issued private handoffs are unaffected. Relays can withhold new events; a successful query is not cryptographic proof that no newer statement exists. Status records make the local freshness decision inspectable.

An explicit signed removal, or an operator root-policy change, removes the corresponding route. Outputs with no remaining authorized signature are removed from the persistent index independently of age or disk pressure. Another valid, currently authorized signature over the same output metadata preserves the item. Archives are reclaimed when no retained record or active private handoff needs them. Expired-but-not-withdrawn routes retain storage authorization separately from execution authorization. Signed versions survive restarts to prevent a stale relay from silently undoing a known withdrawal.

A locally endorsed parent is not revoked merely because a referenced output is removed. Its dependency may need another cache or a build before import succeeds. Cache withdrawal does not reconstruct provenance or invalidate all historical builds that used a former upstream. Compromise recovery remains a separate operator decision.

Each preparation receives the worker's effective Nix configuration at admission; it does not resolve Nostr events itself. Publication checks the current policy again when the preparation finishes. A withdrawal can therefore prevent persistent publication while preserving the current job's private handoff.

Standalone consumers can use:

sh
ngit-ci-cache resolve --lock cache-trust.json --state trust-graph.json \
  --relay wss://relay.example.org > effective-nix.conf
ngit-ci-cache config --lock cache-trust.json --state trust-graph.json
ngit-ci-cache status --lock cache-trust.json --state trust-graph.json

resolve refreshes verified versions; config evaluates saved freshness offline; status reports effective keys, deadlines and signed graph data. These commands do not install or continuously refresh daemon configuration. A standalone operator must arrange updates; a previously generated static Nix configuration does not expire itself. The managed worker enforces the deadlines automatically.