Native provider
Development preview
This page describes a pinned development revision, not a published release. Its package version may still identify the preceding release while the next version is being prepared.
Coordinators process Nix workflows by default, with explicit provider trust required. Set
--nix-workflows=falsefor an act-only coordinator.
ngit-ci-nix-provider is the execution side of ngit-ci's Nix workflow protocol. A coordinator directs each job to one provider Advertisement; the provider receives that allocation on its configured Nostr relays, executes the exact signed source and Nix selector, and publishes a signed Job Result. Run one provider process for each installed environment you want to offer—for example x86_64-linux, aarch64-linux, or aarch64-darwin—without registering those machines with a central server. One signing identity can publish several independent offerings; each process uses its own offering identifier, state/work directories and worker capacity.
Repository authors define these jobs in .grasp/nix/workflows/; see Nix workflows. The signed event contract is in NIP.md.
For a persistent NixOS service, use the flake's nixosModules.nix-provider; see Deploying on NixOS.
Install and run
Disk usage: the managed cache is enabled by default on Nix providers. It grows on demand up to 100 GiB, with cleanup aiming to preserve 10 GiB free on its filesystem. Optional exports have a separate 10 GiB shared staging budget; build stores, workspaces and mandatory artifacts are extra. These are best-effort limits, not quotas. Opt out with
--managed-cache=falseorNGIT_CI_PROVIDER_MANAGED_CACHE=false. Worker staging adds up to one further export-sized copy. See cache setup and retention.
The flake package includes git, git-remote-nostr/ngit, OpenSSH, and nix on the provider's runtime PATH:
bash
nix run .#nix-provider -- \
--system x86_64-linux \
--adapter-socket /run/ngit-ci-adapter-firecracker/adapter.sock \
--blossom-server https://blossom.example \
--relay wss://provider-inbox.example \
--allowed-coordinator npub1...Start the separate-account cache worker and isolation adapter first; see the Firecracker deployment guide. The provider never runs repository code beside its signing credentials. This startup requirement also applies to trusted pushes, releases and manual jobs.
The stable provider signing key is secret and is deliberately not an argv option. Supply NGIT_CI_PROVIDER_SECRET_KEY as an nsec or hex value, or use a systemd credential named ngit-ci-nix-provider-secret-key. A .env file in the working directory is also loaded. The rotating allocation/decryption keys are generated by the provider and exist only in memory. Exact signed Advertisements remain durable without their private keys.
Configuration
A complete provider environment example is provided separately from the coordinator example. The provider loads .env from its working directory. Keep private key values out of argv and the Nix store.
CLI has priority over environment and .env. Comma-separated environment values correspond to repeatable CLI options.
| CLI | Environment | Required | Purpose |
|---|---|---|---|
| No CLI flag (secret) | NGIT_CI_PROVIDER_SECRET_KEY | yes (or systemd credential) | Stable private provider signing key; never put it in argv or the Nix store. NixOS: signingKeyFile. |
--offering-id | NGIT_CI_PROVIDER_OFFERING_ID | no (default) | Provider-local offering identifier used as the kind-39845 d tag. Processes sharing a signing key must use different identifiers. |
--system | NGIT_CI_PROVIDER_SYSTEM | yes | Exactly one native Nix system matching the installed environment. |
--relay | NGIT_CI_PROVIDER_RELAYS | yes | Relays on which the provider publishes Advertisements and keeps its allocation inbox. Use the coordinator's index relays for discoverability; the signed event may name the same relays as its inbox. |
--allowed-coordinator | NGIT_CI_PROVIDER_ALLOWED_COORDINATORS | yes | Coordinator pubkeys allowed to submit work, as hex or npub. |
--state-dir | NGIT_CI_PROVIDER_STATE_DIR | no (default .ngit-ci-nix-provider) | Durable public advertisements, allocation decisions, exact pending results, and the single-process lock. It contains no allocation recipient private keys. |
--work-dir | NGIT_CI_PROVIDER_WORK_DIR | no (default .ngit-ci-nix-provider/work) | Private temporary source, predecessor-input, output, and secret workspaces. |
--blossom-server | NGIT_CI_PROVIDER_BLOSSOM_SERVERS | yes | Upload destinations for the environment manifest, logs and declared artifacts. At least one is required even for artifact-free jobs. |
| No CLI flag (secret) | NGIT_CI_PROVIDER_ENVIRONMENT_UPLOAD_SECRET_KEY | no (provider key) | Optional nsec/hex key used only to upload environment manifests. NixOS: environmentUploadSecretKeyFile, passed as the ngit-ci-nix-provider-environment-secret-key credential. Configure this public key for indefinite retention on your Blossom server to preserve current and old manifests without retaining all job artifacts. Existing blobs are uploaded again to establish ownership; advertisements keep the normal provider identity. |
--environment-manifest-blossom-server | NGIT_CI_PROVIDER_ENVIRONMENT_MANIFEST_BLOSSOM_SERVERS | no (empty) | Optional durable destinations for the environment manifest, which is uploaded at startup and reused in advertisements. Empty uses the job servers; otherwise these servers race until one confirms publication. Alternatively, retain manifests on the job servers with a dedicated environment upload key. NixOS: environmentManifestBlossomServers. |
--blossom-prefer-first | NGIT_CI_PROVIDER_BLOSSOM_PREFER_FIRST | no (false) | Try the first endpoint once, within the Blossom timeout, before starting remaining destinations. Prefer an owned server without racing it against external fallbacks. One confirmed upload suffices. NixOS: blossomPreferFirst. |
--source-and-event-timeout-secs | NGIT_CI_PROVIDER_SOURCE_AND_EVENT_TIMEOUT_SECS | no (30) | Bound for each source and event fetch and each targeted event publication. Predecessor-artifact fetches instead use a fixed 30-second per-request deadline from the shared public-network fetch client, and complete input materialization has a separate budget equal to the allocation’s configured execution timeout. |
--blossom-timeout-secs | NGIT_CI_PROVIDER_BLOSSOM_TIMEOUT_SECS | no (30) | Bound for each Blossom upload or confirmation; each artifact publication attempt is bounded to four times this value. Failed storage transfers retain captured outputs for later retry. |
--environment-upload-timeout-secs | NGIT_CI_PROVIDER_ENVIRONMENT_UPLOAD_TIMEOUT_SECS | no (artifact Blossom timeout) | Separate environment-manifest upload timeout. NixOS: environmentManifests.timeoutSecs in the unified provider module. |
--advertisement-ttl-secs | NGIT_CI_PROVIDER_ADVERTISEMENT_TTL_SECS | no (1800) | Rotating Advertisement lifetime, from 60 through 1800 seconds. |
--max-input-artifact-bytes | NGIT_CI_PROVIDER_MAX_INPUT_ARTIFACT_BYTES | no (1073741824, 1 GiB) | Maximum bytes for one logical direct-predecessor file, including an extracted archive member. Increase this for larger APKs or release archives. |
--max-input-bytes | NGIT_CI_PROVIDER_MAX_INPUT_BYTES | no (2147483648, 2 GiB) | Maximum aggregate predecessor payload bytes (compressed plus extracted) and public-output bytes for one job; must be at least the per-artifact limit. |
--macos-maintenance-socket-enabled | NGIT_CI_PROVIDER_MACOS_MAINTENANCE_SOCKET_ENABLED | no (false) | macOS only: enable the root-only drain socket in the private dedicated provider home. Used by native service activation; unsupported on Linux. |
--max-concurrent-jobs | NGIT_CI_PROVIDER_MAX_CONCURRENT_JOBS | no (automatic with a resource pool; otherwise 1) | Optional legacy ceiling on concurrent accepted allocations. By default a pooled provider derives its ceiling from CPU, enforced host RAM and scratch after reserves. The signed Advertisement reports total/free slots; a capacity race is rejected before execution rather than queued. |
--weak-containerization | NGIT_CI_PROVIDER_WEAK_CONTAINERIZATION | false | Explicitly use a weak-container adapter for trusted native jobs; requires coordinator execution grants. |
--weak-containerization-allow-secrets | NGIT_CI_PROVIDER_WEAK_CONTAINERIZATION_ALLOW_SECRETS | false | Separately admit authorized secrets in weak mode. Coordinator consent and existing secret grants remain required. |
--adapter-socket | NGIT_CI_PROVIDER_ADAPTER_SOCKET | yes | Run source fetch, inputs, Nix, apps, and secrets through the hardware-VM adapter at this Unix socket. Missing/unavailable adapters, or isolation differing from the explicitly configured tier, fail startup; there is no direct execution fallback. |
--resource-pool-config | NGIT_CI_PROVIDER_RESOURCE_POOL_CONFIG | no | Shared host budget JSON, also configured on the Firecracker adapter. Requires enforced adapter cgroups; trusted direct execution is refused in this mode. |
--cache-substituter | NGIT_CI_PROVIDER_CACHE_SUBSTITUTERS | no | Repeatable public, uncredentialed HTTP(S) binary-cache URL copied into isolated guest Nix clients. |
--cache-trusted-public-key | NGIT_CI_PROVIDER_CACHE_TRUSTED_PUBLIC_KEYS | no | Repeatable public name:key cache signing key copied into isolated guest Nix clients. |
--cache-trust-lock-file | NGIT_CI_PROVIDER_CACHE_TRUST_LOCK_FILE | no (unset/empty) | Offline-verified Nostr cache pins, added to guest substituters and keys. NixOS: cacheTrustLockFile. See cache sharing. |
--managed-cache-url | NGIT_CI_PROVIDER_MANAGED_CACHE_URL | no | Job-reachable HTTP(S) base URL of this provider's managed cache worker. Automatically selects the controlled build namespace for preparation, or the allocation's repository namespace for legacy execution, and obtains its public signing key over private IPC. NixOS: managedCacheUrl. Requires the managed worker; no credentials, query or fragment. |
--cache-worker-socket | NGIT_CI_PROVIDER_CACHE_WORKER_SOCKET | no (/run/ngit-ci-nix-provider-cache/worker.sock) | Absolute separate-account worker socket. Required at startup unless publication is explicitly disabled. |
--managed-cache | NGIT_CI_PROVIDER_MANAGED_CACHE | no (true) | Enable worker publication and inventory. Set false to disable; upstream guest cache reads and retained files remain. |
--max-cache-export-bytes | NGIT_CI_PROVIDER_MAX_CACHE_EXPORT_BYTES | no (10737418240) | Guest preflight and per-export host limit, also the aggregate budget for concurrent staged cache exports. The guest selects fitting closures before serialization; the host independently discards an oversized transfer without failing the build. |
--cache-publish-timeout-secs | NGIT_CI_PROVIDER_CACHE_PUBLISH_TIMEOUT_SECS | no (300) | Bound for guest export and the entire managed-cache publication/maintenance operation, including lock waiting; external-cache commands are bounded individually. |
--log-level | NGIT_CI_PROVIDER_LOG_LEVEL | no (info) | Tracing filter. |
The provider publishes its environment manifest and a small stable probe blob before its first Advertisement. Each uses its actual upload signer, destinations and accepts one confirmed upload: a working manifest destination alone does not prove that job logs and artifacts can be published. Startup waits and retries after 2 seconds, doubling the delay up to 60 seconds. Each probe uses at most a 5-second per-transfer timeout and a 20-second total publication budget. No unconfigured endpoint is used; additional copies never delay progress.
At least one configured relay must accept an Advertisement before the provider announces readiness. The NixOS service uses systemd readiness notification and stays activating during a startup storage outage; stopping it cancels startup.
While running, the provider repeats the authenticated small-blob checks every 30 seconds. Three consecutive failed check rounds pause admission and withdraw advertised free capacity; checks between failures use the 2-to-60-second backoff. One failed large artifact, transfer timeout, or pending completion does not pause other jobs. A successful check resets the failure streak and restores admission. A 413 Too Large or other permanent blob rejection fails that publication rather than retaining it as an outage retry. Already executing jobs finish normally.
Rate limits are tracked separately for each destination and upload signer, shared across its blobs and client clones. 429 responses use Retry-After (delta seconds or an HTTP date), or 30 seconds when absent; an invalid header uses a conservative 60 seconds. A 503 with Retry-After also creates a cooldown. Other configured destinations remain usable. If all destinations for a required publication path are cooling down, admission waits for the earliest retry; checks do not send requests during that cooldown. This is backpressure, not a claim that the server is offline. These cooldowns are held in memory.
Allocations racing an old advertisement receive a not-started result. Small checks establish point-in-time publication capability, not that every artifact size or format will be accepted. Sustained permission or storage failures on these checks also pause admission until the operator or service fixes them.
rust-nostr maintains relay reconnects and the persistent allocation subscription. The provider renews its short-lived Advertisement before expiry, and signs a replacement when free capacity changes, with a fresh in-memory recipient each time. Retired keyring generations are wiped at Advertisement expiry plus one minute; an already admitted task keeps only its exact recipient copy until decryption. Restart begins with a fresh recipient and immediately forgets every previous private key.
The shared discovery and inbox relay must accept standalone kind:39845 Advertisements and directed kind:9845 allocations. A repository GRASP relay may accept repository-bound CI results while refusing standalone capability advertisements. In that case add a general-purpose relay to the coordinator's indexRelays and use it in the provider's relays; repository result relays remain independent. Provider discovery and allocation/result traffic use the coordinator's shared relay connections; concurrent allocations share a result subscription per relay. See subscription management for how repository watching leaves room for discovery queries. Check publication acknowledgements and provider readiness before submitting work.
Offerings and environment diagnostics
Advertisements use generic compute kind 39845, W=nix, and qualified selectors such as S=nix:x86_64-linux. Each (pubkey, d) is independently replaceable. Approval remains attached to the provider pubkey. Coordinators choose approved offerings using optional repository resource/environment constraints and exact hash pins, overlaid on local defaults. See workflow selection. A slot count describes one offering, not all offerings under the same identity. Do not configure separate processes to claim the same underlying worker slots.
This replaces the earlier development revision of the provider extension. Stop new intake and let all coordinator runs finish and provider results be acknowledged before upgrading coordinators and providers together. Old persisted unfinished allocations or pending results cannot be recovered with the new advertisement reference format and can prevent provider startup. Shutdown alone does not establish that result delivery completed. Resolve pending delivery with the old version first; retain state rather than deleting it to bypass recovery. Historical terminal run records remain deduplicated. Old providers do not understand allocation f tags, and kind 19845 is no longer discovered.
Every ngit-ci provider Advertisement includes ["x", "<manifest-sha256>", "<blossom-url>"], although the NIP permits other implementations to omit it. The digest hashes the exact JSON file bytes. The manifest is generated from the running provider or its adapter, not from a repository-supplied description. It has no schema-version field. It describes the fixed substrate, not the tools and services supplied by the job's flake.
Allocations pin the exact advertisement version and environment identity. Providers reject a requested environment they no longer have; they do not download execution images to satisfy allocations. Providers check effective requirements at admission and recheck the execution description before starting; a mismatch produces a not-started startup_failure. Resource-constrained isolated jobs require the adapter's configuration.resource-admission: "exact-values-v1" marker: the adapter verifies selected resource values before launching the VM, including after a configuration change. Executed results repeat the environment identity, which is a signed claim, not execution attestation.
Resource declarations use indexable equality expressions such as ["f", "memory-mib:eq:4096"]; allocations use the same grammar with comparisons such as memory-mib:gte:4096 or env/configuration/guest/job-uid:eq:0. Operator resource configuration remains a typed map. This wire revision requires draining queued/running allocations and upgrading coordinators and providers together; old positional f tags are rejected rather than silently ignored.
Each job's uploaded log contains the bounded captured job output, rather than an unlimited stream. App jobs include the separately bounded realization output before runtime output, so successful cache substitution remains visible.
Final isolated-job logs include a provider-generated usage summary in the header, before the job-output separator. The single-job guest samples Linux /proc and scratch filesystem counters once per second from source/input preparation through cache export: CPU time, observed peak MemTotal - MemAvailable, scratch space, swap usage and paging, and OOM kills. Memory includes the guest OS and uses the kernel's estimate of available memory; it is not process RSS, a kernel-maintained peak, or host VMM memory. Scratch is the peak used filesystem blocks on /scratch (total minus free), including the workspace, downloaded or built Nix store paths, and cache exports. It excludes the read-only base image, tmpfs and host storage. This is absolute usage, not growth since job start, and does not include free blocks reserved by the filesystem. Keep headroom for filesystem overhead, cold builds and future dependencies when choosing scratch-mib; it is not a proven minimum. Short spikes between samples may be missed. CPU time excludes idle, I/O wait and stolen time. Elapsed time excludes VM boot and host-side final artifact publication. Measurements are diagnostics from the guest, not trusted billing or admission data.
Snapshots travel separately from job output over the existing bounded evidence stream. A final snapshot marks collection complete; interrupted guests retain the last snapshot as partial. Missing counters are reported as unavailable, never zero. Optional telemetry cannot block shutdown or turn intact logs into partial logs. Direct/native execution and older guests report usage unavailable: whole-host counters cannot be attributed to one concurrent job. There are no new Nostr tags or workflow options.
Execution timeouts retain the captured stdout/stderr tails and append a timeout diagnostic; a timed-out step remains marked as such. Process-group termination, child reaping and pipe draining are bounded, so inherited open pipes cannot indefinitely delay finalization. Granted secret values are masked before these retained logs are published. Each log begins with the environment manifest link and a compact resource summary, including nested host limits. The immutable Job Result retains the manifest reference in x and known actual resource allocations in f tags, including execution error paths. When execution diagnostics are unavailable, the log explicitly labels the fallback as advertised resources; these are not copied into actual-allocation tags. A missing or mismatched manifest URL falls back to the environment hash, never a link to a different environment. Fixed boot artifacts and executor settings belong in the manifest; per-job vCPU/RAM/scratch sizes belong in the resource report. Historical direct-provider manifests remain readable, but direct offerings are ineligible for new allocations. Nix derivation sandboxing does not isolate a nix run app; the disposable VM is the execution boundary.
For example, a Firecracker environment file has this shape (component hashes below are placeholders, and the actual kernel arguments come from the template):
json
{
"system": "nix:x86_64-linux",
"isolation": "hardware-vm",
"components": {
"kernel": "sha256:<kernel-bytes>",
"initrd": "sha256:<initrd-bytes>",
"rootfs": "sha256:<rootfs-bytes>",
"executor": "sha256:<firecracker-binary-bytes>"
},
"configuration": {
"executor": "firecracker",
"kernel-args": "console=ttyS0 panic=1",
"root-filesystem": "read-only",
"scratch-filesystem": "ext4",
"network-interface": "tap",
"jailer": true,
"host-cgroup-limits": true
}
}Tart manifests describe the declared upstream base-image-digest (when known), checked Tart/Softnet versions, selected Swift/Xcode capabilities and configured isolation/network/storage policy. They exclude local disk contents, NVRAM and Tart VM metadata: independently prepared templates must not acquire different configuration identities merely because those bytes differ. The upstream digest identifies shared acquisition provenance, not the locally modified disk; it is an operator claim, not independently attested. The adapter does not hash the VM image at startup. Local file fingerprints still reject template/runtime changes while it is running; startup guest qualification still checks the configured capabilities. This is a configuration claim, not proof of identical guest contents or exact macOS/SDK versions. Network policy and fixed quota settings still distinguish configurations; identical toolchain capabilities alone do not imply identical manifests. Existing Tart environment pins must be updated after deploying this change.
The separate log resource object might be (these scalar values also become offering f tags):
json
{"vcpus": 2, "memory-mib": 4096, "scratch-mib": 65536}Changing RAM alone does not change the environment identity. For the Firecracker example above, changing the kernel, template filesystem or recorded fixed settings does. The Firecracker adapter hashes installed component bytes at startup; keep those files immutable for its lifetime, as with Nix store paths, and restart after changing the template. The adapter checks component file fingerprints before identity-bound execution and rejects detected changes until restart. When host cgroup limits are enabled, the resource report also contains host-cgroup limits such as memory-max-mib, pids-max, and cpu-quota-percent. The manifest does not identify the entire physical host: CPU features, host kernel, outside firewall rules and network speed may still differ. In particular, network-interface: tap does not promise public Internet access. The fields are descriptions that jobs can compare, not a standardized capability vocabulary or a declaration that a job may request arbitrary images.
Templates built with the current Nix guest configuration include an optional guest-configuration.json file (a JSON object, at most 1 MiB), which the adapter embeds as configuration.guest. For example, this excerpt shows matchable fields:
json
{
"job-uid": 0,
"no-new-privileges": false,
"module-autoload": true,
"kernel-features": {"NET_NS": true, "OVERLAY_FS": true, "BRIDGE": true},
"kernel-config": {"NET_NS": "y", "OVERLAY_FS": "m", "BRIDGE": "m"}
}The exact kernel values come from the built kernel configuration. A feature is true for a built-in or loadable module; the raw kernel-config preserves that difference. The generated subset includes namespace, cgroup, overlay, bridge, veth, netfilter and seccomp options. Existing/custom templates without this metadata omit the subtree rather than claiming unverified facilities. These fields describe the guest image and execution settings, not successful Docker execution or runtime delegation of every cgroup controller.
Offering f tags contain scalar/array resource-report fields. Firecracker's vcpus, memory-mib and scratch-mib are per-job allocations; nested host-cgroup diagnostics are omitted. Historical direct-provider observations are not VM resource reservations.
Repository-provided Docker, PostgreSQL and compiler packages belong to the flake, not a provider capability catalogue. An environment description does not establish that Docker works: its kernel and permission requirements still need validation against a real job. Do not publish credentials, secret values, private host paths or machine identifiers as diagnostics.
The coordinator gives result finalization a separate allowance after the latest execution deadline. Its default 900 seconds covers the default cache export and publication budgets (300 seconds each), with time for uploads and delivery. Increase the coordinator allowance when raising provider publication budgets. This wait does not extend the job's execution limit.
Execution and recovery
The provider accepts work in this order:
- require the allocation signer to be an explicitly trusted coordinator;
- validate the exact quoted provider Advertisement, inbox relay, system, offering identity, installed environment and start deadline; require
hardware-vmfor every new allocation before reserving execution capacity; - durably reserve the allocation once;
- fetch and validate the exact source, workflow hash, and signed predecessor results;
- decrypt any secret grant only for execution;
- persist an effect-boundary marker, execute, then persist the exact signed Job Result before publishing it.
Relay redelivery republishes the same result and never reruns the allocation. Already-recorded direct allocations retain terminal recovery and result replay, but cannot execute again. An unrecorded direct allocation is rejected before input preparation or secret decryption, including when it quotes a still-valid Advertisement from an older provider. For scheduling, the coordinator uses the latest unexpired capacity advertisement, minus local preparations and allocations dispatched within the last five seconds. This short deduction limits bursts against a stale snapshot; refreshes, cancellation and restart do not extend its original deadline. Once it expires, missing job results do not reduce advertised capacity. A signed completion can release the deduction early; a refusal retains the remaining window to avoid immediate retry. The provider still enforces actual admission, so a stale snapshot may cause a refusal. Deployment concurrency locks and permission to retry an uncertain job remain dependent on that job's outcome, not this scheduling window. The coordinator listens for results while awaiting relay acknowledgments. Missing or negative acknowledgments do not prove non-delivery: it retries the identical signed allocation at five-second intervals, with at most one send outstanding, until acknowledged or the original start deadline expires. Result listening continues to the existing execution/finalization deadline; cancellation and revoked provider permissions suppress further allocation publication. The daemon also retries undelivered durable results every 30 seconds, in a separate delivery task so an unavailable result relay does not block allocation intake or capability advertisements. This retry never recovers or reruns unfinished allocations. After execution and verified cleanup, the provider checkpoints the completion, sanitized log and captured artifact files under its private state directory. Step artifacts have their own durable snapshots in the progress outbox. Storage outages keep these bytes pending instead of changing a successful execution into an artifact-publication failure or discarding its log. Publication retries run every 30 seconds when storage is healthy, including after provider restart; they never re-execute the allocation. Captured files are removed after the signed result is durable (step snapshots after their references are durable). Cache exports remain best effort and are not replayed from a completion checkpoint.
A permanent step artifact upload failure makes an otherwise successful job fail. Its upload error and captured file names are retained across provider restarts and appended to the final job log, including when publishing that log must be retried. The file inventory describes the failed batch; it does not mean every file in that batch failed to upload. The Job Result's bounded log excerpt may also include these diagnostics. Successfully published files and complete artifact groups are retained in a failed batch's result. An individual-file manifest can describe just the files that uploaded successfully. An archive group requires both its archive and manifest to publish before it can be referenced. Partial references are useful for inspection; the failed step remains ineligible as a downstream step input. If any upload failure is transient, publication remains pending for retry rather than finalizing a partial result.
Pending captures consume provider disk space until publication succeeds; monitor that disk separately from remote publication health. A pending blob alone does not close admission. The coordinator's existing result-recovery deadline still applies; retaining bytes does not extend it or authorize rerunning effects. Shutdown bounds its final delivery phase to one minute and leaves any remaining captures for restart recovery.
If the process stops after reservation but before capturing a completion, restart publishes startup_failure only when cleanup is verified or execution never started; otherwise the allocation remains quarantined. A durable marker proves whether execution may have started: only an unmarked reservation omits the result's started tag and is safe to reassign. A secret-bearing allocation addressed to a pre-restart recipient likewise fails before execution. The coordinator tries a pure build up to three times and can reassign a run only after a signed, provably not-started failure. A missing result never proves an effect safe to repeat. Allocation expiration limits the start of execution. Coordinator recovery continues listening for the exact result after that deadline without asking relays to accept the expired allocation again. If its execution deadline has also elapsed, recovery still allows the normal relay fetch timeout (10 seconds in the coordinator) for a delayed historical response. This grace only waits for the original signed result; it does not allocate or execute another job. When no eligible current Advertisement has free capacity, the coordinator keeps ready jobs pending under the bounded provider wait described in Nix workflows.
Relay ingress is bounded by the configured job capacity. The daemon runs at most max-jobs allocation tasks; while those slots are occupied it signs capacity failures inline, applying bounded backpressure to relay delivery instead of accumulating jobs or rejection tasks in memory. Before touching execution slots or shared-pool leases, it checks coordinator trust, the exact locally stored Advertisement, signed allocation, inbox relay and startability. Instances sharing a provider identity ignore allocations addressed to one another's offerings. Duplicate deliveries of an active allocation do not spend capacity or race its original admission with a false capacity refusal.
Adapter-backed providers do not change the shared host Nix daemon's build limits. Job resource limits are enforced by the adapter; provider publication and cache services have separate host resource accounting.
Nix build jobs select packages or checks; Nix evaluates and builds those inside the isolated guest's local store, with the guest's sandbox policy. Each realised build or app closure is held by a job-local GC root until artifact collection or the effect completes, then removed with the private workspace. Run jobs select a flake app and may receive predecessor files and secrets. Secret values are written as mode-0600 files only after the app closure is realised, and exact granted values are removed from the public log excerpt and scalar outputs before the result is signed. Provider/adapter framing buffers and the guest's raw request buffer are zeroized on drop. This is a last-resort guard, not a general information-flow proof: operate effectful providers under a dedicated account or machine, enforce the desired host firewall/resource policy, and trust only coordinators whose workflows this machine is intended to execute.
Controlled build preparation
On hardware-isolated Linux providers, jobs without secret-assisted build steps use two executions under the existing allocation. The provider first starts ngit-ci-nix-prepare in a fresh VM. That controller realizes the leading build steps and the Nix closures of every run/cleanup command; it never executes those commands, receives no secret values or predecessor artifacts, and enforces Nix sandboxing with unprivileged nixbld users. The whole dependency graph shares one guest store. Pure build jobs finish in this first VM.
The shared host capacity reservation remains held between preparation and runtime. Only verified preparation teardown returns it to the provider for the next VM; failed cleanup keeps it quarantined. Finishing or abandoning the job releases the reservation after cleanup.
Preparation streams NARs directly into the credential-separated cache worker; it writes no handoff archive or binary-cache copy onto guest scratch. The worker validates archive structure, size and hash without unpacking it. Only the provider can issue or seal an upload session. Once preparation is torn down, the provider seals that session and starts a fresh runtime VM, which imports its required closure through a read-only, job-scoped Nix binary-cache endpoint. The runtime receives no upload capability. Its Nix importer streams archives into the store without staging a complete handoff archive.
Required outputs remain pinned until the allocation finishes, independently of persistent-cache admission. A refused optional publication does not make the current job rebuild. The same NAR file serves both handoff and retained cache; there is no additional unpacked host store for this path. Required handoffs are bounded at 16 GiB; ordinary artifacts remain bounded at 2 GiB. Disk floors, operation deadlines, path counts and upload sizes are enforced independently. Expired sessions and failed uploads are reclaimed; cache-worker restarts revoke in-flight session capabilities, so interrupted jobs cannot silently resume. External-cache-only deployments retain the private adapter handoff fallback.
Concurrent preparations can realize the same input-addressed store path with non-identical metadata or contents. Reference order is normalized. If the realizations still differ, the first published version remains immutable and the other job keeps its exact private handoff. The conflicting realization and its dependent outputs are excluded from publication, including when an optional conflicting dependency was not uploaded. This avoids mixing incompatible closures without treating cache admission as a job failure. Temporary private versions can coexist until their jobs release them.
Workflow YAML, flake outputs, and Nostr event shapes are unchanged. Preparation and runtime remain parts of the allocated job; the remaining execution budget is reduced by time spent in preparation. Existing build-step summaries and the nix-realise summary describe preparation. No coordinator-side build graph, new scheduling exchange, or cross-provider discovery is introduced.
Provider-trusted build cache
With managed publication and managedCacheUrl enabled, controlled preparation can populate a separate provider-trusted namespace, including for ordinary PRs. The streamed cache uses /build-cache/, a persistent provider-specific Nix signing key, and NAR-only storage. Its identity is independent of repository and coordinator identity, so identical outputs can be reused across repositories. Existing repository stores remain separately scoped and are never promoted into this namespace. The older controlled store is not automatically migrated.
Preparation reads only operator-configured upstream caches and this controlled namespace. It does not read repository-signed or temporary PR caches. Runtime exports, runtime progress messages and arbitrary workflow files can never publish into the controlled namespace. The trusted guest controller inventories registered store objects, allowing completed dependencies to be exported even when a later Nix build fails or reaches its deadline. Publication remains bounded and best-effort; interrupted guest teardown or export/staging pressure can still prevent retention. This does not provide per-derivation streaming checkpoints. An explicit build-step cache: none disables controlled publication for that preparation, conservatively covering its dependencies too.
This mode trusts Nix's evaluator, sandbox implementation, build-user separation and the guest kernel to isolate derivations. It protects preparation from the surrounding root workflow, but does not claim containment of a sandbox/kernel escape by an earlier derivation. Per-derivation VMs and comparative benchmarks are future work. Secret-assisted builds, weak-container execution and non-Linux providers retain their repository-scoped authorized publication policy; they cannot populate this provider-trusted namespace. Public cache serving is not a read confidentiality boundary. Operators approving upstream cache keys also approve their contents as inputs to controlled builds.
Deploy provider, cache worker, adapter and guest image together for this pre-alpha change. No data migration is required. External cache publication and automatic cache trust between providers are outside this change.
Retention of controlled outputs
Trust and retention are separate. The streamed controlled cache tracks successful preparation references across repositories and revisions, then uses those references, recency, replacement preference and compressed size under pressure. See retention under pressure for the policy, bounds and limits of provider-local branch observations. Active handoffs remain protected and admission refusal does not discard the current job's private handoff. Retention creates no additional signing namespace.
The older managed-store export path retains its one-hour PR and 24-hour normal last-use preferences and tiered GC. Those are soft preferences, not expiry timers. Its private preparation reports refresh use across registered dependencies; anonymous HTTP reads refresh last use without changing the tier. That path does not maintain the streamed cache's consumer-reference index.
Caches and artifact transport
Select upload destinations and discovery relays using the Blossom and relay service considerations. Storage limits, proxy timeouts, retention and relay kind policies are independent of provider input limits.
The published environment includes the effective provider input policy at /configuration/artifacts/max-input-file-bytes and /configuration/artifacts/max-input-total-bytes. These numeric byte limits support ordinary environment gte requirements; changing either changes the environment hash. They describe input admission, not Blossom upload capacity. The total budget includes compressed plus extracted predecessor payload and public-output bytes. Providers still enforce limits during retrieval; advertising them does not automatically derive scheduling requirements from artifact sizes.
All artifact and manifest publications send a signed upload, even when the bytes already exist. A successful existence check alone does not establish ownership. Retention remains server policy: on a server using expiry from first upload, uploading again under an ordinary key does not renew the grace period. Use a retention-exempt identity for permanent metadata; repository logs and artifacts still require timely retention claims by the repository service.
All cache archive processing and publication run in the separate-account cache worker, reached through --cache-worker-socket. Startup fails if the configured worker is unavailable; there is no in-process fallback.
The NixOS module's cacheSubstituters and cacheTrustedPublicKeys options add operator-managed, signature-verified binary caches to the execution side: each disposable guest. Enabling isolated cache consumption does not mutate host-wide Nix daemon settings. Workflow files cannot add substituters or trusted keys. Only validated, uncredentialed cache URLs and public signing keys cross the VM boundary, and NIX_CONFIG is present only on Nix client subprocesses—not in the release app environment. The repository's own Rust packages are built with Crane: one dependency derivation is shared by the coordinator, provider, adapter, and guest binaries, so source/documentation edits do not rebuild third-party crates.
External cache publication and legacy job-controlled exports remain separately authorized. External publication is opt-in with cachePublishUrl. The coordinator signs cache-write=shared only onto pure build allocations triggered by a confirmed repository maintainer. Ordinary pull-request builds cannot publish to that trusted cache; isolated reuse is an internal managed-cache optimisation described below. For an isolated build, the guest copies the successful closure to an unsigned file binary cache under its bounded artifact channel. The host verifies every referenced closure recursively by NAR hash before copying it to the configured destination. Supplied signing keys and S3 credentials exist only in the cache worker's systemd credentials and never enter the provider or guest. Export or publication failure is an optimization miss, not a build failure. The worker has separate low CPU/I/O weights, a 2 GiB memory ceiling, no swap and a 128-task ceiling in the NixOS module.
With the hardened NixOS module, put a local destination beneath the writable state directory, for example file:///var/lib/ngit-ci-nix-provider-cache/shared-cache; arbitrary host paths remain absent from the worker's restricted filesystem view.
Use one destination for the provider fleet rather than one per repository. nix copy skips store paths already present, while binary-cache NAR payloads are keyed by content hash, so identical dependencies and payloads converge on one physical copy across trusted repositories. Authorization remains per signed allocation even though storage is shared. For an external destination, point cacheSubstituters at its public HTTP view and configure its public signing key. Managed storage instead uses the repository-scoped reader configuration.
Successful authorized builds also export realised build-time inputs and their closures, including separately packaged Rust dependency compilation outputs. The executor roots these outputs through publication. This does not manufacture intermediate derivations from a monolithic Cargo build: the flake must expose that structure (for example with Crane). Missing/already-collected inputs or an export exceeding the existing byte/time limits remain cache misses, and failed jobs and run effects do not publish. Cache reads never execute a cached pure check again.
Before serializing an isolated export, the guest queries Nix path sizes and references. It prefers smaller outputs whose complete runtime closures fit the export budget and (for the managed backend) the admission reservation. Large intermediates can be skipped independently of smaller compiled dependencies; this does not export arbitrary Cargo target directories. Exports use uncompressed NARs with metadata slack in the estimate; byte enforcement on the host remains independent. A bounded snapshot of the provider's retained paths lets the guest skip already-cached closures. This is only an optimization: eviction after the snapshot can cause a later miss. Exported closures stay self-contained, including already-cached references needed to verify them.
Optional staging shares one byte reservation budget across all active jobs and completed spools waiting for publication. Reservations are released only when the spool is removed. The filesystem check separately reserves only pending writes: flushed bytes already reduce the filesystem's available space and are not charged twice. Before each cache chunk, the provider also preserves the managed cache's free-space floor (when configured), plus 2 GiB per job slot for mandatory artifacts/results. Block and metadata slack counts against the staging budget. These are conservative best-effort checks; unrelated disk writers are not controlled. Cache-specific filesystem failures discard the optional export while continuing to receive artifacts and results. Guests stage mandatory artifacts and a valid outcome first, leave 16 MiB of export headroom, and replace outcome metadata atomically only after successful cache export.
VM cache readers enable Nix's build fallback if substitution fails. Logs distinguish publication eligibility, read hits/misses (at debug level), selected/admitted/skipped bytes, and admission pressure. Failed export or publication never changes a successful job's conclusion.
Before either managed or external publication invokes host Nix on a received binary cache, ngit-ci validates its metadata references. Archive URLs must name regular files directly beneath the received nar/ directory; URL schemes, parent traversal, encoded paths, duplicate URL fields and symlinks are rejected. Store paths and references must remain valid /nix/store names. This limits what host Nix may open from guest-provided metadata; it does not replace NAR integrity verification or credential separation for the cache-processing worker.
Migrating native provider cache configuration
The native provider no longer accepts cache storage, listener, credential or external-destination CLI flags. Host --nix-bin and --git-bin options are also removed; job tools belong to the guest and cache Nix belongs to the worker. Move those settings to the worker JSON and configure only its socket on the provider. Existing NixOS managedCache, cachePublishUrl and cache credential options now configure the worker service. Legacy v0.1.x act deployments are unaffected.
The NixOS worker uses a new private state directory; old provider cache data is preserved, not automatically migrated or deleted. Preserve its signing identity with cacheSigningKeyFile when provisioning the worker, and verify any retained data migration separately before removing old storage. A fresh worker cache is initially cold. Reserve space for both stores during migration. Never make the provider's private state directory group-readable to expose an old cache.
Managed provider-local cache
The managed cache is enabled by default, unless an external external_destination / cachePublishUrl is selected. It grows on demand; the 100 GiB ceiling is not preallocated. The worker uses <state_dir>/cache and the NixOS module uses /var/lib/ngit-ci-nix-provider-cache/cache. NixOS exposes managedCache.listen, maxBytes, maxAdmissionBytes, minFreeBytes, and checkIntervalSecs.
Disable it with --managed-cache=false, NGIT_CI_PROVIDER_MANAGED_CACHE=false, or services.ngit-ci-nix-provider.managedCache.enable = false. Disabling it stops publication and maintenance but does not delete retained data or disable configured upstream cache reads. External publication remains an explicit alternative and cannot be combined with a managed cache directory.
Repository trust and migration
Each managed trusted cache is scoped by the signed coordinator identity and a coordinator-assigned logical repository ID (cache-scope). Its URL is /repositories/<scope>/, where scope is a SHA256 digest of that binding. Each namespace has an independent Nix store, metadata and generated Ed25519 signing key under <cache-dir>/repositories/<scope>/. Private keys never leave the cache worker; identical immutable files can share disk space without sharing signatures or metadata. Keys persist across restarts and output eviction.
Before assigning an ID, the coordinator requires repository-control replay for the current confirmed members from at least one connected repository relay. Unanswered maintainer invitations never link caches. The coordinator looks up any current confirmed <maintainer>/<identifier> coordinate in its durable cache-identities/identities.json registry beneath the Nix run-store directory. It reuses an existing ID, updates the member aliases, and removes departed aliases. Additions and departures preserve the cache while at least one current member overlaps the previously observed group. If all membership changes happen without an observed overlap, a fresh cache is created.
When two previously separate repositories join, the coordinator keeps one existing ID and retires duplicate mappings. Retired namespaces are no longer selected for new allocations; their outputs remain subject to normal LRU eviction, so reconciliation does not delete data beneath active jobs. Persist and back up the coordinator registry to preserve cache continuity across restarts. This mapping does not change job or secret authorization.
Cache inventory hints contain at most 10,000 paths and 512 KiB of serialized path data. They prevent re-exporting already-retained closures; paths beyond this bounded hint remain available through normal Nix substitution.
Set managedCacheUrl to this worker's guest-reachable base URL. For each allocation the provider obtains only its repository's public key over authenticated private IPC, then injects that key and namespace URL into the disposable guest. No workflow setting or manual public-key copy is needed. The HTTP service is public/read-only, not a confidentiality boundary: knowing another namespace URL does not grant publication authority or make its signatures trusted by your job. For private build contents use a separately protected deployment; this feature does not implement HTTP read ACLs.
Only authorized builds with signed cache-write=shared can publish to their repository's trusted namespace. PRs can read their repository's trusted cache, but cannot write it. Secret-bearing jobs can consume their repository's authorized trusted publications. Cache signatures establish publisher authority, not proof that the bytes came from the claimed derivation; repository maintainers remain trusted by that repository's consumers.
Upgrade the provider and cache worker together (private IPC version 2), before upgrading the coordinator: older providers reject the new allocation tag. Allocations without cache-scope retain the coordinator/primary-coordinate binding, including allocations already frozen for replay. Existing shared-store objects and keys are retained on disk and counted against the disk budget, but the old unscoped HTTP endpoint and inventory are no longer served. They are eligible for ordinary pressure eviction. No old object is silently assigned to a repository: rebuild/re-publish through an authorized allocation. The old cacheSigningKeyFile applies only to the retained legacy store (or an explicit external destination); new repository keys are generated independently.
Remove the old managed-cache URL/key from cacheSubstituters and cacheTrustedPublicKeys, then configure managedCacheUrl. Those upstream options remain available for deliberately trusted external caches such as cache.nixos.org. External cachePublishUrl destinations retain their existing operator-managed trust policy; repository scoping here applies to managed storage. The provider does not discover other providers' repository keys or caches.
Temporary untrusted PR caches have been removed. Non-Linux PR jobs can still read approved caches, but cannot publish reusable untrusted outputs. Remove temporaryCacheUrls / --temporary-cache-url from deployment configuration and temporary from standalone cache-worker JSON. During the stopped pre-alpha rollout, obsolete <state_dir>/cache/temporary data can be discarded; it is no longer served or automatically maintained. Keep the shared-file pool and trusted namespaces intact.
Retention/publication starts automatically; configure managedCacheUrl with the listener's job-reachable HTTP base URL to enable repository-scoped reads. The loopback listener default is not reachable from a guest; VMs need an explicitly reachable address and the adapter's host TCP service exception. The provider does not open firewall ports or publish cache discovery metadata. Keep this uncredentialed reader on the intended job network. Multiple providers on one host need separate cache directories and listener ports. It streams uncompressed NARs to avoid keeping a second archive copy on disk.
Retention roots protect each admitted output and its references. Reads refresh access timestamps across the requested closure. Every configured interval, and before an import, the cache worker checks allocated disk usage and available filesystem space. Under pressure it releases the least-recently-used eligible output roots across all repository namespaces and the quarantined legacy store, aiming for ten percent headroom. It never evicts a whole trusted repository merely because one output is old. Nix GC runs only against each affected private store; the byte budget counts hard-linked files once across the entire managed cache. It first collects unrooted leftovers, then removes up to 32 oldest eligible roots per batch, running GC and measuring actual disk usage once per batch. Shared references remain Nix's responsibility. Batching can release more roots than the minimum needed, but avoids repeated full-store scans for each dependency. Cleanup logs report released roots, GC passes and disk-accounting passes. Publication and maintenance share a writer mutex, but source inspection, verification, copying and signing do not hold the reader-exclusion lock. Only GC/pressure trimming excludes readers. Nix imports are transactional and the HTTP reader requires signatures on the whole requested closure, so in-progress unsigned imports are not served. The publication deadline covers the entire operation, including waiting for another writer or active readers; maintenance has the same overall bound. Recently advertised closures have a five-minute grace period between metadata and payload requests; active transfers hold a reader lock, with a five-minute transfer deadline. Job execution uses its own copied store and GC roots.
If retained data, concurrent disk users or fixed store metadata prevent meeting the limits, the provider logs pressure and refuses further cache admission; the build result remains valid. These are best-effort limits, not filesystem quotas: imports reserve estimated unpacked space, then recheck actual usage. Each admission is bounded by cache-max-admission-bytes and the whole-cache limit before eviction. The reservation estimates twice the NAR size plus one MiB per new path for unpacking and metadata. Selected archives are verified before any existing retention roots are released; invalid or oversized-only exports cannot evict good data. Already-present selected paths remain pinned during reservation cleanup. An interrupted import may leave unrooted data, reclaimed on a later pressure check. Access timestamps and roots survive restart. Do not run another writer, external GC or a second provider against this directory.
Predecessor downloads are bounded, SHA-256 verified, and run four at a time. Environment manifests, artifact inventories and predecessor files support up to five public-CDN redirects. Redirects reject credentials, TLS downgrades, nonstandard ports and private addresses. DNS answers are checked before each new connection; pooled connections retain that checked destination. Existing total deadlines, byte limits and hash checks apply. A cold Firecracker guest starts without mutable state and therefore fetches predecessor blobs from Blossom each time. Artifact hashing and uploads are streaming; different artifacts can upload concurrently. One confirmed destination per blob is sufficient; outstanding uploads to other destinations are cancelled. Optional replication and retention by repository hosting services do not gate job progress. The initial publication must work even when those services do not replicate blobs. Transient upload failures receive up to three attempts within the existing publication deadline. HTTP rejections include the filename and status; permanent client errors are not retried. Rate-limit cooldowns apply across uploads and never shorten a valid server-requested delay.
Hardware isolation
The NixOS provider module no longer modifies the host Nix daemon or creates a direct-job slice. Legacy directMaxBuildJobs, directCoresPerBuild, directCpuQuotaPercent, directMemoryHighMiB and directMemoryMaxMiB options remain accepted as no-ops so existing isolated-provider configurations can upgrade. They do not constrain guests or provider memory. Configure guest and VMM limits on the adapter. directCpuWeight, directIoWeight, directTasksMax and directOomScoreAdjust retain their existing effect on the provider/helper service under their historical names; none enables host job execution.
The host provider requires --adapter-socket and performs admission, resolves signed predecessor events, decrypts the exact authorized secret grant, signs results, and publishes artifacts. Repository checkout, network input downloads, Nix evaluation/builds, apps, predecessor bytes, and plaintext secret files exist only inside a fresh Firecracker VM. The immutable base filesystem is attached read-only, writable Nix/store and operating-system state use a newly formatted scratch disk or tmpfs, and the VM is cold-booted and destroyed for every job. See the microVM adapter guide.
Graceful shutdown and upgrades
The coordinator and provider have different shutdown responsibilities. Delegated Nix workflows are durable: the coordinator can stop without waiting for their jobs, then recover signed results from relays after restart. A coordinator also running act jobs still drains those local executions.
The provider also polls the adapter's read-only environment description every 30 seconds, with a five-second probe deadline. A single independently scheduled monitor keeps relay publication waits from consuming the probe budget; it stops when the provider watcher is dropped. An unavailable/quarantined adapter, a failed probe, or changed environment/resources triggers the same capacity withdrawal and graceful drain as a stop signal, then exits with an error. It does not cancel otherwise running work merely because a probe failed. A supervisor may restart the provider, but startup must obtain a valid adapter description before advertising again. This check applies to Linux and macOS; no new protocol message is required. Polling and relay delivery are not an atomic admission barrier, so the adapter still validates each execution itself.
On SIGTERM or SIGINT the provider stops starting allocations and advertises zero available slots. It waits for admitted jobs to finish or reach their existing allocation deadlines, then finishes artifact/log publication and signed result delivery. Results go to their designated relays; the coordinator need not be online. After jobs finish, a final delivery phase retries pending signed results for up to 60 seconds. Relay failure never erases the durable result: undelivered results remain in the provider state directory for startup replay.
For native macOS service maintenance, --macos-maintenance-socket-enabled enables a mode-0600 Unix socket at /private/var/lib/ngit-ci-provider/maintenance.sock. Its parent must be a private directory owned by the provider. Only a peer with UID 0 may send drain or status (one newline-terminated command per connection). Replies are running, draining, drained or failed. The drain uses the same execution and result-publication path described above. drained means execution and bounded publication have finished; durable undelivered results still survive for retry. The provider then stays alive without admitting work until SIGTERM/SIGINT, so launchd's KeepAlive cannot restart it between drain completion and service stop. A failed drain remains stopped from admission and reports failed; the setup tool refuses to stop services on that response. No remote or workflow-facing maintenance interface is exposed. Linux shutdown behavior remains unchanged.
The NixOS units signal only their supervising process (KillMode=mixed), so SIGTERM does not kill VMMs while they drain. Their systemd stop allowance is unlimited rather than imposing a second, shorter job deadline; execution and publication retain their own bounds. Forced termination, host crashes and power loss still use the existing recovery path.
For adapter-backed execution, services.ngit-ci-nix-provider.adapterService defaults to ngit-ci-adapter-firecracker.service. Start ordering is reversed on stop: the provider completes its drain before the adapter or its network policy is stopped. Adapter restarts propagate to the provider. Set this NixOS-only option to the actual unit for a custom adapter, or to null only when an external supervisor provides equivalent lifetime coordination. Direct providers also stop before the Nix daemon.
Adapters close new admissions and await active handlers, including VM cleanup and terminal messages. They continue honoring caller disconnects; shutting down the coordinator does not disconnect the provider-to-adapter connection. A client connected without an execute message remains bounded by the existing 30-second handshake deadline.
Retiring an undeliverable result
A relay outage should normally be retried. A confirmed permanent refusal, such as an old result exceeding every target relay's event-size limit, may instead need an explicit operator decision. Publication warnings include the result and allocation IDs and relay failure details; no automatic rejection classification retires results.
Stop the provider and run the maintenance command as its service user, using its existing state directory. The command takes the daemon's exclusive state lock and refuses a mismatched result ID, unfinished job, or already-delivered result. It needs no signing key, adapter or network access:
sh
ngit-ci-nix-provider --state-dir /var/lib/ngit-ci-nix-provider/state retire-result \
--coordinator <coordinator-hex> --allocation <allocation-event-id> \
--result <result-event-id> --reason 'All target relays reject the event size'The signed allocation and result stay byte-for-byte intact, preserving execution deduplication. A separate delivery-retirement.json records the exact result, reason and retirement time. Retirement stops retries, including duplicate allocation replay; it neither claims relay acceptance nor publishes a different workflow outcome. Keep this state backed up. Restart the provider afterward. Older provider versions do not honor retirement records and will resume retries.
To restore delivery, stop the provider again and use resume-result with the same coordinator, allocation and result arguments (without --reason). The retirement record gains a resume timestamp; startup retries the original signed result without rerunning its job. A later retirement replaces this administrative record with the latest decision; signed execution history remains immutable.
Publication deadlines
For large artifacts on slow links, set --blossom-timeout-secs to cover the entire upload attempt, not just connection establishment. For example, a 450 MiB archive needs roughly 30 minutes at 0.25 MiB/s before protocol overhead. Upload authorization covers the configured attempt deadline plus a one-minute margin (with a five-minute minimum); each retry receives fresh authorization. Increasing this setting also increases the total publication budget below.
Execution and publication have separate deadlines. Declared artifact publication gets four times --blossom-timeout-secs in total (120 seconds by default); log publication has its own equivalent budget. Optional cache publication runs alongside artifact publication with --cache-publish-timeout-secs (300 seconds by default). Each targeted result publication uses --source-and-event-timeout-secs, and the final shutdown retry phase has its separate 60-second budget. These phases do not consume or extend the allocation's execution timeout.
Cancellation and uncertain cleanup
A coordinator sends allocation-scoped signed cancellation with the original signed allocation proof. The provider authenticates both events, writes a durable cancellation record, and prevents a later delivery of that allocation from starting. Running jobs, including isolated Nix guests, receive TERM on their leader and may finish within the allocation's grace, bounded by the execution timeout. A job that completes successfully within grace keeps its result; an explicit early-stop report retains cancellation semantics. See the workflow cancellation contract.
Forced cancellation (kind 9849) kills the workload without an application grace and escalates an ordinary request already in progress; the stronger request is persisted across restart. Isolated providers forward ordinary cancellation to the guest immediately. Their supervisors retain a bounded best-effort window for evidence capture after force or grace expiry, before sandbox destruction; see the adapter protocol.
The provider returns cancelled only after cleanup is confirmed. Previously persisted results remain unchanged. Upgrade result readers, providers, adapters and guest templates together; incompatible adapters/templates are rejected before job execution. Readers must also support the step-only unknown outcome before deploying the updated step helper.
Public source/input preparation retains its existing phase deadlines and can finish before cancellation is terminalized. Each Git command and predecessor result fetch uses --source-and-event-timeout-secs (30 seconds by default); one clone URL uses up to five Git commands, and fallback URLs are tried sequentially. Each input result is resolved separately, then complete artifact materialization has a separate budget equal to the allocation’s configured execution timeout; each artifact request still has its own fetch timeout. The guest executor's Nix configuration probe has a separate 10-second limit. These are phase budgets, not a single 30-second cancellation deadline. Accepted cancellation still prevents the prepared allocation from crossing the execution-start gate.
If the adapter disconnects, reports cleanup failure, or never acknowledges a cancellation, the provider quarantines that slot and retains the allocation without a fabricated terminal result. Any restart after the execution-start boundary but before a terminal result was persisted also quarantines the slot, including records written by older versions. This prevents a late cancellation from finding capacity already falsely freed. Neither expiry nor restart clears quarantine. Coordinator progress remains unresolved while cancellation is pending; a relay ACK is not a stopped job. Direct providers retain the existing process-group boundary: this is not a security boundary against hostile processes escaping their group, and shared Nix daemon builds may continue serving other clients after this client disconnects. Use hardware-VM providers for untrusted execution.
To recover a quarantined allocation:
Stop the provider and its execution adapter. Independently verify that the exact allocation's VM, process group, containers, and mounts are gone. If that cannot be established, keep the slot quarantined.
As the provider service user, with its existing state directory, record the verified cleanup and an audit reason:
shngit-ci-nix-provider --state-dir /var/lib/ngit-ci-nix-provider/state confirm-cleanup \ --coordinator <coordinator-hex-pubkey> --allocation <allocation-event-id> \ --reason 'Verified allocation VM and associated resources destroyed'Restart the adapter and provider. Recovery signs and persists a
cancelledresult when cancellation was accepted, otherwise an indeterminatestartup_failureretaining its original start evidence, then retries delivery. It preserves the original allocation and execution-start evidence and never executes the allocation again.
The command requires exclusive access to the stopped provider's state directory. It records an operator assertion, not an automatic cleanup check. Do not delete allocation, cancellation, or started records to recover capacity: doing so loses the replay barrier.
Shared host resource pools
A fixed offering can draw from one durable host CPU, RAM and scratch budget. Configure the same --resource-pool-config on all participating providers and adapters. Each provider still has its own fixed environment and --max-concurrent-jobs ceiling. For example, 1-vCPU/4-GiB, 2-vCPU/8-GiB and 2-vCPU/16-GiB offerings can share the allowance; a repository's minimum predicate selects a suitable tier and never authorizes resizing a guest.
The allowance charges each adapter's enforced host cgroup MemoryMax, including VM overhead, rather than only guest RAM. Scratch charges cover the entire fixed VM disk. Reserve additional pool-wide RAM for supervisors and disk headroom for host staging and filesystem overhead. Physical free space is checked again at admission; this check conservatively counts complete outstanding scratch reservations even after some bytes have been written. Pool and guest scratch must use the same filesystem. Cache workers remain independently managed; see disk sizing and pressure for budgeting their growth and transfers without reserving already occupied cache blocks twice.
The provider reserves before accepting work. Its adapter adopts the exact allocation lease before VM creation and validates the fixed tier's full charge. Different service processes serialize through a private durable ledger. A dead provider's unadopted preparation can be reclaimed; an adopted lease is never freed by elapsed time, process death or an uncertain cleanup response. Verified cleanup releases it before the terminal acknowledgment. Quarantined entries require offline cleanup confirmation after stopping the original services and checking their sandbox/cgroup/scratch resources. Pool budget mismatches fail closed; do not replace the ledger or change its identity while leases remain.
To change a budget, stop every provider and adapter using that pool and resolve outstanding leases through verified cleanup first. Prepare the replacement JSON configuration, then run ngit-ci-nix-provider --state-dir <provider-state> reconfigure-pool --configuration <replacement.json> --reason <audit-reason>. The command requires an exclusive pool lifecycle lock and an empty lease ledger; it preserves prior state and records the old/new budgets and operator reason. Restart every participant with the identical replacement configuration. Never remove the ledger to resize or unquarantine a pool. Cleanup and budget audit records are retained without automatic pruning; monitor the small JSON ledger's size and retain operator backups as part of host maintenance.
The optional pool resource is an operator convention carried by existing f resource tags (pool:eq:<opaque-id>), not a new mandatory protocol field. Use one stable identifier per physical pool within one provider signing identity; all its tiers must use that identity and identifier. Equal identifiers under different provider keys do not assert shared hardware. The coordinator uses a fresh offering snapshot to let older eligible large-tier work hold back younger work in that known pool, while unrelated pools continue. It prefers the smallest concrete suitable tier. This is coordinator-local fairness; other coordinators still race at the provider's atomic admission boundary.
For managed NixOS providers, put the budget in services.ngit-ci.providers.<name>.resources; see the unified module guide. The module creates the internal pool identifier and derives offerings and adapter slots. The runtime still uses --resource-pool-config / NGIT_CI_PROVIDER_RESOURCE_POOL_CONFIG for the shared JSON budget; operators running the binaries directly supply that file to each fixed offering and its adapter. Non-pooled adapters, including Tart, retain their existing one-slot default.
For low-level NixOS configurations, define services.ngit-ci-resource-pools.<name> and select that name in both services.ngit-ci-nix-provider.resourcePool and services.ngit-ci-adapter-firecracker.resourcePool. The pool module creates the private group, ledger directory, common configuration and aggregate systemd slice. Every participating provider and adapter joins that slice, so its CPU quota and MemoryMax also bound all guests, supervisors and their caches together. memoryHighMiB controls its pressure threshold; cpuWeight and ioWeight default to 20, and swap is disabled. Set sliceName without the .slice suffix to nest the allowance under an existing host hierarchy, for example apps-background-ngitci-shared; align any host-specific service Slice overrides. Set publicId, vcpus, memoryMiB, scratchBytes, reservedVcpus, reservedMemoryMiB and reservedScratchBytes explicitly for the host allowance. The services retain their own sandboxing and per-job cgroups; enabling a pool grants no additional repository or secret authority. Direct execution cannot opt into this pool until it supplies equivalent per-allocation enforcement and cleanup ownership.
Named NixOS tiers
Keep an existing singleton provider/adapter pair and add fixed tiers with services.ngit-ci-nix-provider-instances.<name> and services.ngit-ci-adapter-firecracker-instances.<name>. Both reuse the singleton options and hardening. Names contain 1–12 ASCII letters, digits, underscores or hyphens. Offerings with the same provider credential share one provider service account. The modules retain separate state directories, sockets, root-owned jailer profiles and service units. Adapters and VMMs share a second account per provider; the cache worker uses a third. No account is created per job or slot. Guest operations check the real caller UID against the selected profile. Offerings of the same provider share that authority; immutable profiles still constrain privileged paths and resources. Configure distinct TAP names and non-overlapping bridge subnets explicitly; evaluation rejects collisions, including with the singleton.
For example, an additional 8-GiB tier uses the following options alongside the existing singleton and shared-pool definition:
nix
services.ngit-ci-nix-provider-instances.tier8 = {
enable = true;
resourcePool = "shared";
signingKeyFile = existingProviderSigningKey;
relays = existingProviderRelays;
allowedCoordinators = existingTrustedCoordinators;
adapterSocket = "/run/ngit-ci-firecracker-tier8/adapter.sock";
maxConcurrentJobs = 1;
maxCacheExportBytes = 4 * 1024 * 1024 * 1024;
};
services.ngit-ci-adapter-firecracker-instances.tier8 = {
enable = true;
resourcePool = "shared";
vcpus = 2;
memoryMiB = 8192;
scratchMiB = 32768;
maxJobs = 1;
tapDevices = [ "tier8tap0" ];
network = {
enable = true;
bridgeAddress = "10.232.8.1/24";
subnet = "10.232.8.0/24";
allowedHostTCPServices = [ { address = "10.231.0.1"; ports = [ 17845 ]; } ];
};
};Give all tiers the same signingKeyFile path and pool identifier. The NixOS module groups enabled offerings by credential path without reading secret material. Each group shares one provider service account and one cache worker, store, namespace keys and disk budget. Different provider credentials must use distinct provider accounts and groups. Different credential paths remain separate, even if their files contain identical keys.
Configure managedCache, managedCacheUrl and external publication credentials on the singleton when it belongs to the group; otherwise configure them on the lexicographically first enabled named offering. These owner settings apply to the whole group, including disabling its cache. Other offerings' cache-owner settings are ignored. All guests must be able to reach the owner's cache URL. Upstream substituters remain per-offering policy.
Each offering retains its own service, state directory and export-staging limit. Configure the shared provider user and group on the group owner as well; conflicting member accounts are rejected. Adapters are associated with providers by matching their sockets. Their shared account defaults to the singleton adapter, or the first named adapter for that provider. Standalone adapters without a locally configured provider retain independent accounts. The cache worker runs under its own account, authorizes the shared provider account and accepts the largest member export limit and publication timeout. Cache credentials stay with the worker. Publication failure remains optional and cannot turn successful execution into failure. Existing secondary stores are left untouched; they are neither merged nor served by the shared worker.
Drain all affected offerings before deploying the account change. Systemd assigns their state directories to the new service accounts at startup; keep the per-offering paths so queue and cleanup state remain available.
Disk sizing and pressure
There is no universal minimum provider disk size: size the execution host for its installed Nix store and guest templates, retained application data, intended concurrent VM scratch disks, and bounded host transfers. Measure the filesystem holding the pool and guest disks, including retired caches and rollback copies. A coordinator using remote providers does not need this execution-host budget.
Keep these settings distinct:
| Setting | Meaning |
|---|---|
Adapter scratchMiB | Full scratch allocation charged for each active VM, not measured guest usage. |
Pool scratchBytes - reservedScratchBytes | Configured aggregate allowance for active VM scratch. |
Pool reservedScratchBytes | Headroom kept unavailable to VM admission for host services, transfers, import overhead and a safety margin. |
Cache managedCache.maxBytes | Best-effort cache size ceiling, not preallocated or guaranteed capacity. |
Cache managedCache.minFreeBytes | Free-space floor maintained by pressure eviction and checked before imports. |
Provider maxCacheExportBytes | Aggregate sender staging budget per offering process, including completed spools awaiting publication. Sharing an account does not combine these budgets. |
Pool admission checks both configured allowance and actual filesystem space: free bytes >= pool reserve + outstanding scratch charges + new job charge. Already allocated cache blocks reduce free bytes; adding the full cache ceiling to the reserve again is unnecessarily conservative when cache growth is controlled by a suitable free-space floor. Keep room for future transfers and imports instead. These checks are best-effort, not quotas on unrelated writers.
An illustrative three-offering provider can budget a 128-GiB pool reserve:
- 64 GiB of host-service and filesystem cushion;
- three 8-GiB sender budgets (24 GiB), plus one 8-GiB worker receive buffer;
- 10 GiB for the serialized cache import's admission estimate;
- 16 GiB for mandatory artifacts/results (2 GiB for each of eight configured slots);
- 6 GiB of additional margin.
The worker receives/imports one publication at a time, so count its receive buffer once per worker. Sender buffers, worker buffers, imports and guest disks can coexist. Use actual configured limits, including all providers sharing the filesystem; the example is not a default or a universal requirement. Import estimates and metadata accounting still need measured headroom.
To retain a 128-GiB active-scratch allowance in this example, configure scratchBytes = 256 GiB and reservedScratchBytes = 128 GiB. If the largest job uses 16 GiB, a cache floor of at least 144 GiB leaves admission room for one such job when the pool is otherwise idle. Increase the floor if cache growth must leave room for more concurrent work. CPU/RAM and the physical-space check may admit fewer jobs than the configured slot count. Optional cache exports also preserve the cache floor plus artifact headroom and can be refused under pressure even when a job fits; that does not promise a cache hit next time.
Pressure eviction removes eligible cache content, not host Nix paths, active VM disks, service data or backups. Active reads/imports can temporarily prevent reclamation. Inspect actual allocation, free space, pool leases and maintenance logs before adjusting limits. If the cache is already small, investigate other disk users; do not lower reserves merely to suppress warnings or run external GC against the cache worker's store. Old caches and rollback copies need an explicit cleanup decision.
Changing a durable pool budget requires draining all participants and using reconfigure-pool as described above. Preserve the intended active-scratch allowance when changing the reserve; update the cache floor at the same time. Individual service limits and shared accounts do not increase host capacity.
For the optional shared-kernel backend, see weak containerization. CLI, environment and .env precedence is unchanged. The NixOS options are weakContainerization and weakContainerizationAllowSecrets, both false by default.
Sharing with other providers
Authenticated cache sharing adds an optional dedicated read-only binary-cache/Blossom listener and provider-signed Nostr discovery. Consumers explicitly pin provider identities, Nix keys and endpoints. No job or repository configuration can modify that operator trust policy. Existing cache files are served directly; publishing them creates no second archive copy.