Execution isolation
Development preview
This page describes a pinned development revision, not a published release. Its package version may still identify the preceding release while the next version is being prepared.
This decision records the native Nix isolation design and deployment qualification status. It is not a declaration that every deployment below has passed the hardware acceptance gates. Legacy v0.1.x act compatibility is retained; its existing container deployments must not inherit the native provider's hardware-isolation claim.
Supported shape
Use a fresh hardware VM for every native job by default. An operator may explicitly select weak containerization for authorized trusted code; that shared-kernel tier is not hardware-equivalent and never an automatic fallback. A complete local Docker recipe supplies its services. See v0.1.1 migration choices. Use Firecracker on Linux with KVM, a jailer, enforced resource limits and host-owned network policy. Operators who already run Incus may use the Incus adapter in VM mode as an alternative implementation of the hardware tier; this is not a deployment qualification, and its container mode is the explicit weak tier described below. Use genuine macOS guests on Apple Silicon for Darwin/Xcode work; the backend is Tart. See the macOS deployment guide and qualification record for its tested configuration and remaining hardware-specific limits. The Apple Silicon guide records the home/dedicated preparation and deployment validation procedure. Keep the coordinator and persistent provider outside these guests. They may run on the same physical computer, under separate service identities, or on different computers. Remote providers do not require exposing a Docker socket or adding cloud credentials to the coordinator.
All repository-controlled code, evaluation, job checkout helpers, dependency builds, commands and cleanup belong in the disposable guest. Scheduling metadata is fetched by a constrained, credential-free discovery worker; it parses hostile Git/YAML data but does not run repository commands. Secrets are grants to that one job, never a reason to execute on the host. Retain the direct executor as an internal guest implementation. Do not expose it as a persistent provider backend. A missing backend is a startup/admission failure, never a direct fallback.
The hardware tier's intended threat model includes malicious guest users, guest root and a compromised guest kernel. The explicit container tier assumes trusted job code and dependencies and does not provide this boundary. Qualification must test host-enforced boundaries. Ordinary namespaces also prevent accidental interference, but a shared host kernel leaves a substantially different attack surface. Hardware virtualization is not a promise against a hypervisor exploit, host-kernel bug, firmware compromise, side channel or denial of shared physical resources. Patch and harden the host, constrain the VMM, separate identities and retain reserves. Firecracker's production guidance requires its jailer or equivalent constraints; this project requires the jailer for its production daemon instead of offering another security mode.
Coordinator location and execution location
| Deployment target | Coordinator | Native job execution | State of support |
|---|---|---|---|
| NixOS bare metal | Existing NixOS service | Local Firecracker provider under separate identities, or a remote provider | In operational use; recorded Linux acceptance and NixOS recipe |
| Other Linux bare metal | Container or standalone coordinator | Local Linux/KVM provider, preferably a small NixOS execution installation; remote provider also works | Standalone binary needs constrained launcher, cgroup delegation and firewall setup; turnkey non-NixOS installer remains work |
| Docker / Compose | Container with --act-runner none | Provider on the Linux host or another Linux/KVM machine; jobs each get a VM | Separate native-only Compose recipe; existing files preserve legacy act. Do not mount the host Docker socket into native jobs |
| Proxmox LXC | Small unprivileged LXC, optionally Compose | Provider in a separate KVM-capable VM or on a dedicated execution host | Coordinator remains usable; nested Docker inside LXC is not the new native isolation boundary |
| Linux virtual machine | Existing service or container | Firecracker if the outer hypervisor exposes working nested KVM; otherwise remote isolated provider | Test KVM and networking before advertising; no host-execution fallback |
| Apple Silicon home Mac | Usually remote coordinator | Local disposable macOS VM, one slot initially, dedicated adapter/provider accounts | Apple Silicon installation and qualification; no personal-folder, agent or keychain sharing |
| Dedicated Apple Silicon CI host | Local or remote coordinator | Same cold-clone macOS VM lifecycle, with explicit host reserves | Same backend; see the qualification limits for the tested configuration |
| Intel Mac / machine without usable virtualization | Coordinator may remain elsewhere | Approved remote Apple Silicon or Linux provider for its advertised native system | No first-class Intel macOS backend yet; do not emulate support with a Linux container or execute directly |
A local installation is therefore a supported design goal, not a requirement to rent an execution service. Docker and LXC users can keep their coordinator and attach a provider without moving secrets into job containers. For Proxmox, creating a separate NixOS VM with nested KVM is the practical initial route; turning the LXC into a privileged general-purpose hypervisor container is not. The NixOS recipe lives in deploy-nixos.md. The table describes the default hardware tier; explicit trusted-code alternatives have separate Docker and unprivileged LXC records. NixOS module availability alone does not establish weak-container deployment acceptance.
Qualification summary
Qualification applies to the recorded revision and environment, not every later commit or compatible machine. These records are the starting point for support claims; rerun relevant acceptance checks when the backend or deployment changes.
| Deployment | Recorded evidence | Remaining limits |
|---|---|---|
| Linux Firecracker | Remote acceptance, 2026-10-02, including jailed revision a4b0b2ba and recovery revision aea0169c | Historical acceptance and operational use do not qualify every later hardening change or hardware configuration. |
| Linux Incus VM | 2026-11-03 disposable lab smoke, plus fake-CLI lifecycle coverage in test-core and a real-daemon e2e harness | Production host, storage/network/firmware and guest-image acceptance remain open. Container mode is weak and never inherits the VM claim. |
| Apple Silicon macOS | Current qualification: installer/guest rehearsals and delegated release at 32f54fd5 | Abrupt power loss, network transitions, optional cache publication and broader Xcode workloads retain separate gates. |
| Native Docker containers | 2026-10-29 Compose record, introduced at 67cbe1eb | Trusted-code tier; no portable job disk quota, disk-exhaustion recovery or Docker Desktop/DinD qualification. |
| Native containers in unprivileged LXC | 2026-10-29 Proxmox record, f0e8f727 | Tested nested lab configuration; unattended host reliability and disk-exhaustion recovery remain open. |
Containers and Docker-in-Docker
A container separates namespaces, capabilities, mounts and optionally resource budgets. It still uses a kernel supplied by the environment running its daemon. A sibling container created through a mounted Docker socket is controlled by that daemon, not contained by the coordinator container. A rootful daemon can create privileged containers and mount host files. Docker documents the daemon attack surface and the need to protect its control socket.
In the default hardware-isolated tier, any native DinD integration must keep one daemon inside one disposable job VM. This is a design constraint, not qualification of a Docker-enabled guest template:
- The daemon, its data directory, container layers, network namespace and sockets all die with that job VM. No shared daemon or writable Docker data volume.
- No host socket, host paths, host PID/network namespace, provider keys, cloud metadata endpoint or other job's secrets are exposed.
- Guest privilege is allowed when the workload needs it; a privileged daemon inside the guest can compromise that guest, whose kernel is disposable.
- Host enforcement still bounds the entire VM and blocks management/private network access. Limits on individual inner containers are not the host limit.
- Cache imports are explicit data with the allocation's trust policy; restoring a previous daemon's writable state is not cache reuse.
A privileged DinD sidecar on a persistent host is not this tier. It can separate Docker daemon state from the host daemon, but its privilege and shared kernel remain outside the mandatory hardware boundary. Existing legacy act Compose configurations stay available for compatibility, with that limitation explicit.
Rootless Docker or Podman reduces the daemon's host privilege by using a user namespace. It does not remove host-kernel exposure, network access, access to that user's files, or cross-job state in a reused daemon. It is useful for local legacy act testing and inside a job VM, not sufficient by itself for the native malicious-job contract. See Docker's rootless documentation.
gVisor interposes a userspace kernel and reduces direct host-system-call exposure; it has its own compatibility and security boundaries. It is not automatically a hardware-VM provider. Kata Containers uses VMs and requires hardware or nested virtualization. Either would need a qualified ngit-ci adapter covering checkout, evaluation, control sockets, networking, lifecycle and data processing, not just a runtime name on an inner container. Neither is an additional first-class backend in this initial design.
macOS choice and validation assumptions
Tart uses Apple's Virtualization framework on Apple Silicon. Apple's macOS virtualization documentation describes genuine macOS guests; Linux containers cannot replace Apple tooling. Tart's pinned clone implementation uses APFS copy-on-write for local clones. Start from a stopped, pinned template; never clone a suspended job. Delete the entire job bundle, including disk, auxiliary/NVRAM and saved state, after every outcome. Keep the template separate from mutable job state and validate its identity before admitting secrets.
Tart's pinned exec command can carry stdin through the guest agent without SSH, on macOS 14 or newer. The prepared ngit-ci stdio guest agent can reuse the existing cancellation/artifact protocol through that channel. Disable clipboard, audio, USB accessories, host shares and unnecessary guest login services. Template provisioning must not copy a personal Apple account, keychain, SSH identity or provider configuration.
Do not assume Tart's default NAT isolates the host. Current Softnet 0.24.1 documentation requires macOS 26, initial privileged network setup and subsequent privilege dropping. Its default policy permits the gateway as well as public IPv4 destinations. The adapter must exclude the host/gateway and management endpoints explicitly and verify IPv6 handling. The @host policy target names the vmnet host address, not every public address of the physical computer. Home Wi-Fi/VPN changes and dedicated hosts' public addresses must be covered in hardware tests. A failed policy setup must prevent VM execution. These are unresolved qualification requirements, not claims that adding --net-softnet completes the design.
Start with one job slot. A reasonable planning estimate, to be measured on the supplied hardware, is 8 GiB guest RAM with at least 4 GiB additional host headroom. A dedicated quota-limited APFS volume holds the template and its copy-on-write clones. Admission checks its remaining quota plus a 16 GiB host reserve, rather than the full virtual disk size. The quota is kernel enforcement; free-space checks alone cannot contain hostile guest writes. Native quota exhaustion, clone sharing and cleanup still require hardware validation; see the storage setup and qualification. Host build outputs, provider artifacts and logs require separate budgeting. A 16 GiB home Mac may be useful for lighter jobs; 32 GiB or more is a more comfortable Xcode test target. Full macOS boot and image storage cost substantially more than the Linux appliance. Record actual cold boot, compile, peak memory, disk growth and teardown measurements before choosing defaults. APFS clone speed does not reserve physical disk capacity.
The applicable Apple macOS license permits up to two additional virtual instances for specified uses on an Apple-branded computer already running macOS, subject to its terms; commercial hosting/leasing has additional conditions. As checked on 2026-10-22, the Tart 2.41.1 source license and Softnet 0.24.1 source license are FSL-1.1-ALv2. They expressly include internal use but restrict competing commercial uses, so these versions should be described as source-available, not unrestricted FOSS. Each version receives an additional Apache-2.0 licence on the second anniversary of its availability; that does not make a newly released version Apache-licensed immediately. Redistribution requires retaining copyright notices and including the terms or a link to them, and remains subject to permitted-purpose restrictions. Downloading directly from upstream instead of bundling does not bypass those restrictions. Pin and review the license of the actual installed version. Do not rely on older Fair Source machine-count summaries. Maintainers must resolve applicability for public/commercial providers before describing that service model as supported; home/internal CI and public hosting are distinct questions.
Devices
No direct-host exception is retained for speculative GPU performance. The initial Firecracker backend advertises no GPU/device passthrough. Its upstream GPU/PCIe discussion is not a qualified ngit-ci device backend. A future dedicated VM/VFIO provider needs IOMMU isolation, device reset, driver threat analysis and job-to-job sanitization, plus accurate capabilities and admission. A macOS virtual GPU is not arbitrary physical-device passthrough; test the required Metal/Xcode workload on the actual host/guest combination. Unsupported device requirements fail selection rather than escaping to host execution.
Current boundaries and work still required
| Lifecycle operation | Current source boundary | Consequence / next gate |
|---|---|---|
| Discovery and trigger matching | src/discovery_worker.rs, src/workflow.rs, coordinator | Native Git discovery and YAML parsing use the separate-account worker; matching remains coordinator policy. Symlinks rejected; Git environment/helpers hardened, each output pipe capped at 8 MiB with a 30-second command deadline. NixOS restricts the worker filesystem view; native Compose omits coordinator credential/data mounts. See discovery boundary and qualification. |
| Checkout, Nix evaluation, build, commands and cleanup | crates/nix-provider/src/nix_job.rs, source.rs, executor.rs, guest | Keep the entire sequence inside the selected job boundary (VM by default, container only with explicit weak-mode authorization), including cleanup after cancellation. Direct nix run has no whole-job sandbox of its own. |
| Native selection and signing | src/nix_allocation.rs | Require hardware-vm by default; the explicit weak tier additionally requires repository/signer execution authorization. Recheck before signing; see authorization rules. |
| Provider startup and allocation admission | crates/nix-provider/src/main.rs, service.rs | Isolated adapter required before advertising; new direct allocations rejected. Recorded historical results may recover without new execution. |
| VMM provisioning and teardown | crates/adapter-microvm/src/firecracker_main.rs, firecracker.rs, server.rs | Daemon requires jailer/cgroups. Prove cleanup before returning capacity; quarantine uncertain cleanup. |
| Incus instance provisioning and teardown | crates/adapter-microvm/src/incus_main.rs, incus.rs, server.rs | One labelled ephemeral instance per job, launched by pinned local fingerprint. VM mode is the hardware tier; container mode is weak and requires explicit acknowledgement. Prove cleanup before returning capacity; quarantine uncertain cleanup. |
| Cache import, indexing and publication | crates/nix-provider/src/cache_worker/, cache.rs, managed_cache.rs | Separate-account IPC supplies only bounded uncompressed exports and minimal cache authority. Worker Nix helpers parse hostile archives with cache-only credentials; the provider has no local publication fallback. NixOS supplies a private filesystem view, limits and readiness ordering. See cache boundary and qualification; the optional Mac service still needs native confinement/resource checks. |
| Artifact reception, archive parsing and publication | crates/nix-provider/src/isolated.rs, crates/artifacts/ | Bounded host parsers handle hostile output. Path/link/size checks bound reception, but artifact/result parsing still occurs in the provider process with signer credentials. Never execute artifact contents on the host. |
Existing protocol declarations are assertions by a trusted provider, not remote hardware attestation. Startup validation must check the actual configured backend, and allocation admission must verify the same immutable environment and concrete resources selected by the coordinator. Do not add a trust-dependent allow-direct policy or an automatic weaker fallback.
Preserve caches as explicit scoped objects, job secrets as allocation-bound inputs, artifacts as bounded content-addressed outputs, cancellation as a controlled guest signal followed by verified teardown, and telemetry as measured resource use with missing fields reported honestly. The existing wire protocol can carry these across Linux and macOS. Backend-specific implementation and real-hardware tests establish whether those promises hold.
Explicit trusted-code container tier
The implemented weak-container tier requires explicit operator opt-in and coordinator-enforced repository/signer execution grants. Open PR submission or a Service Request alone does not authorize execution. Secret consent is separate and required on both coordinator and provider, in addition to the existing secret authorization rules. The owning guide defines standing grants, manual triggers and revocation behavior.
Jobs run in fresh sibling containers outside credential-bearing provider processes. The backend advertises its weaker isolation level; failed VM provisioning never selects it automatically. Deployment-specific qualification and limitations live in the Docker and Proxmox LXC guides. Existing NixOS module options remain available without an additional qualified recipe. The Incus adapter's container mode is another explicit weak backend: it shares the host kernel, requires --weak-containerization, and is not a fallback from its VM mode. Direct host execution remains unsupported; its executor code is retained for use inside disposable guests, not as a persistent-provider performance mode.
Controlled build cache integrity
Controlled build preparation uses a narrower integrity assumption than the host-isolation threat model: Nix's sandbox and the build guest kernel must isolate unprivileged derivations from the provider-owned controller and store registration. Workflow root never runs in that guest. Hardware isolation still protects the host, but a compromised build guest kernel can invalidate the cache-provenance claim. This first mode uses one build graph per VM, not one VM per derivation.