> ## Documentation Index
> Fetch the complete documentation index at: https://orbit.nckrtl.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# 0200 run each task group in its own sandbox vm

# ADR 0200: Run each task group in its own sandbox VM

Each task group runs in its own disposable task VM. The implementers, the reviewer, and the task check all run inside it as one user, in one environment. The security boundary is the edge of the VM, not a Unix user on a shared host. When an Orbit subtask needs a test fleet, that fleet runs nested inside the group's VM.

## Status

In progress.

Principle: [Security fits the real threat model](/docs/mission#principles). The boundary moves to a place an agent cannot cross, and the defenses that live inside it go away. This also serves [Deterministic first](/docs/mission#principles): the environment, the topology, and the lifecycle are decided by code and declarations, not by agent judgment.

## Context

Today a task group shares a node with other groups (then called Shared Instance, now [Task workspace](/docs/reference/tasks#task-workspace)). Agents run as `orbit-worker` without sudo ([One user for every task agent](/docs/reference/pi-server#one-user-for-every-task-agent)). The handoff check runs as the managed user. Measurements from 7 Oct 2026 (groups with IDs of at least 600, 219 check reports on beast):

* **The two environments disagree.** `composer check` run by agents as `orbit-worker` failed 53 of 65 times (82%). Orbit's handoff check passed 130 of 154 times. About 77 Gateway tests need `sudo setfacl` or worker ACLs. Check directories owned by the managed user with mode `0700` return `EACCES` to agents.
* **Blocked turns.** 91 of 233 blocked or assistance requests name the user or permission split. Another 45 name Incus or topology acquisition. Blocked time up to a resolution adds up to 210 h.
* **The boundary does not hold.** `orbit-worker` is in `incus-admin`, which is root-equivalent. Threads on the T3 driver run as the managed user, who has passwordless sudo.
* **Capacity.** Groups that wait for assistance or a merge keep their node slot. Todo groups pinned to a leftover `reserved` Instance wait silently on a full node.
* **The trend.** The median group took 1.5 h for ids below 200 and 5.4 h for ids from 1200.

An UpCloud experiment on 6 Oct 2026 proved the building blocks for the web lane. A VM was created, enrolled with `node:add` as an `app-dev` node in dev Cluster #4, served a DLF Instance on a private Route through the beast router, and ran a Pi coding probe through CLIProxyAPI.

A review of the first web-lane build on 9 Oct 2026 (main at `bd36565c3`) found that no implementer, check, reviewer, and publish cycle had run on either lane. Two choices caused most of the work: offline Project images, and egress limited to HTTP(S) and DNS behind a private bridge reached through host proxy, DNAT, and SNAT rules. The build had about 12.7k production lines, 26 Python programs, 93 Instance guard calls, and a 9-state model with 10 marker timestamps. A spike on beast the same day showed the alternative. A stock Ubuntu 26.04 cloud VM was ready in 31 to 48 seconds. One host forward rule, one bridge ACL, and NIC port isolation formed its whole network boundary.

A spike on beast on 12 Oct 2026 ran Orbit's E2E topology nested inside one task VM with 6 vCPU, 12 GiB, and a 60 GiB disk. Nested KVM worked without extra configuration, because Incus starts VMs with the host CPU model. The cold topology build passed in 1245 seconds, about as long as on the bare host. After the first acquire, which failed once on a readiness timeout, 5 acquires in a row passed in about 157 seconds each, against 58 on the bare host, and each release took about 2 seconds. While a topology ran, the outer VM used its whole 12 GiB on the host. Its memory balloon returns pages that the guest frees only with free page reporting on: without it, 4 GiB that the guest freed stayed on the host, and with it the host got them back within 15 seconds. Page cache is not returned. A stopped outer VM copied as a ZFS clone in 1 second, and the clone acquired a topology too. Adding one node to a topology from the plain image took 1190 seconds, because the harness converges the whole topology again.

## Decision

Orbit assigns each task group its own task VM. This amendment of 9 Oct 2026 replaced the web-lane design of the first build. The amendment of 12 Oct 2026 replaces the separate Orbit lane: the `orbit` Project uses the same task VMs as every other Project. See [Orbit Project](#orbit-project). The amendment of 13 Oct 2026 keeps every GitHub token out of task VMs; see item 5. A second amendment of 13 Oct 2026 deletes shared mode: every task group runs on a task VM, and `task_compute` and `orbit-worker` are gone; see item 7 of [Orbit Project](#orbit-project).

### Task VMs

1. **Image.** Every host launches the same base image. The Gateway builds it once from the stock Ubuntu 26.04 cloud image, on one host, every night and on demand. The build host then exports that exact image and streams it over SSH to each other host, which imports it, checks its fingerprint, and boots a smoke VM before it moves its alias. The image goes host to host over the LAN when the hosts share a Cluster, else over WireGuard, and never through the Gateway. Each host accepts each peer's transfer key only from the peer's address, with `restrict` and a forced command that imports one image into its task VM project. So a peer key gets no shell, and no host opens its Incus API to the network. An earlier `incus image copy` between the Incus APIs needed client certificate trust, an Incus listener, and firewall rules, admitted guests that a host masquerades, and once reported a failure after the image had arrived. Separate builds on beast and shark from one commit gave different images. The build boots the stock image with the task VM cloud-init plus `openssh-server`, Chromium's system libraries, and the tools that Project checks call beyond the `app-dev` role, runs the programs that enrollment runs on an `app-dev` Node, installs the `app-dev` PHP packages of every PHP version an Instance can select, installs the stable Rust toolchain for `orbit`, installs the pinned agent binary, removes the VM's identity, and publishes the disk. Task VMs launch from it as ZFS clones. Their cloud-init only creates the user `orbit` with passwordless sudo, in the group `incus-admin` for a nested topology, and the Gateway's SSH key. The image records the fingerprint of each program it ran, and enrollment skips a program only when it renders the same fingerprint. Any other program runs, with a warning, so an image older than the Gateway's code is slower but never wrong. The image also keeps work off the boot: an IPv4-only network wait and no first-boot or apt timer units. Docker stays enabled at boot, because Doctor and resumed containers need it. A missing base image fails the VM; Orbit never falls back to the stock image. Task VMs have no Project images. Only a task VM with a declared topology builds images, the E2E role images, inside itself (see [Orbit Project](#orbit-project)). Each host may keep warm VMs, as item 14 describes, and a workspace template for each Project, as item 15 describes. [ADR 0206](/docs/decisions/0206-build-task-vm-images-as-chains-of-small-jobs) proposes to build the image as a chain of small jobs that starts from the current image, and to drop the manifest. On beast the base image cut cloud-init from 33 to 21 seconds and the enrollment package programs from 31 to 6 seconds. Skipping the programs it ran, and one shared SSH connection through the jump host, cut node and role convergence from 12.7 to about 3 seconds.
2. **Enrollment.** The VM joins through the normal `node:add` path as an `app-dev` Node of the dev Cluster. Before enrollment, the Gateway reaches it with an SSH jump hop through the compute host, and reads its host key from that trusted host. After enrollment, the Gateway uses WireGuard.
3. **Host network.** Each host has one task bridge under the reserved `orbittask` prefix, with its ACL attached when the bridge is created. The ACL drops private, link-local, CGNAT, and multicast egress and allows all other egress. Ingress allows SSH from the host only. NIC port isolation is on. One persistent host rule, `ufw route allow in on orbittask+`, lets the bridge pass the host's forward policy. This replaces the root-owned helper, the per-bridge policies, the proxy, DNAT, and SNAT rules, and the HTTP(S)-only egress rule.
4. **Fleet limits.** A static nft filter on the WireGuard hub covers the reserved range `10.44.0.128/25`. It lets task VMs reach the Gateway API, Reverb, CLIProxyAPI, and DNS on the hub. It lets the Gateway reach them over SSH and Pi, and the dev Cluster router reach their previews. The range lies inside the fleet VPN subnet `10.44.0.0/24`, above the addresses that existing Nodes use. The address allocator keeps other Nodes out of the range. A task VM Node has no access grants. This replaces the hub table for each sandbox.
5. **GitHub.** No GitHub token enters the VM or a template builder. The Gateway fetches from GitHub into its own mirror of each Project and pushes to GitHub from there. Git objects cross the VM edge only as packs on the standard input and output of the Gateway's SSH commands. The Gateway takes only commit IDs and pack data from the VM, checks every object in quarantine before it joins the mirror, and pushes only the commit it approved, in a queued job so that a slow VM holds up no other group. It refuses to send any command that carries a GitHub credential to a task VM. This reverses "temporary GitHub App tokens enter the owned sandbox" for every task VM. The first build ran the credentialed fetch and push inside the VM as `orbit`, the agent's own user, so the agent was able to read the token. The amendment of 13 Oct 2026 moved them to the Gateway. See [Git on a task VM](/docs/reference/compute-drivers#git-on-a-task-vm).
6. **Pi.** The Gateway installs a pinned Pi artifact. Pi runs as a normal `pi-server` Process under `orbit`, with a token for each VM and a CLIProxyAPI key for each group.
7. **State.** The `task_vms` table holds one row for each VM. `TaskVmState` has seven cases: `provisioning`, `warm`, `ready`, `parked`, `destroying`, `destroyed`, and `failed`. There are no marker timestamps. The provider validates host output once, and `TaskVmGuest` guest output. The Gateway trusts its own rows.
8. **Jobs.** `ProvisionTaskVm`, `EnrollTaskVm`, `PrepareTaskVmRuntime`, `ParkTaskVm`, `ResumeTaskVm`, `DestroyTaskVm`, `SweepTaskVmHosts`, and `RefillTaskVmPool` run on the Horizon `task-vms` queue. Its single worker serializes VM work. A resume uses `release(2)` between readiness checks until its fixed `retryUntil()` deadline; `failed()` maps expiry to `task_vm.resume_timeout`. A release calls `horizon:terminate`, which lets active jobs finish on their original release before the new worker starts. Old queued payloads keep a compatibility reader for one release, including the removed warm-claim guard and resumes without a deadline field. New claims queue no guard.
9. **Lifecycle.** The VM is destroyed when the group ends, in this order: Instance removal, the `destroying` state, model key revocation, VM deletion, offline Node removal, and the `destroyed` state. A group that waits for review longer than `task_vms.park_after_minutes` parks its VM: Orbit stops it, and ZFS keeps its disk. No snapshot is taken. The next work for the group starts the VM and waits for the Node agent and Pi before it continues. A parked VM keeps its slot on the host. A group whose workspace removal or merge cleanup failed never parks, and a VM that was started again runs at least that long before it parks again. A sweep every 10 minutes deletes `tvm-<number>` instances in the host's project that have no live row, and fails live rows whose VM is gone on two sweeps in a row. It fails closed when the project does not exist. Doctor reports both on the host, and skips a parked VM's Node.
10. **No guard.** Generic Instance operations need no sandbox guard. One placement invariant protects task VM Nodes. Task VM Nodes are left out of the fleet rollout.
11. **Public egress.** Task VMs reach public addresses on every port. The VM edge and the hub filter are the boundary, and CI on the pushed commit is the gate.
12. **Placement.** Each host has `max_vms` slots. A new VM goes to the active host with the most free slots, and config order breaks ties. Placement never overcommits a host. SQLite serializes the transaction that counts slots and assigns or records the VM, including pool refills; no cache placement lock is needed. A `failed` VM holds no slot, so it never blocks placement: its group asks for assistance at once and gets no other VM until it is cancelled. When every host is full, the group waits with `Task VM: no capacity.`
13. **Accepted risk.** A VM can query port 53 on any host address, because Incus accepts DNS before the ACL. It can read instance names from the DNS of other bridges on that host. This risk is accepted.
14. **Warm VMs.** Each host keeps `warm_vms` task VMs (default 0) that are created, booted, and enrolled as Nodes with no group. They take slots of `max_vms`. A claim first takes the oldest warm VM that launched the host's current `orbit-task-base`, which the claim reads from the host, on the host with the most free slots among those with one. It sets the group, `provisioning`, and `claimed_at` in one update in the placement transaction, and adds a new Pi token. After commit, the claim dispatches `PrepareTaskVmRuntime` on the Horizon `task-vms` queue and returns. That job runs only the group's steps, the model key and Pi, and uploads the pinned Pi if needed. The next claim after the VM becomes `ready` creates the workspace. Each allocation attempts the same preparation dispatch when the row belongs to a claimed warm VM and is still `provisioning`. This recovers a crash between SQLite commit and Redis enqueue. Laravel's unique job lock for that VM suppresses a duplicate dispatch while the job is queued or running. A confirmed connection-resolution rejection before submission releases only the submitting job's lock owner. Once submission begins, an exception is ambiguous and retains uniqueness until the accepted job completes or fails, or the one-hour lock expires. A lock orphaned before enqueueing also expires after one hour. The preparation job finishes the row or fails it with group assistance. A warm VM holds no Pi token, model key, or workspace, and Pi does not run on it. A refill job every minute keeps each host at its target within its free slots, replaces warm VMs from an older image, destroys failed pool VMs and pool VMs that lost their job for an hour, and drains the pool while task VMs are off. After a failed pool VM, the host waits 1 minute, doubling up to 1 hour, and the log says so. Warm VMs are never parked. A hot pool of VMs saved with a stateful stop may follow.
15. **Workspace templates.** Each host keeps one workspace template for each Project whose groups run on task VMs: an Incus block volume with an ext4 filesystem that holds the Project's apps directory. It holds a checkout of the default branch after the Project's setup steps, with `vendor`, `node_modules`, and the build output, and the Playwright, Composer, and pnpm caches. A builder VM from the base image builds it when a claim finds none, and after each promote for Projects with a group in the last 7 days. Builds run on their own queue and worker, so they never delay task VM jobs. A promote deletes the templates of the old image. A build for every Project after each promote was rejected: with 26 Projects on 2 hosts it held the single task VM worker for about 48 minutes a night, for templates that were mostly never used. No agent or group runs on the builder. The setup steps may add only ignored files. When Orbit creates a group's workspace, it clones the newest template as a ZFS clone, attaches it to the running VM, mounts it at the Project's apps directory with an `/etc/fstab` line, and renames the template checkout to the workspace checkout. Then it fetches and checks out the group's branch as usual, and the baseline check's setup steps find little to do. A missing template, or a failed step, logs a warning and starts the workspace from a fresh clone, so a template only makes a claim faster. The clone goes with the VM, and the sweep deletes orphaned clones. A new template replaces older ones at once, and ZFS keeps their data while a clone uses it. Templates are automatic for every such Project, without a setting. On beast the template cut mealou's setup steps from about 36 to 13 seconds on a new VM.
16. **Baseline before the Route.** A routed workspace on a task VM starts its baseline check before its Route. Workspace creation writes the `.env` and installs the PHP packages, which the check needs, and the claim projects the Route while the check runs. The first implementer and the final review wait for the Route. A Route that keeps failing asks for assistance, and no agent starts. In group 1430 the Route took 43.5 of the 46.4 seconds before the check, 25 of them for PHP packages that the base image now holds. The baseline check of mealou, its browser tests included, passed on beast without a Route.

The first provider is Incus, on beast and shark. Shark's ZFS pool is a degraded single-disk mirror, which is acceptable for disposable VMs. UpCloud is the overflow (Phase 3), off by default: a Gateway-wide switch and a per-Project opt-in let a `vm` group start there when no Incus host has a free slot, up to a VM budget. It uses the same cloud-init and enrollment path. Its template follows the Incus base image contract: built from the stock image with the same enrollment programs, identity removed, cloud-init that installs nothing, and a smoke boot before use. The Gateway pins a host key it generates, because UpCloud has no trusted host that reads the guest's key. ADR 0204 on a nightly UpCloud base image (pull requests #1072 and #1074) was closed in favour of this path. See [UpCloud overflow](/docs/reference/compute-drivers#upcloud-overflow).

### Orbit Project

A group of the `orbit` Project runs on a task VM, exactly like a web Project's group. It gets the same base image, enrollment, workspace template, Pi, handoff check, and destroy. Its workspace is the Instance `task-<group id>` without a Route. Its setup steps install all five Composer projects and the web app with Bun, and its task check is root `composer check`, which runs `bin/review-check`. The outer VM joins the live fleet as an `app-dev` Node, like every task VM. Nothing else of the group ever does.

1. **Declared topology.** A subtask declares the test nodes it needs in its `topology` field before the group is claimed, for example `["app-dev"]` or `["app-dev", "app-prod+ingress"]`. Each entry names one node, `app-dev`, `app-prod`, or `app-prod-2`, and a group declares at most two. It may add the role `ingress` after a `+`; the harness then puts the node in its own Cluster, which the role needs. Other roles follow when a task needs one: `websocket` failed to converge on a test node in the lab. Each node is a bare machine that gets its base role and exactly the roles its entry names. Orbit never infers a topology from the work, and an agent cannot add one during a turn.
2. **VM size.** At the claim, Orbit takes the union of the `topology` fields of the group's subtasks. With none, the group gets a normal task VM: the host's size, one slot, and a warm VM when one is ready. A Project listed in `task_vms.medium_projects`, such as `orbit`, gets a medium task VM instead: 4 vCPU, 8 GiB, the host's disk, and two slots, always new. Orbit's check runs its test suites in parallel, so it needs the cores. A topology with one test node also gets a medium VM. With two test nodes, a group gets a large task VM: 6 vCPU, 12 GiB of memory, a 60 GiB disk, and three slots. The sizes come from the spike: the test Gateway, an `app-dev` node, and the operator used about 2.7 GiB, and the outer VM of a four-machine topology reached its whole 12 GiB under IO. A medium VM has a 40 GiB disk. Medium and large VMs are always new, never warm VMs, and never at UpCloud. Their memory balloon has free page reporting on, so the host gets back the memory that the guest frees. Placement counts the slots of every VM against the host's `max_vms`. The size and limits are caps, not reservations.
3. **No silent fallback.** A subtask added after the claim whose topology needs a larger VM than the group got waits with a reason that names the cause. Orbit never runs it without its nodes, and never moves the group to another VM by itself. Resizing the VM when it next starts may follow.
4. **Nested topology.** A declared topology runs inside the group's VM. The VM runs its own Incus and starts only the declared nodes, plus the test Gateway and the operator. Orbit composes them from prepared images, one for each kind of machine: Gateway, `app-dev`, `app-prod`, and operator. Each image holds the packages of its base role and no identity. Before a subtask with a topology runs its baseline check or its implementer, Orbit prepares the group's topology in the VM as `orbit`, in one systemd unit: it runs `bin/e2e-topology-host-setup`, builds the images that the declared nodes need and the VM lacks from the group's checkout, and acquires `TASK-<group>` with the union of the group's declarations. The test Gateway runs its bootstrap and enrolls the declared nodes, and the operator adds each declared extra role with `orbit node:role:add`. There is no snapshot of a whole topology and no image for each combination of roles. A topology group starts slower, about 9 minutes for the images and 3 to 5 minutes for the acquisition, and that cost is accepted for a simpler and more robust build. A failed preparation asks for assistance, and a resolution prepares again. The next subtasks of the group reuse the topology; one that declares more prepares it again. A parked VM loses its topology, and the next subtask prepares it again.
   * **Evidence.** The reviewer proves each deliverable on the declared nodes with `bin/e2e-topology exec --record`, which writes the node, the command, the exit code, and the output to the evidence log. Orbit refuses the approval of a subtask with a topology until that log holds a successful recorded command that does work on each node the subtask declares, started after the implementer handed off, and adds those commands to the approval for a person to audit.
   * **Safety net.** The Gateway setting `task_vms.topology_paths` lists, for each Project, the paths that need real machines; for Orbit, its Node, agent, guest, deployment, and routing code. A handoff check fails when the group's own changes touch one of them and no subtask of the group declares a topology. The diff runs from the merge base of HEAD and the current `origin/main` to the candidate tree, including uncommitted files. Changes inherited from main do not count; the group's own changes and merge resolutions still count. An unreadable merge base or diff fails the handoff.
5. **Isolation inside the VM.** The test Gateway and the test nodes never join the live fleet. They live on a private bridge inside the VM and reach the internet through its NAT. Rules inside the VM drop their traffic to the VM's own WireGuard interface, and from the topology bridges and the inner bridge `incusbr0` to the VM's own WireGuard address. So they cannot reach the live fleet even when their own WireGuard is down. The task VM's bridge ACL already keeps them away from the host and other private networks.
6. **Main caches.** The Project's setup step `@orbit/ci-caches` imports the test-impact graph and the Pint and PHPStan caches of the newest successful CI run on `main` into the checkout, in the workspace template and in a workspace from a fresh clone. The Gateway reads them with its GitHub App, and `bin/tia-cache import-ci` checks them in the VM. So an Orbit check runs only the affected tests, as on beast. No readable caches fail the step. A cache that does not fit the checkout leaves that project cold, with the reason in the check log.
7. **Shared mode is gone.** After Orbit groups ran on task VMs, Phase 5 deleted shared mode, the `task_compute` setting, `orbit-worker`, the `tasks.worker` Node setting, the per-Node task ceilings, and the reviewer's topology request. Every task group runs on its own task VM. A Project may still opt in to [review and merge](/docs/reference/tasks#review-and-merge); CI is the merge gate, and the VM goes when the task completes.

All items are built.

### Every task VM

There is no `orbit-worker` in a task VM. Agents, the Pi server, and the task check run as the managed user with passwordless sudo, so the privileged tests run. Orbit's handoff check stays as fast feedback and still compares tree hashes. **CI on the pushed commit is the authoritative gate.** `bin/pr-head-check` already requires it before merge.

* **Model calls** go only through CLIProxyAPI. Each group gets its own key, created at claim and revoked at the end. No subscription sign-in is stored in any sandbox.
* **The Pi token is per VM.** The Gateway-wide `ORBIT_PI_TOKEN` is not used for sandboxes.
* **Blocked destinations.** No sandbox reaches `10.44.0.0/16` outside its allowed endpoints, the LAN, the host, other groups' networks, or `169.254.169.254`.
* **Task status and VM power are separate.** After the last approval Orbit opens the PR, and the group moves to the status `waiting_for_review`.
* **Base updates** come from the GitHub update-branch API, not an agent fixup.

During the rollout, each Project had the setting `task_compute: shared | vm`, and Projects moved to `vm` once their groups were proven there. Since the amendment of 13 Oct 2026 the setting is gone: every group runs on a task VM. A group that cannot get a task VM waits with a visible reason.

This decision changes [Task workspace](/docs/reference/tasks#task-workspace) and [One user for every task agent](/docs/reference/pi-server#one-user-for-every-task-agent). The `orbit-worker` account and its `incus-admin` membership are retired with shared mode.

## Rejected alternatives

* **Keep `orbit-worker` and grant it narrow sudo.** This removes the permission-related test failures, but keeps two environments on a shared host. It does not fix `incus-admin` root equivalence or T3 agents running as the managed user.
* **Run agents as the managed user on shared hosts.** Every agent would get passwordless sudo, Docker, and the managed user's credentials. One prompt injection can reach the fleet.
* **Test nodes join the live fleet as well as the test topology.** Both networks use `10.44.0.0/16`. Mixing live and disposable traffic is already refused in [The operator guest](/docs/reference/incus-topologies#the-operator-guest).
* **A separate Orbit lane: an operator VM and a test Gateway VM on their own Incus network, outside the fleet.** It needed its own control path through Incus proxy devices and `incus exec`, its own images, its own parking with snapshots, and its own network helper on the host. A task VM with the topology nested inside it uses the one VM type, control path, and boundary that every Project already has.
* **Topology VMs on the host beside the group's task VM.** Each group would need its own host network and rules, and the host would hold four or five VMs for one group. Nested, the topology is one VM to place, park, and delete.
* **A snapshot of a whole topology, or one for each combination of roles.** Node identity, keys, and the enrolled Gateway database are baked into such a snapshot, and adding a node to one took 1190 seconds in the spike. Images for each role and enrollment at preparation compose any declared set.
* **Stateful snapshots of a running topology.** The harness attaches the worktree as a virtiofs disk, which Incus cannot migrate statefully, and restored memory carries old addresses and a clock jump. Boot time is not the cost that matters.
* **Role images baked into the workspace template, or a cache of them on each host.** The template build would need a nested build of about 9 minutes on a large builder, on both hosts after each promote, for the few groups with a topology. Its images would come from `main`, not from the group's commit. A host cache needs a build on the host and a copy path into the VM. Building in the VM costs about 9 minutes once per topology group, uses the group's own code, and needs nothing on the host.
* **Approval on the reviewer's word.** A summary can claim a proof that never ran. The evidence log is written by the harness, names the node, and holds the exit code and the output.
* **A topology on demand, requested by an agent during a turn.** The reviewer has no declaration to check the setup against, and the VM size changes while the group runs. A declaration before the claim decides the size and the nodes up front.
* **One size for every Orbit VM.** Most Orbit groups need no topology. A large VM for each would take three slots and skip the warm pool for nothing.
* **Resize a running VM, or move the group to a larger VM, when a subtask added after the claim declares more.** It changes the environment under a running group without anyone deciding it. The subtask waits with a reason instead.
* **Isolated sandboxes for every Project.** Project features need real routes, deployment, and a preview a human can open. Joining the fleet with no grants and hub-enforced limits keeps that and stays contained.
* **Copy subscription sign-ins into each sandbox.** It exposes the subscriptions, and rotating OAuth refresh tokens break when many copies refresh.
* **A token in the VM for each fetch and push.** The agent has sudo and can read any process in the VM. A read token exposes every private branch, and with the write token the agent pushes any commit it likes.
* **`git fetch` and `git push` over SSH between the Gateway mirror and the VM.** A template builder has no address of its own and would need a second transport. Packs on the existing SSH commands also keep the check that a command reaches only its group's workspace.
* **Fall back to `shared` silently when no sandbox is available.** It would mix environments again, which is the problem this decision removes.
* **WireGuard configured in cloud-init.** It bypasses normal enrollment, so the Node record, peer, and roles would not come from one path.
* **An Incus proxy device for SSH to a task VM.** Its host port does not match the Node firewall catalog, and it needs forward and DNAT rules on the host.
* **The Incus REST API with a restricted certificate, from the Gateway.** It needs more host setup, and the Gateway already has root SSH to the host. Hosts copy images to each other over SSH instead.
* **A base image build on each host.** The same commit gave different images, and each host spent about two minutes on its build.
* **`incus image export` streamed over SSH through the Gateway.** Each copy would cross the Gateway twice. A direct copy keeps the image on the LAN when it can.
* **A hub table for each VM.** One static filter on a reserved range enforces the same limits without per-VM rules on the live hub.
* **Egress limited to HTTP(S) and DNS.** The first build spent most of its acceptance work on clock sync, package sources, and key installs that this limit broke. It adds no protection beyond the VM edge.
* **The stock image on every VM.** Phase 1 used it. Each VM then spent about 40 seconds more in cloud-init and the `app-dev` package installs on beast, and each claim depended on package mirrors.
* **A base image for each Project.** Each Project would need its own image, built and kept current. A workspace template is a volume that a claim attaches to any VM, warm VMs included.
* **A virtiofs filesystem volume for the workspace template.** It hot-plugs, but the setup steps ran 10 to 18 times slower on it than on an ext4 block volume.
* **A copy of the workspace with `cp --reflink` inside the VM.** The guest root is ext4 on a ZFS volume, so it has no copy-on-write.
* **Skip enrollment programs by the image's Gateway commit.** Most commits change no program, so every release would make enrollment run them all until the next build. One fingerprint per program skips exactly the programs that did not change.
* **Bake only the PHP version of the current `vm` Projects.** A configured list needs a setting and fails silently for a Project that selects another version: its Instance installs the packages before the check, about 30 seconds. Both selectable versions cost about 72 MiB.
* **Skip the Instance's PHP step through the image manifest.** The PHP step runs at Instance creation on every Node, not at enrollment. A check of the Node's real state, the pinned key, the source file, and the installed packages, serves task VMs and long-lived Nodes alike.
* **A queued job for the Route.** The single `task-vms` worker runs enrollments and image builds one at a time, so the Route would wait minutes behind them. The claim projects the Route right after the check starts instead.
* **A copy of a finished task VM as the next image.** An agent and Project install scripts ran on that disk, so no cleanup can prove it clean.

## Consequences

* Agents get the same verdict as the gate. The largest source of blocked turns and failed local checks goes away.
* The privileged tests run inside the sandbox. The ACL, `safe.directory`, and check-directory ownership work is removed.
* A task VM uses the same enrollment, Instance, Route, check, and publication code as the shared groups did. The new code is one table, one enum, one provider, four jobs, a cloud-init renderer, two setup scripts, and SSH jump support.
* Phase 1 deletes the code of the first web-lane build that the new lane leaves unused. Phase 3 replaces the old UpCloud driver with the UpCloud task VM provider. Phase 5 deletes shared mode, `orbit-worker`, and the old Incus controller.
* An UpCloud task VM costs money while it exists, also while it is parked. The overflow is off by default, `max_vms` caps it, and the sweep deletes servers without a live row. Its template is refreshed by hand.
* Each host needs a base image before it can start task VMs. The nightly build keeps it at most a day old, and the build queues behind other task VM jobs. Every sync copies the whole image. If copy time becomes a problem, ZFS send and receive can copy only the changes.
* A parked VM's Route does not answer, and Doctor does not check its Node, until the next work starts it. `preview` keeps a VM running.
* A single worker started by the scheduler runs the jobs one at a time. A release that stops the scheduler interrupts a running job, which runs again.
* Task VM Nodes can appear in doctor while they exist.
* Warm VMs use host memory while they wait. Each one takes a slot that a new VM then cannot use.
* Each host keeps about 1 GiB for each Project's workspace template, plus the changed blocks of each clone. Template builds run one at a time on their own worker. A builder VM has the size of a task VM, is not counted in `max_vms`, and can run beside image builds and new VMs on its host. The base image holds the PHP packages of both selectable PHP versions, which adds about 72 MiB to it (2,687 to 2,759 MiB on beast) and about a minute to its build.
* A root agent can tamper with the in-sandbox check result. CI on the pushed commit stays the authoritative gate.
* A topology group needs nested virtualization on its host. Incus VMs on beast have it, because Incus starts them with the host CPU model. A large VM takes three slots and, while its topology runs, all of its 12 GiB on the host. A topology group waits about 9 minutes for its role images and 3 to 5 minutes for its acquisition before its first check. Free page reporting returns freed memory, but not the guest's page cache.
* The base image holds what Orbit's check needs beyond the `app-dev` role: `rsync`, `dig`, a C compiler, and the stable Rust toolchain. In a task VM from the base image without them, Orbit's root `composer check` failed 7 tests for the missing `rsync` and `dig`, and the Node agent did not build. They add about 1 GiB to the image: 360 MiB of packages and 640 MiB of toolchain.
* A task VM has no main cache store. Without imported caches, each Orbit check in it runs every test suite from a fresh test-impact graph: in a task VM with 2 vCPU and 4 GiB on beast, a cold root `composer check` on `main` took 2,756 seconds, 1,902 of them for the Gateway tests. The `@orbit/ci-caches` step brings the caches of CI instead, and a medium VM doubles the cores.
* An Orbit group without a topology has no test fleet. Its agents work in the checkout and the handoff check, and the reviewer cannot run Orbit commands against real Nodes.

## Affects

* Components: apps/gateway, apps/e2e, apps/cli, apps/docs
* ADRs: none in progress on main. ADR 0204 (pull requests #1072 and #1074) was closed; Phase 3 replaced it.
* Detail: [Compute drivers: Task VMs](/docs/reference/compute-drivers#task-vms). [Compute drivers: UpCloud overflow](/docs/reference/compute-drivers#upcloud-overflow). [Node provisioning: Enroll through a jump host](/docs/reference/node-provisioning#enroll-through-a-jump-host). [Tasks: Task VM workspace](/docs/reference/tasks#task-vm-workspace). [Pi server: Run Pi on a task VM](/docs/reference/pi-server#run-pi-on-a-task-vm). [Incus topology registry](/docs/reference/incus-topologies): nested topology from role images. [Projects](/docs/reference/projects): `task_compute`.
* Verify: The disposable-environment proofs check these outcomes.
  * One DLF group on beast runs claim, VM, enrollment, Route, implementer, check, reviewer, pull request, green CI, and merge. After merge, its Instance, Node, VM, and model key are gone, and `node:list` and `incus list` are clean.
  * A task VM from the host's base image enrolls without installing packages, and its stage timings are recorded.
  * With beast and shark configured, VMs go to the host with the most free slots, and a full fleet makes the group wait with `Task VM: no capacity.`
  * With `warm_vms` set, a claim takes a warm VM and the group's VM is `ready` within seconds. The pool refills, and a new base image replaces the warm VMs.
  * A routed task VM workspace starts its baseline check before its Route, and its first implementer starts only once the Route is active. A Route that keeps failing asks for assistance.
  * A group of a Project with a workspace template starts its workspace from a clone of the template, and its baseline setup steps find the dependencies in place. Without a template the workspace starts from a fresh clone, and the Gateway logs why.
  * One build on beast gives shark the image with the same fingerprint, copied over the LAN, and a failed copy leaves shark's current image in place.
  * A group that waits for review parks its VM. A review fix starts it, and the fix runs in the same workspace.
  * A task VM reaches public addresses, and cannot reach private, link-local, CGNAT, or multicast addresses, another task VM, or the fleet outside the hub filter.
  * A local check run by an agent and Orbit's handoff check give the same verdict on the same tree.
  * A one-subtask Orbit group without a topology runs end to end on a normal task VM: claim, workspace template, baseline `composer check`, implementer, handoff check, reviewer, pull request, green CI, and merge. After merge, nothing of it is left.
  * A group whose subtasks declare two test nodes gets a new large VM with three slots, 6 vCPU, 12 GiB, a 60 GiB disk, and free page reporting. One test node gets a medium VM with two slots. On a full host it waits without a VM. A subtask whose topology needs a larger VM than its group got waits with its reason.
  * An Orbit group without a topology gets a medium VM, and its baseline check runs only the tests that its change affects, from the CI caches.
  * A declared topology starts nested in the group's VM with only its declared nodes and roles. Its test nodes cannot reach the live fleet, also with their own WireGuard down.
  * The reviewer's approval of a subtask with a topology passes only with recorded evidence on each declared node. A handoff whose own changes touch a path in the Project's `task_vms.topology_paths` fails when no subtask of the group declares a topology; changes inherited from `origin/main` do not count.
  * No App private key or provider credential enters a sandbox. No GitHub token enters a task VM or a template builder, and a command that would carry one there fails with `task_vm.git_credential_refused`. GitHub tokens never appear in argv, logs, stored origins, or Git configuration.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.