Skip to main content

​
ADR 0200: Run each task group in its own sandbox VM

Each task group runs in its own disposable task VM. The implementers, the reviewer, and the task check all run inside it as one user, in one environment. The security boundary is the edge of the VM, not a Unix user on a shared host. When an Orbit subtask needs a test fleet, that fleet runs nested inside the group’s VM.

​
Status

In progress. Principle: Security fits the real threat model. The boundary moves to a place an agent cannot cross, and the defenses that live inside it go away. This also serves Deterministic first: the environment, the topology, and the lifecycle are decided by code and declarations, not by agent judgment.

​
Context

Today a task group shares a node with other groups (then called Shared Instance, now Task workspace). Agents run as orbit-worker without sudo (One user for every task agent). The handoff check runs as the managed user. Measurements from 7 Oct 2026 (groups with IDs of at least 600, 219 check reports on beast):
  • The two environments disagree. composer check run by agents as orbit-worker failed 53 of 65 times (82%). Orbit’s handoff check passed 130 of 154 times. About 77 Gateway tests need sudo setfacl or worker ACLs. Check directories owned by the managed user with mode 0700 return EACCES to agents.
  • Blocked turns. 91 of 233 blocked or assistance requests name the user or permission split. Another 45 name Incus or topology acquisition. Blocked time up to a resolution adds up to 210 h.
  • The boundary does not hold. orbit-worker is in incus-admin, which is root-equivalent. Threads on the T3 driver run as the managed user, who has passwordless sudo.
  • Capacity. Groups that wait for assistance or a merge keep their node slot. Todo groups pinned to a leftover reserved Instance wait silently on a full node.
  • The trend. The median group took 1.5 h for ids below 200 and 5.4 h for ids from 1200.
An UpCloud experiment on 6 Oct 2026 proved the building blocks for the web lane. A VM was created, enrolled with node:add as an app-dev node in dev Cluster #4, served a DLF Instance on a private Route through the beast router, and ran a Pi coding probe through CLIProxyAPI. A review of the first web-lane build on 9 Oct 2026 (main at bd36565c3) found that no implementer, check, reviewer, and publish cycle had run on either lane. Two choices caused most of the work: offline Project images, and egress limited to HTTP(S) and DNS behind a private bridge reached through host proxy, DNAT, and SNAT rules. The build had about 12.7k production lines, 26 Python programs, 93 Instance guard calls, and a 9-state model with 10 marker timestamps. A spike on beast the same day showed the alternative. A stock Ubuntu 26.04 cloud VM was ready in 31 to 48 seconds. One host forward rule, one bridge ACL, and NIC port isolation formed its whole network boundary. A spike on beast on 12 Oct 2026 ran Orbit’s E2E topology nested inside one task VM with 6 vCPU, 12 GiB, and a 60 GiB disk. Nested KVM worked without extra configuration, because Incus starts VMs with the host CPU model. The cold topology build passed in 1245 seconds, about as long as on the bare host. After the first acquire, which failed once on a readiness timeout, 5 acquires in a row passed in about 157 seconds each, against 58 on the bare host, and each release took about 2 seconds. While a topology ran, the outer VM used its whole 12 GiB on the host. Its memory balloon returns pages that the guest frees only with free page reporting on: without it, 4 GiB that the guest freed stayed on the host, and with it the host got them back within 15 seconds. Page cache is not returned. A stopped outer VM copied as a ZFS clone in 1 second, and the clone acquired a topology too. Adding one node to a topology from the plain image took 1190 seconds, because the harness converges the whole topology again.

​
Decision

Orbit assigns each task group its own task VM. This amendment of 9 Oct 2026 replaced the web-lane design of the first build. The amendment of 12 Oct 2026 replaces the separate Orbit lane: the orbit Project uses the same task VMs as every other Project. See Orbit Project. The amendment of 13 Oct 2026 keeps every GitHub token out of task VMs; see item 5. A second amendment of 13 Oct 2026 deletes shared mode: every task group runs on a task VM, and task_compute and orbit-worker are gone; see item 7 of Orbit Project.

​
Task VMs

  1. Image. Every host launches the same base image. The Gateway builds it once from the stock Ubuntu 26.04 cloud image, on one host, every night and on demand. The build host then exports that exact image and streams it over SSH to each other host, which imports it, checks its fingerprint, and boots a smoke VM before it moves its alias. The image goes host to host over the LAN when the hosts share a Cluster, else over WireGuard, and never through the Gateway. Each host accepts each peer’s transfer key only from the peer’s address, with restrict and a forced command that imports one image into its task VM project. So a peer key gets no shell, and no host opens its Incus API to the network. An earlier incus image copy between the Incus APIs needed client certificate trust, an Incus listener, and firewall rules, admitted guests that a host masquerades, and once reported a failure after the image had arrived. Separate builds on beast and shark from one commit gave different images. The build boots the stock image with the task VM cloud-init plus openssh-server, Chromium’s system libraries, and the tools that Project checks call beyond the app-dev role, runs the programs that enrollment runs on an app-dev Node, installs the app-dev PHP packages of every PHP version an Instance can select, installs the stable Rust toolchain for orbit, installs the pinned agent binary, removes the VM’s identity, and publishes the disk. Task VMs launch from it as ZFS clones. Their cloud-init only creates the user orbit with passwordless sudo, in the group incus-admin for a nested topology, and the Gateway’s SSH key. The image records the fingerprint of each program it ran, and enrollment skips a program only when it renders the same fingerprint. Any other program runs, with a warning, so an image older than the Gateway’s code is slower but never wrong. The image also keeps work off the boot: an IPv4-only network wait and no first-boot or apt timer units. Docker stays enabled at boot, because Doctor and resumed containers need it. A missing base image fails the VM; Orbit never falls back to the stock image. Task VMs have no Project images. Only a task VM with a declared topology builds images, the E2E role images, inside itself (see Orbit Project). Each host may keep warm VMs, as item 14 describes, and a workspace template for each Project, as item 15 describes. ADR 0206 proposes to build the image as a chain of small jobs that starts from the current image, and to drop the manifest. On beast the base image cut cloud-init from 33 to 21 seconds and the enrollment package programs from 31 to 6 seconds. Skipping the programs it ran, and one shared SSH connection through the jump host, cut node and role convergence from 12.7 to about 3 seconds.
  2. Enrollment. The VM joins through the normal node:add path as an app-dev Node of the dev Cluster. Before enrollment, the Gateway reaches it with an SSH jump hop through the compute host, and reads its host key from that trusted host. After enrollment, the Gateway uses WireGuard.
  3. Host network. Each host has one task bridge under the reserved orbittask prefix, with its ACL attached when the bridge is created. The ACL drops private, link-local, CGNAT, and multicast egress and allows all other egress. Ingress allows SSH from the host only. NIC port isolation is on. One persistent host rule, ufw route allow in on orbittask+, lets the bridge pass the host’s forward policy. This replaces the root-owned helper, the per-bridge policies, the proxy, DNAT, and SNAT rules, and the HTTP(S)-only egress rule.
  4. Fleet limits. A static nft filter on the WireGuard hub covers the reserved range 10.44.0.128/25. It lets task VMs reach the Gateway API, Reverb, CLIProxyAPI, and DNS on the hub. It lets the Gateway reach them over SSH and Pi, and the dev Cluster router reach their previews. The range lies inside the fleet VPN subnet 10.44.0.0/24, above the addresses that existing Nodes use. The address allocator keeps other Nodes out of the range. A task VM Node has no access grants. This replaces the hub table for each sandbox.
  5. GitHub. No GitHub token enters the VM or a template builder. The Gateway fetches from GitHub into its own mirror of each Project and pushes to GitHub from there. Git objects cross the VM edge only as packs on the standard input and output of the Gateway’s SSH commands. The Gateway takes only commit IDs and pack data from the VM, checks every object in quarantine before it joins the mirror, and pushes only the commit it approved, in a queued job so that a slow VM holds up no other group. It refuses to send any command that carries a GitHub credential to a task VM. This reverses “temporary GitHub App tokens enter the owned sandbox” for every task VM. The first build ran the credentialed fetch and push inside the VM as orbit, the agent’s own user, so the agent was able to read the token. The amendment of 13 Oct 2026 moved them to the Gateway. See Git on a task VM.
  6. Pi. The Gateway installs a pinned Pi artifact. Pi runs as a normal pi-server Process under orbit, with a token for each VM and a CLIProxyAPI key for each group.
  7. State. The task_vms table holds one row for each VM. TaskVmState has seven cases: provisioning, warm, ready, parked, destroying, destroyed, and failed. There are no marker timestamps. The provider validates host output once, and TaskVmGuest guest output. The Gateway trusts its own rows.
  8. Jobs. ProvisionTaskVm, EnrollTaskVm, PrepareTaskVmRuntime, ParkTaskVm, ResumeTaskVm, DestroyTaskVm, SweepTaskVmHosts, and RefillTaskVmPool run on the Horizon task-vms queue. Its single worker serializes VM work. A resume uses release(2) between readiness checks until its fixed retryUntil() deadline; failed() maps expiry to task_vm.resume_timeout. A release calls horizon:terminate, which lets active jobs finish on their original release before the new worker starts. Old queued payloads keep a compatibility reader for one release, including the removed warm-claim guard and resumes without a deadline field. New claims queue no guard.
  9. Lifecycle. The VM is destroyed when the group ends, in this order: Instance removal, the destroying state, model key revocation, VM deletion, offline Node removal, and the destroyed state. A group that waits for review longer than task_vms.park_after_minutes parks its VM: Orbit stops it, and ZFS keeps its disk. No snapshot is taken. The next work for the group starts the VM and waits for the Node agent and Pi before it continues. A parked VM keeps its slot on the host. A group whose workspace removal or merge cleanup failed never parks, and a VM that was started again runs at least that long before it parks again. A sweep every 10 minutes deletes tvm-<number> instances in the host’s project that have no live row, and fails live rows whose VM is gone on two sweeps in a row. It fails closed when the project does not exist. Doctor reports both on the host, and skips a parked VM’s Node.
  10. No guard. Generic Instance operations need no sandbox guard. One placement invariant protects task VM Nodes. Task VM Nodes are left out of the fleet rollout.
  11. Public egress. Task VMs reach public addresses on every port. The VM edge and the hub filter are the boundary, and CI on the pushed commit is the gate.
  12. Placement. Each host has max_vms slots. A new VM goes to the active host with the most free slots, and config order breaks ties. Placement never overcommits a host. SQLite serializes the transaction that counts slots and assigns or records the VM, including pool refills; no cache placement lock is needed. A failed VM holds no slot, so it never blocks placement: its group asks for assistance at once and gets no other VM until it is cancelled. When every host is full, the group waits with Task VM: no capacity.
  13. Accepted risk. A VM can query port 53 on any host address, because Incus accepts DNS before the ACL. It can read instance names from the DNS of other bridges on that host. This risk is accepted.
  14. Warm VMs. Each host keeps warm_vms task VMs (default 0) that are created, booted, and enrolled as Nodes with no group. They take slots of max_vms. A claim first takes the oldest warm VM that launched the host’s current orbit-task-base, which the claim reads from the host, on the host with the most free slots among those with one. It sets the group, provisioning, and claimed_at in one update in the placement transaction, and adds a new Pi token. After commit, the claim dispatches PrepareTaskVmRuntime on the Horizon task-vms queue and returns. That job runs only the group’s steps, the model key and Pi, and uploads the pinned Pi if needed. The next claim after the VM becomes ready creates the workspace. Each allocation attempts the same preparation dispatch when the row belongs to a claimed warm VM and is still provisioning. This recovers a crash between SQLite commit and Redis enqueue. Laravel’s unique job lock for that VM suppresses a duplicate dispatch while the job is queued or running. A confirmed connection-resolution rejection before submission releases only the submitting job’s lock owner. Once submission begins, an exception is ambiguous and retains uniqueness until the accepted job completes or fails, or the one-hour lock expires. A lock orphaned before enqueueing also expires after one hour. The preparation job finishes the row or fails it with group assistance. A warm VM holds no Pi token, model key, or workspace, and Pi does not run on it. A refill job every minute keeps each host at its target within its free slots, replaces warm VMs from an older image, destroys failed pool VMs and pool VMs that lost their job for an hour, and drains the pool while task VMs are off. After a failed pool VM, the host waits 1 minute, doubling up to 1 hour, and the log says so. Warm VMs are never parked. A hot pool of VMs saved with a stateful stop may follow.
  15. Workspace templates. Each host keeps one workspace template for each Project whose groups run on task VMs: an Incus block volume with an ext4 filesystem that holds the Project’s apps directory. It holds a checkout of the default branch after the Project’s setup steps, with vendor, node_modules, and the build output, and the Playwright, Composer, and pnpm caches. A builder VM from the base image builds it when a claim finds none, and after each promote for Projects with a group in the last 7 days. Builds run on their own queue and worker, so they never delay task VM jobs. A promote deletes the templates of the old image. A build for every Project after each promote was rejected: with 26 Projects on 2 hosts it held the single task VM worker for about 48 minutes a night, for templates that were mostly never used. No agent or group runs on the builder. The setup steps may add only ignored files. When Orbit creates a group’s workspace, it clones the newest template as a ZFS clone, attaches it to the running VM, mounts it at the Project’s apps directory with an /etc/fstab line, and renames the template checkout to the workspace checkout. Then it fetches and checks out the group’s branch as usual, and the baseline check’s setup steps find little to do. A missing template, or a failed step, logs a warning and starts the workspace from a fresh clone, so a template only makes a claim faster. The clone goes with the VM, and the sweep deletes orphaned clones. A new template replaces older ones at once, and ZFS keeps their data while a clone uses it. Templates are automatic for every such Project, without a setting. On beast the template cut mealou’s setup steps from about 36 to 13 seconds on a new VM.
  16. Baseline before the Route. A routed workspace on a task VM starts its baseline check before its Route. Workspace creation writes the .env and installs the PHP packages, which the check needs, and the claim projects the Route while the check runs. The first implementer and the final review wait for the Route. A Route that keeps failing asks for assistance, and no agent starts. In group 1430 the Route took 43.5 of the 46.4 seconds before the check, 25 of them for PHP packages that the base image now holds. The baseline check of mealou, its browser tests included, passed on beast without a Route.
The first provider is Incus, on beast and shark. Shark’s ZFS pool is a degraded single-disk mirror, which is acceptable for disposable VMs. UpCloud is the overflow (Phase 3), off by default: a Gateway-wide switch and a per-Project opt-in let a vm group start there when no Incus host has a free slot, up to a VM budget. It uses the same cloud-init and enrollment path. Its template follows the Incus base image contract: built from the stock image with the same enrollment programs, identity removed, cloud-init that installs nothing, and a smoke boot before use. The Gateway pins a host key it generates, because UpCloud has no trusted host that reads the guest’s key. ADR 0204 on a nightly UpCloud base image (pull requests #1072 and #1074) was closed in favour of this path. See UpCloud overflow.

​
Orbit Project

A group of the orbit Project runs on a task VM, exactly like a web Project’s group. It gets the same base image, enrollment, workspace template, Pi, handoff check, and destroy. Its workspace is the Instance task-<group id> without a Route. Its setup steps install all five Composer projects and the web app with Bun, and its task check is root composer check, which runs bin/review-check. The outer VM joins the live fleet as an app-dev Node, like every task VM. Nothing else of the group ever does.
  1. Declared topology. A subtask declares the test nodes it needs in its topology field before the group is claimed, for example ["app-dev"] or ["app-dev", "app-prod+ingress"]. Each entry names one node, app-dev, app-prod, or app-prod-2, and a group declares at most two. It may add the role ingress after a +; the harness then puts the node in its own Cluster, which the role needs. Other roles follow when a task needs one: websocket failed to converge on a test node in the lab. Each node is a bare machine that gets its base role and exactly the roles its entry names. Orbit never infers a topology from the work, and an agent cannot add one during a turn.
  2. VM size. At the claim, Orbit takes the union of the topology fields of the group’s subtasks. With none, the group gets a normal task VM: the host’s size, one slot, and a warm VM when one is ready. A Project listed in task_vms.medium_projects, such as orbit, gets a medium task VM instead: 4 vCPU, 8 GiB, the host’s disk, and two slots, always new. Orbit’s check runs its test suites in parallel, so it needs the cores. A topology with one test node also gets a medium VM. With two test nodes, a group gets a large task VM: 6 vCPU, 12 GiB of memory, a 60 GiB disk, and three slots. The sizes come from the spike: the test Gateway, an app-dev node, and the operator used about 2.7 GiB, and the outer VM of a four-machine topology reached its whole 12 GiB under IO. A medium VM has a 40 GiB disk. Medium and large VMs are always new, never warm VMs, and never at UpCloud. Their memory balloon has free page reporting on, so the host gets back the memory that the guest frees. Placement counts the slots of every VM against the host’s max_vms. The size and limits are caps, not reservations.
  3. No silent fallback. A subtask added after the claim whose topology needs a larger VM than the group got waits with a reason that names the cause. Orbit never runs it without its nodes, and never moves the group to another VM by itself. Resizing the VM when it next starts may follow.
  4. Nested topology. A declared topology runs inside the group’s VM. The VM runs its own Incus and starts only the declared nodes, plus the test Gateway and the operator. Orbit composes them from prepared images, one for each kind of machine: Gateway, app-dev, app-prod, and operator. Each image holds the packages of its base role and no identity. Before a subtask with a topology runs its baseline check or its implementer, Orbit prepares the group’s topology in the VM as orbit, in one systemd unit: it runs bin/e2e-topology-host-setup, builds the images that the declared nodes need and the VM lacks from the group’s checkout, and acquires TASK-<group> with the union of the group’s declarations. The test Gateway runs its bootstrap and enrolls the declared nodes, and the operator adds each declared extra role with orbit node:role:add. There is no snapshot of a whole topology and no image for each combination of roles. A topology group starts slower, about 9 minutes for the images and 3 to 5 minutes for the acquisition, and that cost is accepted for a simpler and more robust build. A failed preparation asks for assistance, and a resolution prepares again. The next subtasks of the group reuse the topology; one that declares more prepares it again. A parked VM loses its topology, and the next subtask prepares it again.
    • Evidence. The reviewer proves each deliverable on the declared nodes with bin/e2e-topology exec --record, which writes the node, the command, the exit code, and the output to the evidence log. Orbit refuses the approval of a subtask with a topology until that log holds a successful recorded command that does work on each node the subtask declares, started after the implementer handed off, and adds those commands to the approval for a person to audit.
    • Safety net. The Gateway setting task_vms.topology_paths lists, for each Project, the paths that need real machines; for Orbit, its Node, agent, guest, deployment, and routing code. A handoff check fails when the subtask’s diff touches one of them and no subtask of the group declares a topology.
  5. Isolation inside the VM. The test Gateway and the test nodes never join the live fleet. They live on a private bridge inside the VM and reach the internet through its NAT. Rules inside the VM drop their traffic to the VM’s own WireGuard interface, and from the topology bridges and the inner bridge incusbr0 to the VM’s own WireGuard address. So they cannot reach the live fleet even when their own WireGuard is down. The task VM’s bridge ACL already keeps them away from the host and other private networks.
  6. Main caches. The Project’s setup step @orbit/ci-caches imports the test-impact graph and the Pint and PHPStan caches of the newest successful CI run on main into the checkout, in the workspace template and in a workspace from a fresh clone. The Gateway reads them with its GitHub App, and bin/tia-cache import-ci checks them in the VM. So an Orbit check runs only the affected tests, as on beast. No readable caches fail the step. A cache that does not fit the checkout leaves that project cold, with the reason in the check log.
  7. Shared mode is gone. After Orbit groups ran on task VMs, Phase 5 deleted shared mode, the task_compute setting, orbit-worker, the tasks.worker Node setting, the per-Node task ceilings, and the reviewer’s topology request. Every task group runs on its own task VM. A Project may still opt in to review and merge; CI is the merge gate, and the VM goes when the task completes.
All items are built.

​
Every task VM

There is no orbit-worker in a task VM. Agents, the Pi server, and the task check run as the managed user with passwordless sudo, so the privileged tests run. Orbit’s handoff check stays as fast feedback and still compares tree hashes. CI on the pushed commit is the authoritative gate. bin/pr-head-check already requires it before merge.
  • Model calls go only through CLIProxyAPI. Each group gets its own key, created at claim and revoked at the end. No subscription sign-in is stored in any sandbox.
  • The Pi token is per VM. The Gateway-wide ORBIT_PI_TOKEN is not used for sandboxes.
  • Blocked destinations. No sandbox reaches 10.44.0.0/16 outside its allowed endpoints, the LAN, the host, other groups’ networks, or 169.254.169.254.
  • Task status and VM power are separate. After the last approval Orbit opens the PR, and the group moves to the status waiting_for_review.
  • Base updates come from the GitHub update-branch API, not an agent fixup.
During the rollout, each Project had the setting task_compute: shared | vm, and Projects moved to vm once their groups were proven there. Since the amendment of 13 Oct 2026 the setting is gone: every group runs on a task VM. A group that cannot get a task VM waits with a visible reason. This decision changes Task workspace and One user for every task agent. The orbit-worker account and its incus-admin membership are retired with shared mode.

​
Rejected alternatives

  • Keep orbit-worker and grant it narrow sudo. This removes the permission-related test failures, but keeps two environments on a shared host. It does not fix incus-admin root equivalence or T3 agents running as the managed user.
  • Run agents as the managed user on shared hosts. Every agent would get passwordless sudo, Docker, and the managed user’s credentials. One prompt injection can reach the fleet.
  • Test nodes join the live fleet as well as the test topology. Both networks use 10.44.0.0/16. Mixing live and disposable traffic is already refused in The operator guest.
  • A separate Orbit lane: an operator VM and a test Gateway VM on their own Incus network, outside the fleet. It needed its own control path through Incus proxy devices and incus exec, its own images, its own parking with snapshots, and its own network helper on the host. A task VM with the topology nested inside it uses the one VM type, control path, and boundary that every Project already has.
  • Topology VMs on the host beside the group’s task VM. Each group would need its own host network and rules, and the host would hold four or five VMs for one group. Nested, the topology is one VM to place, park, and delete.
  • A snapshot of a whole topology, or one for each combination of roles. Node identity, keys, and the enrolled Gateway database are baked into such a snapshot, and adding a node to one took 1190 seconds in the spike. Images for each role and enrollment at preparation compose any declared set.
  • Stateful snapshots of a running topology. The harness attaches the worktree as a virtiofs disk, which Incus cannot migrate statefully, and restored memory carries old addresses and a clock jump. Boot time is not the cost that matters.
  • Role images baked into the workspace template, or a cache of them on each host. The template build would need a nested build of about 9 minutes on a large builder, on both hosts after each promote, for the few groups with a topology. Its images would come from main, not from the group’s commit. A host cache needs a build on the host and a copy path into the VM. Building in the VM costs about 9 minutes once per topology group, uses the group’s own code, and needs nothing on the host.
  • Approval on the reviewer’s word. A summary can claim a proof that never ran. The evidence log is written by the harness, names the node, and holds the exit code and the output.
  • A topology on demand, requested by an agent during a turn. The reviewer has no declaration to check the setup against, and the VM size changes while the group runs. A declaration before the claim decides the size and the nodes up front.
  • One size for every Orbit VM. Most Orbit groups need no topology. A large VM for each would take three slots and skip the warm pool for nothing.
  • Resize a running VM, or move the group to a larger VM, when a subtask added after the claim declares more. It changes the environment under a running group without anyone deciding it. The subtask waits with a reason instead.
  • Isolated sandboxes for every Project. Project features need real routes, deployment, and a preview a human can open. Joining the fleet with no grants and hub-enforced limits keeps that and stays contained.
  • Copy subscription sign-ins into each sandbox. It exposes the subscriptions, and rotating OAuth refresh tokens break when many copies refresh.
  • A token in the VM for each fetch and push. The agent has sudo and can read any process in the VM. A read token exposes every private branch, and with the write token the agent pushes any commit it likes.
  • git fetch and git push over SSH between the Gateway mirror and the VM. A template builder has no address of its own and would need a second transport. Packs on the existing SSH commands also keep the check that a command reaches only its group’s workspace.
  • Fall back to shared silently when no sandbox is available. It would mix environments again, which is the problem this decision removes.
  • WireGuard configured in cloud-init. It bypasses normal enrollment, so the Node record, peer, and roles would not come from one path.
  • An Incus proxy device for SSH to a task VM. Its host port does not match the Node firewall catalog, and it needs forward and DNAT rules on the host.
  • The Incus REST API with a restricted certificate, from the Gateway. It needs more host setup, and the Gateway already has root SSH to the host. Hosts copy images to each other over SSH instead.
  • A base image build on each host. The same commit gave different images, and each host spent about two minutes on its build.
  • incus image export streamed over SSH through the Gateway. Each copy would cross the Gateway twice. A direct copy keeps the image on the LAN when it can.
  • A hub table for each VM. One static filter on a reserved range enforces the same limits without per-VM rules on the live hub.
  • Egress limited to HTTP(S) and DNS. The first build spent most of its acceptance work on clock sync, package sources, and key installs that this limit broke. It adds no protection beyond the VM edge.
  • The stock image on every VM. Phase 1 used it. Each VM then spent about 40 seconds more in cloud-init and the app-dev package installs on beast, and each claim depended on package mirrors.
  • A base image for each Project. Each Project would need its own image, built and kept current. A workspace template is a volume that a claim attaches to any VM, warm VMs included.
  • A virtiofs filesystem volume for the workspace template. It hot-plugs, but the setup steps ran 10 to 18 times slower on it than on an ext4 block volume.
  • A copy of the workspace with cp --reflink inside the VM. The guest root is ext4 on a ZFS volume, so it has no copy-on-write.
  • Skip enrollment programs by the image’s Gateway commit. Most commits change no program, so every release would make enrollment run them all until the next build. One fingerprint per program skips exactly the programs that did not change.
  • Bake only the PHP version of the current vm Projects. A configured list needs a setting and fails silently for a Project that selects another version: its Instance installs the packages before the check, about 30 seconds. Both selectable versions cost about 72 MiB.
  • Skip the Instance’s PHP step through the image manifest. The PHP step runs at Instance creation on every Node, not at enrollment. A check of the Node’s real state, the pinned key, the source file, and the installed packages, serves task VMs and long-lived Nodes alike.
  • A queued job for the Route. The single task-vms worker runs enrollments and image builds one at a time, so the Route would wait minutes behind them. The claim projects the Route right after the check starts instead.
  • A copy of a finished task VM as the next image. An agent and Project install scripts ran on that disk, so no cleanup can prove it clean.

​
Consequences

  • Agents get the same verdict as the gate. The largest source of blocked turns and failed local checks goes away.
  • The privileged tests run inside the sandbox. The ACL, safe.directory, and check-directory ownership work is removed.
  • A task VM uses the same enrollment, Instance, Route, check, and publication code as the shared groups did. The new code is one table, one enum, one provider, four jobs, a cloud-init renderer, two setup scripts, and SSH jump support.
  • Phase 1 deletes the code of the first web-lane build that the new lane leaves unused. Phase 3 replaces the old UpCloud driver with the UpCloud task VM provider. Phase 5 deletes shared mode, orbit-worker, and the old Incus controller.
  • An UpCloud task VM costs money while it exists, also while it is parked. The overflow is off by default, max_vms caps it, and the sweep deletes servers without a live row. Its template is refreshed by hand.
  • Each host needs a base image before it can start task VMs. The nightly build keeps it at most a day old, and the build queues behind other task VM jobs. Every sync copies the whole image. If copy time becomes a problem, ZFS send and receive can copy only the changes.
  • A parked VM’s Route does not answer, and Doctor does not check its Node, until the next work starts it. preview keeps a VM running.
  • A single worker started by the scheduler runs the jobs one at a time. A release that stops the scheduler interrupts a running job, which runs again.
  • Task VM Nodes can appear in doctor while they exist.
  • Warm VMs use host memory while they wait. Each one takes a slot that a new VM then cannot use.
  • Each host keeps about 1 GiB for each Project’s workspace template, plus the changed blocks of each clone. Template builds run one at a time on their own worker. A builder VM has the size of a task VM, is not counted in max_vms, and can run beside image builds and new VMs on its host. The base image holds the PHP packages of both selectable PHP versions, which adds about 72 MiB to it (2,687 to 2,759 MiB on beast) and about a minute to its build.
  • A root agent can tamper with the in-sandbox check result. CI on the pushed commit stays the authoritative gate.
  • A topology group needs nested virtualization on its host. Incus VMs on beast have it, because Incus starts them with the host CPU model. A large VM takes three slots and, while its topology runs, all of its 12 GiB on the host. A topology group waits about 9 minutes for its role images and 3 to 5 minutes for its acquisition before its first check. Free page reporting returns freed memory, but not the guest’s page cache.
  • The base image holds what Orbit’s check needs beyond the app-dev role: rsync, dig, a C compiler, and the stable Rust toolchain. In a task VM from the base image without them, Orbit’s root composer check failed 7 tests for the missing rsync and dig, and the Node agent did not build. They add about 1 GiB to the image: 360 MiB of packages and 640 MiB of toolchain.
  • A task VM has no main cache store. Without imported caches, each Orbit check in it runs every test suite from a fresh test-impact graph: in a task VM with 2 vCPU and 4 GiB on beast, a cold root composer check on main took 2,756 seconds, 1,902 of them for the Gateway tests. The @orbit/ci-caches step brings the caches of CI instead, and a medium VM doubles the cores.
  • An Orbit group without a topology has no test fleet. Its agents work in the checkout and the handoff check, and the reviewer cannot run Orbit commands against real Nodes.

​
Affects

  • Components: apps/gateway, apps/e2e, apps/cli, apps/docs
  • ADRs: none in progress on main. ADR 0204 (pull requests #1072 and #1074) was closed; Phase 3 replaced it.
  • Detail: Compute drivers: Task VMs. Compute drivers: UpCloud overflow. Node provisioning: Enroll through a jump host. Tasks: Task VM workspace. Pi server: Run Pi on a task VM. Incus topology registry: nested topology from role images. Projects: task_compute.
  • Verify: The disposable-environment proofs check these outcomes.
    • One DLF group on beast runs claim, VM, enrollment, Route, implementer, check, reviewer, pull request, green CI, and merge. After merge, its Instance, Node, VM, and model key are gone, and node:list and incus list are clean.
    • A task VM from the host’s base image enrolls without installing packages, and its stage timings are recorded.
    • With beast and shark configured, VMs go to the host with the most free slots, and a full fleet makes the group wait with Task VM: no capacity.
    • With warm_vms set, a claim takes a warm VM and the group’s VM is ready within seconds. The pool refills, and a new base image replaces the warm VMs.
    • A routed task VM workspace starts its baseline check before its Route, and its first implementer starts only once the Route is active. A Route that keeps failing asks for assistance.
    • A group of a Project with a workspace template starts its workspace from a clone of the template, and its baseline setup steps find the dependencies in place. Without a template the workspace starts from a fresh clone, and the Gateway logs why.
    • One build on beast gives shark the image with the same fingerprint, copied over the LAN, and a failed copy leaves shark’s current image in place.
    • A group that waits for review parks its VM. A review fix starts it, and the fix runs in the same workspace.
    • A task VM reaches public addresses, and cannot reach private, link-local, CGNAT, or multicast addresses, another task VM, or the fleet outside the hub filter.
    • A local check run by an agent and Orbit’s handoff check give the same verdict on the same tree.
    • A one-subtask Orbit group without a topology runs end to end on a normal task VM: claim, workspace template, baseline composer check, implementer, handoff check, reviewer, pull request, green CI, and merge. After merge, nothing of it is left.
    • A group whose subtasks declare two test nodes gets a new large VM with three slots, 6 vCPU, 12 GiB, a 60 GiB disk, and free page reporting. One test node gets a medium VM with two slots. On a full host it waits without a VM. A subtask whose topology needs a larger VM than its group got waits with its reason.
    • An Orbit group without a topology gets a medium VM, and its baseline check runs only the tests that its change affects, from the CI caches.
    • A declared topology starts nested in the group’s VM with only its declared nodes and roles. Its test nodes cannot reach the live fleet, also with their own WireGuard down.
    • The reviewer’s approval of a subtask with a topology passes only with recorded evidence on each declared node. A handoff that changes a path in the Project’s task_vms.topology_paths fails when no subtask of the group declares a topology.
    • No App private key or provider credential enters a sandbox. No GitHub token enters a task VM or a template builder, and a command that would carry one there fails with task_vm.git_credential_refused. GitHub tokens never appear in argv, logs, stored origins, or Git configuration.