Skip to main content

​
Prove the verify-only Doctor on an Incus topology

​
Problem

A proof of the verify-only Doctor runs on a leased gateway_app-dev_app-prod topology. It needs a healthy baseline on all three Nodes, exactly one declared drift, exactly one declared unverifiable condition, and evidence that Doctor writes nothing. Each fixture must produce one finding and leave every other inspection untouched. Informational tool discoveries may appear in a healthy baseline. They do not change health or exit status. Drift and unverifiable findings still fail the report. Incus proves Linux behavior. macOS enrollment, native platform and home-volume observation, and tool operations need a separate task-owned fixture on a real Mac.

​
Cause

Doctor reports one finding per inspector that fails, so a fixture must break exactly one inspector. The Node inspector also reads the free space and free inodes for root and the managed user’s home filesystem, including on the Gateway. Low disk space can therefore spoil a baseline even when the intended fixture affects another family. The production Instance inspector runs sudo bash. The role inspector runs sudo ufw. The firewall inspector also runs it when the selected Node has a persisted or synthetic firewall target. The public Route edge inspector, in the instance family, runs sudo ufw status numbered on a Node that serves a public Route edge. Doctor checks current projections only. The proof does not need to fixture local DNS snippet compatibility checks. The public Route edge inspector checks the certificate actually served by the ingress against Mozilla roots, the Route hostname, and a 30-day expiry window. A certificate finding is distinct from the proof’s declared drift and must be absent in the healthy baseline. A file inventory of the Gateway home also sees changes that are not Doctor writes. SQLite creates and removes its -wal and -shm sidecars for any connection, including a read-only one. The Caddy build check creates a lock file under the Orbit home’s locks/caddy-build/ directory, and the file stays.

​
Solution

Lease a topology, apply four fixture patterns with bin/e2e-topology exec, and record each step with --record=LABEL. The Incus topology registry describes every command. The four patterns give the proof its baseline, its drift, its unverifiable condition, and its evidence that Doctor writes nothing.

​
Lease the topology

Run bin/e2e-topology acquire ISSUE . --nodes=app-dev,app-prod from the worktree, where the branch name contains the issue. Run bin/e2e-topology status ISSUE to see the lease. Every exec below runs one argument vector as orbit on one Node. Run the Orbit CLI on gateway. An action on app-prod, which has no checkout, uses sudo or a short sh -c string. exec allows 60 seconds, so pass --timeout=SECONDS when a Doctor request needs more. Each --record appends the command, its UTC start and end times, its exit code, and its redacted output to <worktree>/.e2e/evidence.log. orbit doctor exits 0 when the report is healthy and 1 when it has a finding. The log records both. Find the numeric ID of app-prod and the production user of one production Instance on it. A composed topology has no sample workloads, so first create a development Instance on app-dev and clone it to app-prod with the Orbit CLI, and record those steps.
instance:list --json gives the production_user of each Instance. The steps below use <node-id> and <user> for these values.

​
Baseline

A leased topology is composed from role images and enrolled by its own Gateway from the mounted worktree, and acquire verified its readiness. Record one Doctor request for all registered Nodes as the healthy baseline.
The entry shows exit 0 and no findings. When it shows node.disk_low, check the affected Node with df --output=source,avail,size,iavail,itotal -k -- / "$HOME" as its managed user. Doctor does not free space; remove only fixtures owned by this lease, or use a fresh topology if the low-space files are not yours. Do not use a role converge to try to clear disk usage. For projection drift, converge the affected role with orbit node:role:add NODE ROLE --converge, and record the baseline again before you apply a fixture. An active systemd status alone does not prove that a Process is healthy. Doctor reports process.crash_loop for a Process desired running when it observes auto-restart or a growing restart count during its bounded inspection. If the baseline has this finding, record its active state, sub-state, and restart counts before changing a task-owned fixture. A stable count left by earlier restarts is not drift. The Process runtime state reference owns these rules. Restore the fixture to healthy operation and record a clean baseline before the mutation scans; Doctor does not restart it for you. For a nested Laravel fixture, configure the web root explicitly, for example apps/site/public, before the baseline. Verify that its environment and cached APP_URL live in apps/site, PHP-FPM uses the application directory, and the production release’s apps/site/.env links to the durable home file. Run the same healthy, drift, and restored Doctor checks without adding repository-root environment files to satisfy an incorrect probe. Git identity checks still use the whole checkout. Keep the fixture within the task’s lease.

​
Mutation scan, before

Record the Orbit home inventory, the table row counts, and the service states after the baseline and before the first fixture. The inventory excludes the SQLite sidecars and the Caddy build lock files.
Record the same service list on gateway and app-dev, each with its own label. For a repository-probe proof, use only a disposable development checkout owned by the lease. Save its Git configuration before planting a hook or custom filesystem monitor that writes a marker. Install the fixture before the mutation scan, then run Doctor’s project and instance families and verify that the marker is absent. Restore the saved configuration and remove the planted files before releasing the topology. Doctor’s repository probes pass -c core.hooksPath=/dev/null and -c core.fsmonitor=false. They must ignore those planted callbacks without changing the stored configuration. A read-only Git verb alone is not evidence that a checkout-selected program cannot write files.

​
Drift

Doctor checks the PHP-FPM projection only for production Instances. It compares each generated file byte for byte with the expected content. So edit the pool file /etc/orbit/php-fpm/<user>/generated/pool.conf of the production Instance on app-prod. Change the line listen.mode = 0660 to listen.mode = 0666.
Doctor reports exactly one instance.php_fpm_projection_mismatch. Restore 0660 with a second sed, and record the next report, which is clean.
Do not edit pm.max_children. That line is in local.conf, and Doctor checks only the structure of local.conf, not its values. The running master reads the pool file only at its next reload, so the edit changes no running worker.

​
Unverifiable condition

Add a sudoers drop-in on app-prod that keeps NOPASSWD: ALL for the orbit user and denies /usr/sbin/ufw. The command checks the file with visudo before it installs it, because a broken drop-in stops every sudo call.
The role request reports exactly one role.inspection_failed. A full-family request isolates the same failure only when the selected Node has no persisted firewall rules, no synthetic Metrics firewall targets, and no public Route edge. Remove the file and record the restored report.

​
Mutation scan, after

Record the inventory, the row counts, and the three service lists again, with labels that end in after. Compare each pair of entries in the evidence log. Only activity_log grows, by one audit row per Doctor request between the two scans.

​
Release

Copy the evidence log entries that you need, then run bin/e2e-topology release ISSUE.

​
Limits

These fixtures depend on these properties of the harness and the Nodes.
  • acquire enrolls the Nodes with the Gateway code of the worktree at that moment. A later edit that changes an expected projection shows drift in the baseline until you converge the role.
  • The Gateway runs Doctor from the mounted worktree.
  • A converge rewrites the files Orbit owns on the Node, including the pool file. Converge before the first scan, never during a fixture.
  • Denying one sudo command works because sudoers applies the last matching entry. So the drop-in name must sort after /etc/sudoers.d/orbit, which holds Orbit’s grant.
  • The drop-in name must not contain a dot, because sudo skips such a file.
  • Denying bash breaks more than one inspector. The production Instance, private Route, public Route edge, custom proxy Route, Schedule, and Caddy build inspectors all run sudo bash.
  • Every exec holds the issue lock, so the steps run in order, and the evidence log keeps that order.

​
Verification

<worktree>/.e2e/evidence.log holds every labelled entry in order. doctor baseline and both restored reports show exit 0 and no findings. doctor app-prod drift shows exit 1 and exactly one instance.php_fpm_projection_mismatch. doctor app-prod unverifiable shows exit 1 and exactly one role.inspection_failed. The before and after scan entries differ only in the activity_log row count.