Node provisioning
orbit node:add <name> [host] adds a machine to the fleet. For a Node that already exists, the same command converges it again after you change its TLD, roles, or settings. orbit node:remove <node> removes the Node from Orbit and leaves the machine reachable over public SSH. node lists the options.
Add a Node
A new Node needs a public SSH host and an approved SSH host key fingerprint,--host-key-fingerprint. Without the fingerprint, the Gateway refuses with node.ssh_host_fingerprint_required. A Node with app-dev needs a TLD, unless it joins an active Cluster that has one. Otherwise the Gateway refuses with node.tld_required. Platforms are linux for Ubuntu 26.04 service Nodes and macos for selected tools. macOS Nodes use a separate enrollment path.
For Ubuntu, the Gateway records the Node as provisioning and runs these steps in order:
A task VM’s enrollment trusts its base image for the
app-dev software. It keeps the VM-specific host key, WireGuard, firewall, Caddy, DNS, agent and secret, and Pi steps. Every other Node runs every program at enrollment.
After the agent, the Gateway installs the Orbit CLI on a Node of the fleet rollout set. A role converge does the same after its agent converge. Both are best effort: an unpublished CLI release or a failed install only logs a warning, and the fleet catch-up installs the CLI later.
The Gateway console command orbit:node-provision runs the same steps for the first Node. Role firewall state comes from the shared Node firewall rule catalog, keeping initial provisioning and later role reconciliation consistent.
Bootstrap identity
The Gateway picks the bootstrap user from the Node record, unless the request names one.
Name a bootstrap user with
--user when root cannot log in, or when the managed user cannot log in. --orbit-user names the managed user. A new Node uses orbit, and an existing Node keeps its recorded user. The API fields are user and orbit_user. The API refuses an empty or null user.
Machine architecture
After it connects as the managed user, the Gateway reads the architecture withuname -m, such as x86_64 or aarch64. A Node without a recorded architecture stores it. The value stays even when a later step fails. An existing Node keeps its recorded value. An explicit --architecture for a new Node must equal the observed value. A mismatch fails with node.architecture_mismatch (HTTP 409) at step machine-architecture, before any role converges.
Failure codes
Each failure names the check or step that stopped the request.macOS Nodes
A macOS Node supports selected Homebrew formulae, casks, and Vite+ global packages. It needs no role. It uses an existing account, a working WireGuard connection, and SSH access authorized for the Gateway. Enrollment pins the approved SSH host identity and verifies the actual platform, architecture, account, and tunnel address before it succeeds. The Gateway resolves the host name. An SSH alias on the caller’s machine is not a fleet address. Usenode:add with --platform=macos, the approved host fingerprint, and --user and --orbit-user set to the same existing account. The Gateway does not default those fields to root or orbit. A missing field returns node.macos_account_required. Different names return node.macos_account_mismatch. A name that fails the existing portable user check returns node.invalid_linux_user. All three are HTTP 422 and change nothing.
The request must name a WireGuard address that is already on the machine, or the Node row must already have one. The Gateway does not allocate a new address, install a tunnel, or rewrite the host tunnel, DNS, or firewall. A missing address returns node.wireguard_required. A requested address that differs from the address stored on the Node returns node.wireguard_required and changes nothing. A role, DNS server override, WireGuard endpoint override, storage settings change, Cluster change, TLD change, or LAN change returns node.platform_unsupported and changes nothing.
An eligible existing peer can be enrolled in place. It already has a row, no roles, a WireGuard address, and no pinned SSH fingerprint. After the platform, architecture, account, and tunnel checks pass, the Gateway stores the observed platform and uname -m architecture and the named account. It keeps the Node ID, name, access grants, WireGuard keys, address, and endpoint. Apple silicon stays arm64. The Gateway does not rewrite arm64 to aarch64.
A requested platform or architecture that disagrees with the machine fails. node.platform_mismatch and node.architecture_mismatch are HTTP 409 at machine-architecture. An already managed Node has a pinned SSH fingerprint. Enrollment does not change its platform, architecture, or user. A disagreement fails and leaves the record as it was. A different account on that Node returns node.user_change_unsupported (HTTP 409).
The macOS path creates no account and applies no Ubuntu bootstrap, apt packages, UFW rules, systemd units, or managed DNS. It keeps the existing account, tunnel, Homebrew prefix, and Vite+ global scope. Discovery and adoption report a missing or conflicting manager without installing, replacing, or repinning it. Tool mutations require an active Node with verified WireGuard and pinned SSH identity.
Enrollment steps
The Gateway does not record a new Node until enrollment succeeds or a remote check fails. A refusal before SSH creates no row. It runs these steps and does not run Ubuntu steps 2 to 10.uname -s must be Darwin. A Linux host requested as macOS fails with node.platform_mismatch before any account, package, DNS, or firewall change.
Enrollment recovery
A failed check changes nothing on the Mac. Retry with the samenode:add. The retry does not run the Ubuntu bootstrap. The Gateway writes known_hosts before the registry pin. A known_hosts entry with no registry pin is harmless, and the retry completes it. The registry never stores an active pin without those entries.
An already managed Node that fails stays as it was. Storage settings, Metrics reconcile, and the Node agent do not run, so a macOS failure never stops on steps 8 to 10.
Unsupported operations
These operations refuse macOS before remote mutation and before they write a role or other new assignment.
Doctor checks lifecycle, SSH reachability, platform, architecture, tunnel identity, tools, and free space on the enrolled account’s home volume. It does not expect systemd, a Node agent, an exporter, or a Linux disk path. A missing agent is not
node.agent_missing. Realtime and the web page treat the Mac as a Node with no agent, not as a failed Linux service.
Removal
Removing a macOS Node deletes the registry row and the hub peer. It does not delete the user, uninstall Homebrew or Vite+, or edit the host tunnel. It skips the Node agent step andfirewall-recovery. retained_on_node lists user, package-managers, and host-wireguard. A failed hub or registry step uses the normal removal rollback: the Node returns to its previous status, and a removed peer is restored when that rollback succeeds. macOS OS updates, firewall management, application hosting, and a macOS agent are separate features.
JSON and the human node:remove output both report that list, so the operator can see the account, package managers, and host tunnel that remain.
Nodes without roles
node:add without --role adds a Node that hosts no Orbit service. Use it for an operator machine that runs the Orbit CLI over WireGuard.
On Ubuntu, the Gateway uses this setup when the request names no role and the Node has no role assignment. macOS uses the macOS enrollment path instead of the steps below. Ubuntu runs the same steps as for any Linux Node, with these differences.
When the package sources do not resolve, the bootstrap fails. Unlike a Node with roles, it does not clear Orbit DNS on the
orbit link first. See Add the machine again.
The DNS = line is in /etc/wireguard/orbit.conf, and wg-quick applies it when the tunnel starts. The Gateway deletes an existing /etc/wireguard/orbit.dns-link.
A later node:add without --role keeps this setup. To call the Gateway API from the Node, grant it access with node:access:add. The Gateway identifies the caller by its WireGuard address.
On Ubuntu, node:role:add accepts the same roles as on any other Linux Node. Adding the first role closes public SSH and moves the Node to managed DNS in the same operation. macOS refuses every role with node.platform_unsupported before SSH.
orbit:node-dns-repair still refuses a Node without roles with node.dns_repair_operator_owned.
Node retarget keeps a Node without roles in operator DNS mode. The DNS = line stays, and the Node does not get an orbit.dns-link.
Doctor checks a Node without roles like any other Node when the Gateway has a pinned SSH host key for it. The role family reports nothing. The schedule family skips its orphan scan unless the Node hosts a Schedule. A Node without roles and without a pinned SSH host key gets only the lifecycle check.
Enroll through a jump host
The Gateway can enroll a Node that it reaches only through another Node. Task VMs use this: a task VM has only a private address on its Incus host’s bridge. The jump is internal.node:add and the API have no field for it.
The Node records the jump Node in ssh_jump_node_id. While the Node has no active role, SSH to its public address goes through an ssh hop to the jump Node, as <managed user>@<WireGuard address>:22 of the jump Node. The hop uses the same key, pinned host keys, and strict options as the command. See An explicit hop, not ProxyJump. The host key scan runs ssh-keyscan on the jump Node, and the scanned key must match the expected fingerprint. A task VM’s fingerprint comes from its host, before any code in the VM runs.
Commands through a jump Node share one connection too, once the Node’s host key is pinned. Nodes behind different jump Nodes can have the same private address, and a new Node can reuse an address. So that connection is named after the jump Node, the target, and the target’s pinned host key fingerprint. Before the key is pinned, each command opens its own connection. A jump Node without a WireGuard address fails with vpn.peer_address_missing before SSH.
The enrollment steps are the same as for any Node. When the first active role closes public SSH, the Gateway reaches the Node over WireGuard and ignores the jump. A Node without a jump Node connects directly, as SSH connections describes.
Converge an existing Node
node:add for a recorded Node converges the machine again. It refuses a Node that owns Instances with node.has_instances. One exception: it changes only the TLD of a Node with an active app-dev role.
What a failure leaves depends on the step. A macOS Node uses enrollment recovery instead of this table.
A storage-settings failure returns its
node.settings_* code, and the request skips steps 9 and 10. Only an unexpected error in step 8 marks the Node failed at node-storage-root.
Step 9 runs whenever a Metrics role exists, except for a task VM, which never hosts an exporter. A task VM also holds back the reconcile that each role converge runs. A Metrics reconcile failure does not fail provisioning or demote an active Node, including the gateway Node. The Node stays active; Metrics reports the failure as degraded. This applies to exporter, cAdvisor, runtime, and unexpected failures, both when node:add provisions a Node and when it converges an existing Node. See Metrics role for how exporter state and degradation are reported.
Orbit CLI
Every Node of the fleet rollout set runs the Orbit CLI, so the fleet rollout can runorbit self-update there. Provisioning, role converge, and the fleet rollout install it with the same step.
This is the layout
orbit self-update keeps, so either one can update what the other installed. The profile holds no secret. The Gateway identifies the Node by its WireGuard address, as for every CLI call, and its certificate covers that address. sudo orbit self-update reads root’s profile.
The step reads the link path first, as the managed user, never as root. A link may name only an orbit-0.N.0 file beside it, without / or ... The file must be a root-owned ELF executable. Only then does the step run it with --version, which must print Orbit 0.N.0 or, for a pre-release build, Orbit <40-character commit>.
The step reads the path again under the Node’s update lock before it installs, because a self-update may have changed it meanwhile. The download, the move, and the link switch run under that lock,
/run/lock/orbit-self-update.lock, which orbit self-update holds too (Node agent). A checksum mismatch deletes the candidate and fails with cli.checksum_mismatch; the link stays as it was. Then the step writes root’s profile when it differs.
Converge the Orbit footprint
orbit node:converge <node> [--force] re-applies what the Gateway renders for a Node, under the Node’s role lock. The fleet rollout runs the same step on each Node. It never changes an Instance, a Process unit, or a role, and it never runs node:add, so it works on a Node that owns Instances.
Each artifact has a digest that the Gateway computes from its own code and pins, without SSH. The Caddy package digest covers the package program and the
CaddyRelease floor. The Caddy digest covers every Gateway source file the Caddy build renders from: the build, its site sources, and the classes they use, such as DevelopmentSite and the CaddyRelease pin. The other digests cover the private-DNS listener release and publication code, the agent pin and the inputs its unit renders from, the collector script, the annotator files, and the tmpfiles rule.
A user’s sites, Routes, and DNS records never change a digest. Their own operations publish them, and Doctor reports their drift. The route-residue digest is the exception: it covers the residues of offline Route removals, which are Orbit’s own unfinished work, so the Node drifts until a converge removes them. The Gateway keeps the digests each Node last received, and a converge re-applies only the artifacts whose digest changed. It takes the Node’s update lock over SSH first. So even a converge that changes nothing runs a few lock commands on the Node.
Removing a registered artifact changes the aggregate footprint digest. Converge invokes only artifacts that are still registered and apply to the Node. Unless forced, it skips each remaining artifact whose own digest is unchanged. After a successful converge, the Gateway drops the removed name from the stored artifact metadata. Removal itself does not delete or rewrite any Node files; files left by the removed artifact stay untouched.
--force re-applies every artifact; each one still leaves a matching live copy alone. A Caddyfile that the render or the validation refuses is skipped, not failed; the fleet rollout fails a refused validation on its first Node. A failed artifact fails the command with its own error code and details.artifact, and the next converge applies it again. The digest of all artifacts is the Node’s footprint digest, which the fleet catch-up compares.
Metrics exporters, cAdvisor, and the FPM exporter are not part of the footprint: their converge always restarts them, and the Metrics fleet reconcile owns them. Reverb and the third-party pins keep their own update paths.
Role compatibility
Some roles never share a Node. The Gateway refuses a conflicting role before it claims or converges anything.router, metrics, and websocket share a Node with any role. node:role:add and node:role:relocate answer validation.failed with Role [ROLE] conflicts with assigned role [OTHER]. node:add answers node.role_conflict (HTTP 422), with Role [ROLE] conflicts with requested role [OTHER]. when one request names both roles. Doctor reports each assignment of a conflicting pair as role.assignment_conflict.
Not every role can be added the same way.
Package sources
A role installs its packages from the Ubuntu archive, except PHP and Caddy.
For PHP, the Gateway downloads the signing key and refuses it unless it matches a pinned SHA-256 digest and a pinned fingerprint. It writes the keyring and a deb822 source file as
root:root mode 0644, and restores the earlier pair when a later step fails. It refreshes only the Sury source, so another source that fails to fetch cannot fail the PHP step. It refuses a package candidate from any other origin. Orbit never uses apt-key or add-apt-repository.
When the keyring already holds the pinned key, by its SHA-256 digest and fingerprints, the source file is the one Orbit writes, and every package is installed, the step installs nothing. It then skips the key download, the source refresh, and the candidate checks, which guard only what apt installs, and logs PHP packages are in place; skipped the package source refresh. It still refuses a conflicting PHP source and unsafe source files first.
The step never upgrades an installed package on its own. Installing a missing package can upgrade packages it depends on: each Sury extension depends on the exact version of php<version>-common, so apt also upgrades php<version>-common and the installed extensions that depend on it. The shortcut applies only when nothing is missing, so it never meets that case. After such an upgrade, each package is still install ok installed, and the version step still checks that every required module loads in the CLI and in PHP-FPM.
For Caddy, the Gateway pins a release and the SHA-512 digest of its .deb for amd64 and arm64. It downloads the package only when Caddy is missing or below the floor, refuses a download whose digest does not match, and installs it with apt. A Node on another architecture fails the step. The step adds no apt source. It deletes /etc/apt/sources.list.d/orbit-caddy.sources and /usr/share/keyrings/orbit-caddy.gpg, which earlier Orbit releases wrote for the Caddy apt source on Cloudsmith.
Caddy must be at least 2.9.0. A lower release fails the caddy-package-source step and names both releases. Doctor reports it as role.caddy_version_unsupported. Converging a role upgrades an archive Caddy in place. A Caddy at or above the floor stays as it is, so a newer pin does not restart it. /etc/caddy/Caddyfile is a symlink into Orbit’s own versions directory, so the upgrade keeps the live configuration.
The roles gateway, router, ingress, app-dev, app-prod, websocket, and analytics install Caddy when they converge. ProxyCli publication does the same on its Node. On the Gateway machine, the bootstrap and php artisan orbit:gateway-web install Caddy through local sudo. A failure there stops at step gateway-caddy-install with gateway.caddy_install_failed, and the live Caddy configuration stays unchanged. Caddy configuration describes how the Gateway builds each Node’s Caddyfile.
CaddyPackageProgram::command() supplies the same fixed arguments and shell program to role convergence, the fleet footprint, Gateway installation, and Analytics and ProxyCli publication. The rename from CaddyPackageSourceProgram changes the digest of the caddy-package footprint once, so the next footprint converge repeats this idempotent step. Its host command stays byte-for-byte the same. A Caddy at or above the floor is not reinstalled or restarted. The Gateway ordering step can fail with gateway.caddy_start_failed if its WireGuard ordering drop-in cannot be installed.
Caddy reloads
The Caddy step also writes/etc/sysctl.d/60-orbit-caddy.conf, owned by root:root with mode 0644:
sysctl --load first, so a kernel that refuses the setting fails the step and leaves no file. It applies the setting on every run, so a changed live value returns to 1. It refuses a live file that is a symlink, not a regular file, or not root:root mode 0644. Doctor does not check the setting. Converge the role to repair it.
HTTP/1.1 clients can still get an empty reply on a new connection during a reload. The CLI and the PHP SDK use HTTP/1.1. The SDK retries a GET or HEAD request once after 250 milliseconds when the connection fails without a response. A request that changes state is never retried, so it can fail while the Gateway’s Caddy reloads. Run it again. Browsers and curl use HTTP/2 and do not see this.
Caddy on the Gateway machine
Converging thegateway role runs the Caddy step over SSH before its firewall step. A failure returns node_role.convergence_failed for step caddy-package-source, and the role becomes failed with error code gateway.caddy_install_failed. Run orbit node:role:add <node> gateway --converge to repair it. While the role is failed, php artisan orbit:gateway-web refuses with gateway.web_node_missing, because it needs an active gateway role.
Caddy also serves the Gateway API. A convergence that installs a newer Caddy restarts it, so the CLI can report Could not reach the gateway. while the Gateway finishes. orbit activity:list shows the real result. Run the command again after a minute.
Role operations on one Node
The Gateway runs one role operation per Node at a time. The role lock covers the role’s baseline convergence or removal, the Tool Manager setup forapp-dev and app-prod, and the Node agent converge that follows.
node:role:add, node:role:remove, and the role steps of node:add take the lock before they claim the assignment. A second operation waits up to 2 minutes. Then it fails and leaves the assignment as it was.
Every failed role operation returns its step’s own code in
details.error_code, such as node_role.tool_manager_locked, a metrics.* code, or ingress.caddy_config_failed for a Caddy removal. node:add and node:remove do the same when a role step failed.
node:role:relocate and Cluster Router changes take the lock only around the baseline step. The Metrics fleet reconcile runs inside the lock for node:role:add, node:role:remove, and node:add. It converges exporters on other Nodes without their locks, so two operations can never wait on each other. Operations on different Nodes run in parallel. An offline role removal runs outside the lock, because it changes nothing on the unreachable machine.
An operation that dies, such as a killed Gateway worker, leaves its role provisioning or removing. The claim is stale once it is 11 minutes old and no operation holds the lock. That is the role lock’s request term plus a 1-minute margin. node:role:add --converge and node:role:remove then take it over, and Doctor reports role.claim_stale.
Per-Node locks
Four locks guard work on one Node. They live in a file cache store underORBIT_HOME, whatever CACHE_STORE says.
A Tool operation takes its Tool lock, then its manager lock. Several manager locks are taken in the order
apt, vp, composer, brew. brew-cask takes the brew lock rather than a fifth lock. A role operation on app-dev or app-prod takes the vp and composer manager locks first, and then the role lock. The Node agent lock comes last. The locks cannot deadlock: the Tool and manager locks never wait, and no code takes the role lock while it holds the agent lock.
Lock renewal
The 10-minute term matches the Gateway’s PHP-FPM request limit. In an Artisan command, such asorbit:node-provision, the Tool, manager, and role locks have a 20-minute term. Each remote command times out after 15 minutes by default.
Before each command that a process runs, on the Node or on the Gateway, the Gateway renews every lock that the process holds for its full term. So a long operation keeps its locks, and a process that dies blocks the Node for at most one term after its last command started. A renewal fails only when the lock has expired. The command then does not run and fails with node.lock_lost, and so does every later command of the operation. A role operation that lost its lock stops, reports details.error_code node.lock_lost, and leaves the role failed with that code.
SSH connections
The Gateway runs every remote command as a channel on one shared OpenSSH connection per Node. The first command opens the connection. Later commands reuse it, and it closes after 60 idle seconds. A command on a shared connection takes about 18 ms, and a command on a new connection about 190 ms. The sockets live inORBIT_HOME/ssh/mux, and the directory has mode 0700. Every Gateway process runs as the orbit user. So web requests, the scheduler, and commands share the same connections. Each socket name hashes the user, the Node address, and the port. A Node with a new address or port therefore gets a new connection at once.
- A dead Node ends its connection within about 10 seconds, through
ServerAliveInterval=5andServerAliveCountMax=2. - A key or host-key change on the Node applies when the connection closes.
- When the socket path is too long for a Unix socket, or the directory cannot be created, each command opens its own connection.
- When a Node refuses another channel, OpenSSH opens a direct connection for that command and writes two warning lines to its stderr.
--offline probe of role and Node removal, and the Node probe of task cancellation use it. File copies between Nodes for Instance transfer and clone use scp on their own connections.
A Node that the Gateway reaches through a jump Node shares a connection named after the jump Node, the Node’s address, and its pinned host key.
Public SSH
The bootstrap adds the UFW ruleorbit:public-ssh-recovery and enables UFW. An UpCloud task VM boots with both from its template, so the bootstrap finds them in place. Once SSH answers over WireGuard, the Gateway adds orbit:wireguard-members and keeps public SSH open. The first active role removes the public SSH rule, so the Gateway then reaches the Node only over WireGuard.
- A Node without an active role stays reachable over public SSH.
- A later
node:addof such a Node connects over public SSH again. It finishes the WireGuard peer over the verified tunnel. - Removing the last role restores
orbit:public-ssh-recoveryover WireGuard before the Gateway deletes the assignment. - When that restore fails, the role stays
failedat stepremove:firewall-recoverywithnode.firewall_recovery_failed. A retry repeats the removal. - Removing a role while another role remains keeps public SSH closed.
- Node retarget picks its path from the same boundary.
WireGuard hub
The Node with thevpn role holds the WireGuard hub: its private key, address, listen port, and the list of peers. When a peer changes, the Gateway renders the hub configuration and installs it there. node:wireguard-key sets or replaces an unmanaged Node’s public key and runs this projection without configuring the peer machine. It refuses managed Nodes, Gateway Nodes, and keys that collide with another Node or the hub, including concurrent assignments. A successful projection preserves non-target peers.
Repeating the same key still projects the hub when it has no collision; after a projection failure, the stored key remains and the operator retries after repairing the hub. A crash can leave either the old or new hub configuration, without guaranteed rollback or automatic reconciliation. WireGuard key recovery explains how to restore an authorized management path and replay the desired key.
When the vpn and gateway roles share a machine, or no gateway role is active, the Gateway installs the configuration locally with sudo. When they are on different Nodes, the Gateway sends the configuration over SSH to the vpn Node, with the private key on standard input, and activates wg-quick@orbit there. Without SSH, the projection fails with vpn.server_config_install_failed and never installs the configuration in /etc/wireguard on the Gateway machine. The Gateway always keeps the rendered hub configuration, private key included, at ORBIT_HOME/generated/wireguard/orbit.conf.
Remove a Node
orbit node:remove <node> [--offline] [--force] removes the Node record and its Gateway configuration. It does not clean the machine. macOS removal also skips the Linux agent and firewall steps and reports the retained user, package managers, and host tunnel.
The Gateway refuses the removal until the Node is empty. Remove its Instances, Routes, Schedules, roles, Processes, Database servers, and Orbit firewall rules first. Orbit never removes the caller’s Node, the gateway Node, or the vpn Node.
When task VM hosts are configured, the Gateway then removes the Node’s base image transfer key from each of them, also for an offline removal, and also when
task-vms:retire-host did not run. It skips the Node of a task VM. This changes only the hosts, and a repeat finds nothing to remove. When it fails, the removal fails with the step task-vm-image-keys and the code node.task_vm_image_key_cleanup_failed, and nothing else has changed.
Online removal runs these steps in order.
A failed step returns the Node record to the status it had, restores the Metrics selection, and restores the WireGuard peer when the Gateway removed it. So a
failed Node stays failed. When a rollback itself fails, the Gateway returns node.removal_rollback_failed with the step wireguard-rollback, persistence-rollback, or metrics-exporters-rollback.
Offline removal
Use--offline only for a Node that the Gateway cannot reach. The Gateway probes the Node first. A Node that answers keeps every guard above, but the removal still skips firewall-recovery. So omit --offline for a Node that is up. A reachable Linux removal returns an empty retained_on_node. The Gateway still removes the Node agent, and public SSH stays as it was because firewall recovery did not run.
The Node of a task VM is the exception: DestroyTaskVm deletes the VM first and removes the Node without a probe, and without a Metrics pass, because a task VM Node never hosts an exporter. See Destroy a task VM.
For a Node that does not answer, the API needs force, or the Gateway refuses with node.confirmation_required. The CLI sends it after --force or a yes at the prompt. The guards for Instances, Routes, Schedules, protected Nodes, and firewall rules still apply. The Gateway then removes every role on its own side, deletes the Node’s Process records, removes the WireGuard peer, and deletes the record. It changes nothing on the machine. Caddy sites, checkouts, containers, Process units, Orbit UFW rules, the Metrics exporter, and the Node agent stay in place, and public SSH stays closed.
retained_on_node lists the Node agent, the Metrics exporter, and each shed role’s leftovers. follow_up says those leftovers stay until you clear them. macOS does not use that list. macOS removal always reports user, package-managers, and host-wireguard.
Add the machine again
After an online removal, the machine keeps its WireGuard tunnel, which can still point DNS at Orbit. When a new bootstrap cannot resolve the package sources for that reason, and the request names at least one role and no--dns-server override, it clears only the DNS server and route-all domain that match Orbit’s ownership record on the orbit link.
The bootstrap then uses the machine’s own DNS for the base packages and restores the earlier link values. It keeps operator overrides, refuses missing or changed ownership state, and changes no global resolver file or IP route. When the machine’s own DNS cannot resolve the sources either, the bootstrap fails before it sets up the managed user. A Node added again without a role, or with a DNS server override, keeps its DNS as it is, and the bootstrap fails when the sources do not resolve.
Provisioning then writes a new tunnel configuration and restarts wg-quick@orbit. When the Node sends private DNS through WireGuard, the configuration has a PostUp hook that points the link at Orbit DNS. It never has a PreDown hook, because the AppArmor profile that Ubuntu 26.04 ships for wg-quick denies the resolver revert call.
Why it works this way
These reasons explain the design. Check them before you propose a change.Removal does not clean the machine
node:remove is a registry and VPN change, plus one firewall rule. Cleaning the machine during removal would change it while the command says it does not, and a failed cleanup would leave a half-removed Node. So you remove Processes, roles, and Instances with their own commands first. node:add covers both provisioning and convergence, so the pair is add and remove. Create and destroy stay reserved for machines at a hosting provider.
One shared SSH connection per Node
Converges and removals run long chains of commands, and a new connection costs about ten times the command. A persistent SSH tunnel is rejected, because WireGuard already gives the private network. A higherMaxSessions on every Node is rejected, because OpenSSH already falls back to a direct connection. A reachability check cannot use the shared connection, because that connection outlives a stopped sshd and would report a Node as reachable.
An explicit hop, not ProxyJump
OpenSSH’sProxyJump does not pass the identity file, the known-hosts file, or the strict host-key options to the jump hop. The Gateway keeps its key and pinned host keys in ORBIT_HOME/ssh, so that hop would not find them. An explicit ProxyCommand hop passes them.
Public SSH before the peer goes
Role convergence closes public SSH. Without the recovery rule, a removed machine is reachable only through its provider console. So the Gateway reopens public SSH while the tunnel still works, and then removes the peer.A jump host for private Nodes
A Node on a private bridge has no public SSH address. SSH through its host needs no proxy device or DNAT rule on the host, and the Node keeps SSH on port 22, as the firewall catalog expects. Reading the host key from the host, not from the new Node, keeps the trust anchor outside the machine being enrolled.The hub stays on the vpn Node
Whengateway runs on another machine, that machine is itself a WireGuard peer. Installing the hub key there would overwrite its peer configuration and cut the Gateway off the network. So the hub configuration always goes to the vpn Node, and a missing SSH path fails instead of falling back to a local install. When both roles share a machine, a local install avoids SSH to itself.
A kernel setting for Caddy reloads
Caddy’sgrace_period and shutdown_delay, a reload through the admin API, and a certificate cache that survives reloads leave the reset count unchanged in measurements. Handing Caddy a systemd socket would change every listener for the same effect. net.ipv4.tcp_migrate_req cut the resets by about 93%. It needs Linux 5.14 or newer, which every supported Ubuntu release has.
Caddy from its GitHub release, not its apt source
The Caddy project publishes its apt source on Cloudsmith. That source broke four times: a used-up bandwidth quota in 2024, an expired key in December 2025, an expired signing subkey in September 2026, and402 Payment Required on every request from October 2026. A source that fails to fetch fails every apt-get update on the Node, so it also broke steps that never install Caddy, such as the PHP step.
The .deb on the GitHub release is the Caddy project’s own build, and a digest pin needs no third-party signing key. A converge does not upgrade a Caddy that already reaches the floor, because an upgrade restarts Caddy on every Node of the rollout.
The footprint converge, not node:add
After a Gateway release, a Node needs only what the Gateway renders from its own code and pins.node:add refuses a Node that owns Instances, and a role converge does far more than the Orbit footprint. So the footprint converge re-applies only the artifacts whose digest changed, and it never changes an Instance.
A Mac needs no service role
A Mac is a managed Node for tools, not a service host. It uses the account, WireGuard identity, and SSH access that already exist. Enrollment pins the approved host key and checks the platform, architecture, account, and tunnel before it succeeds. It creates no account, installs no Ubuntu packages, and does not change host DNS or the firewall. An application role was rejected because tools do not need application services and the role list stays empty. Package operations through the Node agent were rejected because SSH already makes the changes and the agent stays observation-only. The Ubuntu bootstrap was rejected because its packages, users, resolver, firewall, and systemd units do not apply to macOS. A second Homebrew or Vite+ install was rejected because the packages already on the machine would stay outside the scope Orbit manages. Platform support is checked per operation. Tool management and Doctor run on the enrolled Mac. Linux roles, exporters, Processes, Schedules, and the Node agent do not. A role assignment fails before any remote change. macOS OS updates, firewall management, application hosting, and a macOS Node agent are separate features. A roleless Ubuntu Node proves the same empty-role boundary on Linux, including apt adoption without a reinstall. A real Mac proves enrollment and the Homebrew and Vite+ lifecycle. Low free space on that Mac makes the Node drift withnode.disk_low while informational package findings stay informational.