Skip to main content

​
Update and recover a Gateway

This guide helps an operator keep Gateway state safe during a source update and recover when an update fails. It uses the installation layout from the Quickstart. Test the procedure on a disposable copy before you rely on it for important data.

​
Gateway request logs

The Gateway logs an exception at ERROR when its response status is 500 or higher. A lower status is a client refusal. The Gateway does not log it, because Activity already records the failed request. An invalid task definition is HTTP 422, so it is one of those refusals. Every log entry written during an HTTP request carries that request’s request_id. When the request creates an Activity row, its activity_log.request_id has the same value. Search the Gateway logs for the Activity row’s request ID to find all related request log entries. Gateway log files rotate daily and keep 14 days. LOG_DAILY_DAYS changes the count. Gateway web setup lets Caddy read the regular files and directories under the checkout’s public directory, also files restored with restrictive permissions. It does not follow symlinks there or change the permissions of other source files. The Gateway .env stays at mode 0600. In the release layout, each release gets this access when it is prepared.

​
Preserve a complete state set

The database alone is not a recoverable Gateway backup. Keep these inputs together and store the backup outside the machine, with access limited to administrators. A nonempty APP_KEY in the environment takes precedence over ORBIT_HOME/gateway.app-key. Preserve the effective key with its encrypted data. Do not generate a replacement key during an update or restore. A retained public CA certificate cannot replace its private key.

​
Back up before an update

Schedule a maintenance window and stop CLI users and automation from changing the fleet. Record the current commit, Gateway status, and the first application’s successful HTTPS response. Keep provider-console access available. On the Gateway, record the source revision as orbit:
Resolve or preserve local edits before checkout changes. Stop incoming requests and any running Gateway console operations before copying state. For a complete machine recovery point, take a provider or hypervisor snapshot while the machine is shut down. Store the recorded commit and snapshot identifier with the backup. For a state archive on a quiescent machine, run these commands from a root console. Use a new backup directory for each attempt:
In the release layout, archive home/orbit/shared/gateway.env instead of home/orbit/orbit/apps/gateway/.env, which is a link there. Adjust the paths when ORBIT_HOME or DB_DATABASE differs. Include an external database path and its -wal and -shm files, as SQLite WAL default explains. Pause Gateway timers and external automation too; stopping the web services alone does not stop console writers. Do not copy a live SQLite file without its journal state. Transfer the archive to protected storage and verify that it can be read before changing source. The archive contains secrets. Do not attach it to a bug report or commit it to Git. A successful archive listing verifies readability, not restore behavior.

​
Release layout

A Gateway in the release layout runs from immutable releases instead of an in-place checkout. Each release is one exact commit, built beside the live one, so preparing a release never changes what the Gateway serves. The paths come from ORBIT_GATEWAY_CHECKOUT, which names apps/gateway below the current link. Run release commands through that link, as php /home/orbit/orbit/apps/gateway/artisan. A shell that changed into the directory before a switch still runs the release it entered. A release directory is never changed after it is prepared. Its source directories are read-only, so an in-place git checkout or composer install inside a release fails instead of changing it. Only apps/gateway/bootstrap/cache and apps/cli/storage stay writable. The Gateway reports its version in gateway:status. When APP_VERSION is unset or empty, the version is the full commit in the release’s REVISION file, so bin/deploy-verify --sha works without an env edit. Without either, the version is dev. Leave APP_VERSION out of the shared env file in the release layout; a set value always wins.

​
Prepare a release

As orbit, build a release for one commit:
Name the commit by its hex SHA, 7 to 40 characters. Branch names, tags, and other revision syntax are refused. Prepare fetches the repository’s branches through the GitHub App when the commit is not present yet, and fetches a full SHA directly when no branch holds it any more. Then it:
  1. creates the worktree releases/<id> for the exact commit;
  2. links the shared env file and the shared storage directory into it;
  3. runs composer install --no-dev and composer check-platform-reqs --no-dev for apps/cli and apps/gateway, with the committed locks. A release holds only the packages it runs, not Pest, PHPStan, Pint, Rector, or Boost;
  4. installs the commit’s web build from CI into the web directory, without serving it yet;
  5. gives Caddy the same access to the release’s public directory that Gateway web setup gives a checkout, and makes the source directories and files read-only;
  6. writes REVISION, then runs php artisan config:cache in the release, so the cached configuration holds the release’s version. The cache holds APP_KEY, so it is mode 0600.
Every artisan command of a release runs with a clean environment that has only HOME, PATH, and LANG, so the release reads its configuration from the shared env file alone. These commands, and composer install, hold ORBIT_HOME/gateway-release-step.lock while they run. When the process that started one dies, for example with its SSH session, the command still finishes. Until it has, the next release step is refused with gateway.release_in_progress. Only a release with a REVISION file is prepared. When the configuration cannot be cached, prepare removes REVISION again. The Gateway runs with a cached configuration, so a change to the shared env file takes effect only after gateway:release:configure. A release that already has it is reused without another build step. Only its web build is installed again when it is missing. A partial release from a failed or interrupted prepare is removed and built again on the next run of the same commit, or pruned by a later verified release once it is 1 hour old. Prepare never touches the current release link, the database, or a running service. Prepare refuses before it fetches or writes when the releases directory has less free space than ORBIT_GATEWAY_RELEASE_MIN_FREE_MB, 1024 MiB by default. Each release has its own vendor/ directories. Releases share the Git objects in shared/orbit.git, so a release costs about the size of its source and its two vendor/ directories, about 115 MB without development packages. The command prints one JSON object. Success exits 0 with release, sha, path, reused, and duration_ms. A refused commit exits 2. Every other failure exits 1 with error_code, step, and message.

​
Web build

The web app ships with the Gateway release of the same commit. On every CI run on main, a push or a manual dispatch, CI uploads the build of apps/web as the artifact web-dist-<sha> and keeps it for 14 days (Feature delivery: CI). Prepare installs it into web/releases/<id> of the web directory, in the layout bin/web-deploy uses:
  1. It asks the GitHub App for a token with only Actions: read, for the repository that shared/orbit.git names as origin.
  2. It finds the newest unexpired artifact with that name from an accepted CI run of exactly this commit.
  3. It downloads the archive, at most 128 MiB, from GitHub’s artifact storage without the token. The download must match the artifact’s SHA-256 digest.
  4. It extracts the archive into a hidden staging directory and refuses unsafe entries, listed below.
  5. It requires index.html at the top. It gives directories mode 0750 and files 0640, with group caddy.
  6. It moves the staging directory to web/releases/<id> in one rename.
Extraction accepts only regular files and directories with plain relative names. It refuses links, special files, absolute names, .., duplicate and encrypted entries, and entries whose size or checksum differs from their header. It also refuses more than 20,000 entries or more than 512 MiB of files. A complete build is reused. Nothing serves it until the release verified: deploy then switches web/current to it in one rename. Open pages read the new build’s version.json and move to it on their next navigation or resume (Updates to open pages). A failure leaves no partial build behind. After a verified release, the Gateway removes every web build that belongs to no retained release, such as the build of a commit whose prepare failed later, or an older bin/web-deploy build. It never removes the build current serves. When the releases directory cannot be read, it skips this cleanup and logs a warning. CI’s Required checks job needs the Web job, and the Web job uploads the artifact before it succeeds. So a commit with passing checks has its artifact, and a missing or expired one fails the commit at once instead of waiting for it. A newer commit is released instead. The artifact expires after 14 days, so deploying or adopting an older commit fails at this step.
Accepted CI runs
The run must be a push or a workflow_dispatch of .github/workflows/ci.yml on main, in this repository. A run from a fork, a pull request, another workflow, or another branch does not count. An artifact without a SHA-256 digest is refused.

​
Deploy a release

Deploy is the manual release. Before it changes anything, it:
  1. prepares the commit;
  2. refuses a commit that does not descend from the current release, with gateway.release_downgrade;
  3. caches the release’s configuration again from the shared env file, so a broken env file stops the release before any migration.
--force skips only the commit ancestry check. Laravel owns migration discovery, including migrations registered by vendor service providers. Then it takes the pre-release database snapshot, runs php artisan migrate --force, and switches /home/orbit/orbit to the release with one mv -T. The switch runs only when the current path is already a release link. An in-place checkout, or a link to something else, is refused until gateway:release:adopt.
After the switch, deploy:
  1. runs the serving phase of the runtime handoff: Caddy, PHP-FPM, and the units;
  2. verifies the release, as described below;
  3. runs the scheduler phase of the handoff;
  4. switches web/current to the release’s web build with one rename;
  5. runs smoke against the switched release.
Verify runs before the scheduler phase, so a broken release is found and switched back without waiting for a long scheduled command or an idle pool. Verify checks GET /up and Gateway status at ORBIT_GATEWAY_VERIFY_ORIGIN (default https://gateway.orbit). Status must be ok, and the version must be the full commit or its 12-digit id. A busy pool can queue /up behind long requests, so a failed check runs again after 2, 4, 8, 16, and 30 seconds. Each request has a 15-second overall deadline. HTTPS trusts only /etc/caddy/orbit-cert-current/root-ca.pem, with the system CA directory disabled. A missing CA file fails verification. Laravel’s HTTP client retries the complete check, including status and version validation. After the last failure, it reports gateway.release_verify_failed at the verify step with the failed endpoint or the observed status and version. Smoke runs after the web switch. Any failure after the switch counts, including gateway.release_unexpected_failure. Deploy pauses after running the migrator. It writes ORBIT_HOME/gateway-release.paused and records outcome paused. This also applies when Laravel has no pending migrations. Deploy does not inspect migration files or infer schema compatibility. A failed switch pauses too; the previous code then still serves. Rollback does not run migrations. A failed rollback switches back to the previous release and restores web/current. It repeats the handoff phases that started before the failure. A previous release whose code has no handoff command or phases gets the handoff from the deploying process’s code. After a verified release, deploy removes old releases. It keeps at most ORBIT_GATEWAY_RELEASES_KEEP releases, 3 by default and at least 2: the current one and the previous one, whatever their age, and then the newest others. After a rollback to an older release, the previous release takes the place of the oldest of the newest others, so the count never grows. It also removes each release directory without REVISION that is more than 1 hour old, with its web build. Such a directory is left by a prepare that stopped, for example an adoption that lacked a GitHub App permission, and no later prepare of another commit removes it. Every prepare holds the release lock, and so does deploy while it prunes, so no prepare is writing to the directory. The age is a margin on top. Deploy keeps an incomplete directory that is the current or the previous release, because that release needs a repair. It also keeps a directory whose REVISION names another commit. Remove that one by hand. A directory that cannot be removed stays, and each verified release logs a warning with its id.

​
Migrations and the snapshot

Before every deploy switch, deploy first writes a consistent copy of the Gateway database to ORBIT_HOME/backups/pre-<id>-<time>-<random>.sqlite, with mode 0600. Each attempt writes its own file, so a retry after a failed migration never replaces the clean copy. A retry of a paused commit keeps naming the copy from before its first attempt, in its record and in the pause marker. It uses SQLite VACUUM INTO, so it needs no sqlite3 binary. Then the release migrates with its own code, php releases/<id>/apps/gateway/artisan migrate --force, while the previous release still serves.
  • The Gateway keeps the newest ORBIT_GATEWAY_RELEASE_SNAPSHOTS_KEEP snapshots (default 5), and always the one the pause marker names.
  • The release migrates with migrate --isolated, so a second migration waits for a running one.
  • A snapshot needs room for the database and its WAL plus the free-space floor. Prepare keeps that room too, so a release refuses with gateway.release_disk_low before it builds anything.
  • A failed snapshot changes nothing, and the commit is tried again later.
  • A failed migration may have applied part of its changes. Deploy then pauses with gateway.release_migrate_failed and names the snapshot. To go back to it, follow Recover a failed update with the snapshot as the database file.

​
Runtime handoff

The handoff runs php releases/<id>/apps/gateway/artisan gateway:release:handoff --phase=serve, then, after verify, --phase=schedule, with the code of the release that just became current. A release that changes what the Gateway renders applies that change at once, although the deployer itself runs from the previous release. In order, the handoff: The serving phase:
  1. publishes the Gateway Node’s Caddyfile when the render changed, with a graceful Caddy reload;
  2. compares the rendered pool with /etc/php/8.5/fpm/pool.d/orbit-gateway.conf and reloads PHP-FPM only for a difference, as a reload ends requests in flight;
  3. installs the hibernator, agent-view, and release units again and restarts agent-view. The units name the stable /home/orbit/orbit/apps/gateway path;
  4. moves Laravel Horizon to the new release, with php artisan horizon:terminate while it runs.
The scheduler phase:
  1. moves the scheduler to the new release without cutting off a scheduled command;
  2. restarts every other Gateway Node Process whose directory is in the Gateway application, except the horizon Process;
  3. resumes document cleanup;
  4. moves the jobs left in the old database queues to the Gateway queue;
  5. updates the Gateway Node’s own orbit-agent to the release’s pin, described below;
  6. resets OPcache, described below.
PHP-FPM is not restarted for a release. Caddy resolves the /home/orbit/orbit link for each request (resolve_root_symlink) and passes PHP-FPM the release’s real script path. A request that started before the switch finishes on its release, and the next one runs the new release. The /grafana authorization resolves the link the same way. A fixed script path through the link would let each PHP-FPM worker keep the old release in its realpath cache for up to two minutes. Release files never change in place, so OPcache never marks the scripts of an old release as wasted, and the cache fills up. The scheduler phase therefore ends with an OPcache reset through the pool’s socket, with a script outside public/. All pools of the PHP-FPM master share one OPcache. OPcache restarts once no request uses the cache. A restart that stays pending for opcache.force_restart_timeout (180 seconds) kills the workers that still serve a request, and a Gateway request may run 600 seconds. So the handoff resets only while no enabled pool has a connection on its listen socket, and the restart happens with the next request. It waits up to 60 seconds for that moment. Then it asks the pool again until the restart has happened. A request that starts in the milliseconds between the idle check and the reset, and runs longer than 180 seconds, could still be killed. Setting opcache.force_restart_timeout above 600 seconds closes that gap, but it needs a PHP-FPM restart, and a restart ends requests in flight while process_control_timeout is 0. Leave it for a planned PHP-FPM restart. The handoff never reloads PHP-FPM for OPcache. The scheduler is the Gateway Node’s systemd Process that runs schedule:work in the Gateway application directory. Every Gateway runs one. A missing one fails the handoff with gateway.release_scheduler_missing, and one in another directory with gateway.release_scheduler_mismatch, because it would keep the old code. The handoff never cuts off a scheduled command in the normal path. It moves the scheduler in these steps: The handoff compares recorded working-directory paths to find the scheduler and other Gateway systemd Processes. A null Docker working-directory override is not a Gateway application path and is left alone. Ordinary Process convergence leaves an unchanged completed unit untouched; the release handoff still explicitly moves Gateway services to the new release. The drain needs the scheduler’s PHP to have the pcntl extension, so schedule:work can handle SIGTERM. Without it, the handoff takes the forced path at once and records reason: no_pcntl.
  1. It runs schedule:interrupt, so repeating events such as tasks:tick stop after their current run.
  2. It sends SIGTERM to the schedule:work main process only. That process starts no new schedule:run and waits for the running ones, so a long command such as a development deploy finishes on the release it started on.
  3. When the main process has exited, it starts the unit, which now runs the new release, and waits until it is active.
Each command ran to its end and released its own overlap lock, so the handoff clears none. While the old scheduler drains, it starts nothing, so tasks:tick pauses for the drain at most. While the Gateway queue is set, the scheduler only dispatches, so the drain takes seconds, and a running tick finishes in Laravel Horizon. The drain waits at most ORBIT_GATEWAY_RELEASE_SCHEDULER_DRAIN_SECONDS (default 600). After that limit, the handoff takes the tasks:tick lock and stops the unit, which ends what still runs. It then clears the schedule’s overlap locks with schedule:clear-cache, because the stopped commands cannot release them, and starts the unit. The release record’s handoff.scheduler_drain shows outcome (drained, forced, or not_running), waited_ms, the commands that were running when the drain began, and, after a forced stop, the commands it stopped. The new scheduler pauses document cleanup when it starts. The handoff waits for the new cleanup generation, runs project-documents:cleanup:reconcile, and resumes with that report. Without configured document storage it reports skipped. When the report has differences or resume is refused, cleanup stays paused. The release record then has cleanup_paused: true and the handoff’s cleanup_error_code, and the release continues. The handoff also installs the fleet rollout units. A failure there only logs a warning. When any step fails, the handoff still starts the scheduler unit, so the scheduler never stays down. The handoff prints one JSON object with caddy, fpm, opcache, scheduler, scheduler_unit, scheduler_drain, processes_restarted, horizon, queue_moved, cleanup, cleanup_error_code, agent_view, cleanup_paused, and gateway_agent. Run it again by hand after fixing a handoff failure. It takes no release lock, so run it only when no release step is running.
Gateway Node agent
The fleet rollout never visits the Gateway’s own machine, so the scheduler phase updates that Node’s orbit-agent itself. It runs with the release’s own code, so it uses that release’s agent pin. It runs after verify, so an agent restart never delays verify. When smoke later fails and the release switches back, the scheduler phase runs again with the previous release’s code, which moves the agent to that release’s pin. It leaves the Gateway’s CLI alone: the Gateway runs the release’s own apps/cli. The update runs one script through local sudo, never over SSH to itself. The script holds /run/lock/orbit-self-update.lock, the lock that orbit self-update and the agent converge hold, and waits up to 120 seconds for it. It does nothing when the unit at /etc/systemd/system/orbit-agent.service lacks the # Managed by Orbit: agent marker, or when the binary already matches the pinned SHA-256. Otherwise it takes the converge’s steps: it downloads a candidate, checks the SHA-256, sets root:root and mode 0755, and renames the candidate into place. Only then does it restart orbit-agent.service. The agent must stay active without a restart for 5 seconds, the health window of orbit self-update. When the restart fails or the agent does not stay up, the script restores the previous binary from /usr/local/bin/orbit-agent.orbit-previous and restarts it. Each restart has a 60-second limit. The handoff result records the update as gateway_agent. It has an outcome and the pinned version: A failed update never fails or switches back the release. The release stays verified, and the record raises one release_gateway_agent_failed alert, stored as gateway_agent.alert. Doctor keeps reporting node.agent_binary_mismatch for the Gateway Node until a later release updates the agent. To retry sooner, run the scheduler phase of the handoff again by hand, as described above. Do not run orbit self-update on the Gateway machine, because it also replaces the Gateway’s CLI.

​
Smoke

Smoke runs bin/gateway-smoke of the new release, releases/<id>/bin/gateway-smoke, as the Gateway account. So it is the exact code under test, and it uses that release’s apps/cli/orbit with the account’s Gateway profile. Delivery-line proofs: bin/gateway-smoke lists its checks. The release passes: The Python checks trust Orbit’s root CA through SSL_CERT_FILE, set to Caddy’s root-ca.pem unless the environment already sets it. The document write check is off by default, because it writes a Project Document on every release. To turn it on, create a dedicated Project and set ORBIT_GATEWAY_RELEASE_SMOKE_PROJECT to its ID or slug in the shared env file, then apply the env change. The release record stores the smoke JSON as phases.smoke.report, also when smoke fails. Smoke fails when it exits nonzero, reports passed: false, prints no JSON, or the release has no bin/gateway-smoke. When it runs 15 seconds past its own limit, the step fails with gateway.release_smoke_timeout. timeout then sends SIGTERM to smoke, which kills every check command it started, each in a session of its own, and prints a terminated result that the record keeps as the report. After 5 more seconds, timeout kills what is left. No check outlives the step. A smoke failure is handled like a failed verification: switch back without migrations, pause after them. Smoke does not wait for the first tasks:tick, because a restarted scheduler starts it only at the next full minute. It checks that the scheduler’s process runs from the new release and that the release’s artisan schedule:list lists tasks:tick. The first tick is confirmed after the release. Pass --wait-for-tick to bin/gateway-smoke by hand to wait for it instead.

​
Post-release tick confirmation

A verified release starts with the step tick set to pending. The step records since, the time the runtime handoff started. It also records deadline, which comes ORBIT_GATEWAY_RELEASE_TICK_CONFIRMATION_SECONDS after the release was verified. The default is 180 seconds, and the minimum is 60. Every tick of the release runner decides the pending confirmations it can, also while automatic releases are disabled or paused, so a manual deploy and a rollback are confirmed too. tasks:tick records when it started and the version of the code that ran it, the commit in the release’s REVISION. The step ends as one of these: While the Gateway queue is set, Horizon runs tasks:tick. During a release, the old Horizon master can wait up to 1700 seconds for a long job, and no master is registered until it exits. While no master is registered, a confirmation past its deadline stays pending, for at most those 1700 seconds more. The alert then names the scheduler Process and the horizon Process. | superseded | A newer verified release went live. That release has a confirmation of its own. While another release is only being tried, the step stays pending, because a failed attempt switches back | A missed confirmation does not switch back or pause. The release passed verify and smoke, and its scheduler runs the new code, so a forward fix still ships automatically. The record stays verified, and the fleet rollout follows it as usual. Check the scheduler Process with orbit process:list and orbit process:logs. While the Gateway queue is set, check the horizon Process too. gateway:release:show lists the step with the others, and gateway:release:auto:status shows it for the current release as tick_confirmation. Run the same smoke by hand against the live Gateway. It runs bin/gateway-smoke of the current release for its commit, or for the commit you name. It changes nothing and writes no release record.
From any Node with access to the Gateway, orbit gateway:release:smoke asks the Gateway to run it. Smoke restarts nothing and writes no record, so it runs inside the API request, unlike a deploy. An API request gets 570 seconds, 20 of them reserved for cleanup, and PHP-FPM ends it after 600. So the API lowers the smoke limit to what the request has left, about 520 seconds at most. With the 15-second grace period and the stop and kill delays, the run ends within the request. The default limit of 90 seconds stays as it is. Checks that did not pass answer with the failed outcome and the report, and the CLI exits 1. One API smoke runs at a time, because a run and its checks hold several PHP-FPM workers. A second one fails with gateway.release_smoke_in_progress. On the Gateway host, the Artisan command runs the same smoke without the lower limit:
It prints one JSON object with release, sha, outcome, and the smoke report. A failed smoke exits 1 with error_code, step, message, and the report under detail.report. A commit or --since value it cannot read exits 2.

​
Release records

Every attempt writes one release record. The record exists from the start of the attempt and stores each step as it ends, so a caller can follow it. The same attempt writes an Activity entry with the command gateway:release:<trigger>. A refusal that changes nothing, such as an unknown commit or a downgrade, ends its record failed and retryable, so the commit is not marked failed. A step refused by the release lock writes nothing, unless it ran for a queued record, which then ends failed. A failure the commit itself causes, such as a downgrade, a failed migration, or a missing or invalid web build, is final at once. Any other failure, such as low disk, a fetch error, a slow verify, or an unexpected error, is recorded with retryable: true for the first two attempts of a commit; the third is final. Only earlier finished attempts that failed, switched back, or were interrupted count, from a deploy, an automatic release, or an adoption. A rollback, a verified release, and the attempt that runs do not count. gateway:release:list reads the 50 newest records. gateway:release:show takes a record id of one to six digits, or a hex SHA of 7 to 40 characters for the newest record of that commit. A record is interrupted when its process ended before it wrote an outcome, for example after a kill or a unit timeout. Three places end such a record with gateway.release_interrupted:
  • the release unit’s ExecStopPost, which runs gateway:release:settle as soon as its main process exits;
  • every tick of the automatic runner, under the release lock;
  • every new release attempt and every API request, under the release lock.
Under the lock no release process runs, so a running record is dead. ExecStopPost also ends a running record only while the lock is free. A queued record is dead when its unit does not run two minutes after the request. When an interrupted record’s steps show a snapshot, a migration, or a switch, it pauses automatic releases and alerts, because a retry could run old code on a migrated schema. Otherwise it spends one attempt of the commit’s retry budget, and it alerts once that budget is spent. An attempt that died before prepare named the commit counts by the full SHA it requested. While a record is running, the Gateway Node reads as updating. One release step holds ORBIT_HOME/gateway-release.lock. A second step is refused with gateway.release_in_progress.

​
Deploy through the API

orbit gateway:release:deploy <sha> and orbit gateway:release:rollback <id> ask the Gateway API for the release. The release never runs in the PHP-FPM worker that answers, because it can restart the scheduler and reload PHP-FPM. Instead the Gateway:
  1. ends the records of dead releases, then checks that no release step holds the lock and no requested release waits to start;
  2. writes a queued record;
  3. starts orbit-gateway-release-run@<record>.service with sudo systemctl start --no-block;
  4. answers 202 with the queued record.
The unit runs gateway:release:run <record>, which claims the record and runs the same deploy or rollback as the commands above. The CLI follows the record until it finishes. A queued start is no proof, so the Gateway reads the unit state for up to five seconds after it. When systemd refuses the unit, the unit fails, or it neither runs nor claims the record in time, the request fails with gateway.release_unit_failed and the record ends as failed. Install the units with orbit:gateway-web on the Gateway. When the handoff reports gateway.release_scheduler_busy, the old scheduler still finishes its commands and exits. With Restart=always, systemd then starts it on the current release. Otherwise, start its unit by hand.

​
Roll back

Roll back to return the Gateway to a release it still keeps, for example after a pause or a bad release that verify and smoke did not catch.
<id> is the first 12 hex digits of a retained release. With the default of 3, a rollback can reach the previous release and one other retained release. Rollback switches to it and runs the same handoff, verify, web switch, and smoke as a deploy. Rollback switches code only. It names the newest pre-release snapshot for recovery, but never runs migrations or restores the database. Check that the target code can use the current database. If a database restore is needed, keep access and automation paused and follow Recover a failed update with the matching snapshot; restore is an explicit operator step. A failed verification switches back to the release that was current, because rollback itself does not migrate. Rollback switches the web app to the target’s build. Pruning a Gateway release removes its web build too, so every retained release keeps its own, and a manual bin/web-deploy never prunes it. When the build is gone anyway, rollback installs it from the commit’s CI artifact, which exists for 14 days. When the artifact has expired too, a rollback does not block the code on assets: it leaves web/current as it is, records the web step as kept with a warning, and smoke skips its web check. A deploy always fails without its web build.

​
Apply an env change

Every release caches its configuration, so an edit to /home/orbit/shared/gateway.env changes nothing until you cache it again. As orbit, after the edit:
It caches the current release’s configuration beside the live cache and renames it into place, so a request never reads half of it. Do not run php artisan config:cache in a release: it rewrites the live cache in place. A deploy or rollback caches its target again, so a retained release picks up the edit when it goes current.

​
Automatic releases

The Gateway can release itself from its own branch. A timer looks for a new commit that passed CI’s Required checks every minute, and the release goes live without a maintenance window. Automatic releases are disabled by default. An operator turns them on after adoption and a first release by hand. The operator also decides what happens after a pause.

​
Release units

The Gateway runs releases in systemd units, never in PHP-FPM or the scheduler. orbit:gateway-web and every runtime handoff install them. The handoff writes the units and enables the timer, but it never starts or restarts a release unit, so the release that runs the handoff keeps running. Each unit runs as orbit from /home/orbit/orbit/apps/gateway, so a run starts from the release that is current. It sets PATH, HOME, and LANG=C.UTF-8, so php, git, and composer resolve as they do for the account. A unit may run for one hour, because prepare runs composer install. A timer tick that comes while the service still runs starts nothing. After the main process exits, ExecStopPost runs gateway:release:settle: the run unit with its record id, the service without one.

​
What a tick does

Each tick of gateway:release:auto takes these steps and stops at the first that ends it:
  1. It decides the pending tick confirmations.
  2. It stops when automatic releases are disabled or paused. A stall ends then.
  3. A requested release that waits for its unit goes first. The tick stops.
  4. It takes the release lock and ends the records of dead releases. A busy lock stops the tick.
  5. It stops without a deployed commit: the Gateway runs from no release, or REVISION is unreadable.
  6. It asks GitHub for the newest green commit of the branch that descends from the deployed commit. Commits with a release that failed for the commit itself are left out.
  7. A manual deploy that pinned an older commit after the last resume is never undone. The tick pauses with the reason manual_deploy.
  8. It deploys that commit with the trigger auto, under the release lock.
The repository is the origin of the shared release repository. ORBIT_GATEWAY_RELEASE_BRANCH and ORBIT_GATEWAY_RELEASE_CHECK name the branch and the check, by default main and Required checks. A commit whose release failed or switched back with a retry left is tried again after 10 minutes, up to three attempts. Every tick stores its result as the last tick. A tick prints it as one JSON object. It exits 1 only when it ran a release that failed.

​
Pause

A pause is durable state with a reason. It holds until resume or until a deploy ends verified, whatever other release records come after it. The start of every pause raises one release_paused alert that names the reason. A manual deploy or rollback still runs during a pause. A manual deploy, an automatic release, or an adoption that ends verified ends a pause, so a forward fix hands back to automation. An adoption that ends resumed serves a release that was never verified, so it keeps the pause. A rollback that goes live starts a rollback pause instead. When a manual deploy goes live, the Gateway asks GitHub whether a newer green commit exists. It stores the answer on the record as the newest_green step: superseded with that commit, newest, or unknown when GitHub did not answer. superseded and unknown make the next tick pause with manual_deploy, so a pin is never undone on a guess. After a GitHub outage, a manual deploy may need one resume. A manual deploy of the newest green commit hands back to automation, and a newer commit that turns green later is released as usual. After a failure, decide between a forward fix, a rollback, and a restore from the snapshot. Then clear a pause that is left:
Resume removes the pause and the marker. The next tick releases the newest green commit, also over a manual release from before the resume. Resume refuses with gateway.release_not_paused when nothing is paused. A commit that failed is not released again automatically.

​
Failure and alerts

A release record that ends paused, or that ends switched_back, failed, or interrupted with no retry left, raises one release alert. An interrupted record that touched live state alerts at once, because it pauses. A pause that a rollback or a manual deploy starts alerts too. So does a release that goes live but leaves document cleanup paused. The receipt is stored on the record, so the record never alerts twice. A retryable failure does not alert by itself. Two stalls alert once with the kind release_stalled. The subject is the deployed commit.
  • A GitHub error, a retryable failure, or a busy release lock lasted 30 minutes. The next tick that reaches GitHub ends the stall.
  • The branch head stayed unreleased for 6 hours. This check also runs during a pause. A deployed head ends it.
While it is up to date, the runner reads the branch head at most every 15 minutes. The second stall covers a deployed commit that left the branch after a force push, because then no newer commit can qualify.

​
Read the state

orbit gateway:release:auto:status shows the switch, a pause with its release, error code, and snapshot, the current release, the last tick, the stalls, and the current release’s tick confirmation. orbit gateway:status shows the current release and a short summary. A last check older than two minutes means the timer does not run. Check it with systemctl status orbit-gateway-release.timer on the Gateway.

​
Adopt the release layout

A Gateway installed with the Quickstart runs from an in-place checkout at /home/orbit/orbit. gateway:release:adopt converts it into the release layout once, and the Gateway keeps serving throughout. Run it from a temporary checkout of the commit to adopt into, so the in-place checkout needs no update first. After adoption, deploy, roll back, and configure with the release commands. An update in place then fails, because each release is read-only.

​
Before you adopt

Check these conditions on the Gateway before you run the command.
  • Back up as in Back up before an update.
  • git status --short --untracked-files=no in /home/orbit/orbit prints nothing. Untracked files, such as .env backups, are fine.
  • The disk has room for one release, about the size of the checkout without .git, plus the free-space floor and a database snapshot.
  • python3 is installed. Ubuntu installs it by default. Without it, the swap uses two renames, and the path is missing for the microseconds between them.
  • The Gateway GitHub App installation has accepted Actions: read. See Read CI artifacts. Phase 2 needs it for the web build.
  • orbit node:list succeeds as orbit on the Gateway. The Quickstart step “Connect the CLI” sets this up. Smoke needs it.
Without the permission, phase 2 stops in prepare with gateway.release_web_build_unavailable. Nothing changes live. Without the CLI profile, smoke fails with gateway.profile_missing, and phase 2 pauses after its migrations.

​
Run adoption

As orbit, make a temporary checkout of the commit to adopt into, <SHA>, and install its dependencies. Its env file is a link to the Gateway’s, so the command reads the Gateway’s configuration and state. Keep its logs, which record the run:
Adoption holds the release lock and runs in two phases. The first changes the layout but not the code. The second changes the code through a normal deploy. So no process ever runs code from two commits. Phase 1 runs these steps for the commit the in-place checkout has: Phase 2 deploys <SHA> exactly as Deploy a release describes, with its snapshot, migrations, handoff, verify, web build, and smoke. Without --commit, it deploys the checkout’s commit itself, which then must have the command; that adds the web build, the exact version check, and smoke. The web build comes from the commit’s CI artifact, which CI keeps for 14 days, so phase 2 needs a commit built within that time. The command prints one JSON object with release, sha, from, pre_adopt_path, shared, switch, phase1, and deploy, the phase-2 release record. switch.method and shared.storage_method are exchange, or rename with the gap in switch.gap_us and shared.storage_gap_us. Each phase writes a release record with trigger adopt and an Activity entry. Running it again on a Gateway that finished adoption prints "already": true and changes nothing. When an earlier run swapped and then stopped before it verified, for example because its session dropped, the next run hands the runtime over and checks serving again. It records outcome resumed, prints "resumed": true, and then runs phase 2, which verifies the exact version. Only the phase-1 release can resume this way, and a resumed record keeps any pause marker; only a verified release clears it. The kept checkout is a complete way back. Its .env.pre-adopt holds the original env file, so keep its permissions, and remove it once a few releases have verified:
The first release runs the checkout’s own commit. That commit may predate the release version, so the release reports dev, and a rollback to it fails verify and switches back. With --commit, it also has no web build of its own. Roll back to a later release instead.

​
When adoption fails

Each step checks whether it already ran, so after fixing the cause, run the command again. The phase-1 release may have no handoff command of its own. A switch-back to it then gets the handoff from the deploying process’s code. When even the swap back fails, the outcome is gateway.release_switch_back_failed. Swap back by hand as orbit, then hand the runtime over with the temporary checkout’s code:

​
After adoption

The release directories are read-only, and bin/bootstrap refuses to run in a release. orbit:gateway-web converges the release link: it keeps /home/orbit and /home/orbit/releases traversable for Caddy and the shared env file at mode 0600, and leaves the releases themselves unchanged.

​
Recover a failed update

Keep requests and automation paused. Save redacted failure output and preserve the failed state separately for diagnosis. Do not use a generic migration rollback to guess compatibility with older code. If a command failed before any state or machine change, restore the previous source commit and its locked dependencies, then verify status and HTTPS before reopening access. Once migrations or machine changes ran, recover the complete pre-update state set. The safest recovery for this walkthrough is the matching stopped-machine snapshot. Do not connect a restored clone to the original fleet at the same time: it contains the same WireGuard, SSH, and CA identities. For an archive-based restore, stop all Gateway writers and use a root console. Move the failed ORBIT_HOME aside rather than extracting over it, restore the archive’s original ownership and permissions, restore the matching source commit, and reinstall its locked CLI and Gateway dependencies. Restore any external database and service configuration from the same recovery point. Clear stale Laravel configuration, then start services and repeat status, Node, Doctor, DNS, and HTTPS checks before reopening access. Do not migrate the restored database forward while attempting to run the older code. If the effective encryption key, CA private key, database, or required machine state is missing, stop: an apparently healthy process cannot prove recovery. This procedure does not roll back changes already made to other Nodes or restore application data; recover those from their own backups and review their consistency with the Gateway.

​
Release alerts

A release command raises an alert when a release fails, when it pauses automatic releases, or when a fleet rollout halts. The alert leaves an Activity entry and a problem for the outer loop, and it posts a signed webhook when one is configured. Each part is recorded on its own. A failed part never stops the others, and the alert never fails the release command that raised it. A release command raises the alert outside a database transaction, so a rollback cannot discard the records after the webhook went out. Every alert names its subject, a summary, and an optional evidence link. The summary passes through the Gateway log redactor and is cut at 1,000 characters. A field outside these limits is a bug in the calling command, which refuses it before it records anything.

​
Activity entry

Each alert writes one Activity entry with the command release:alert. orbit activity:list --command=release:alert lists them. The entry starts as running and keeps a request ID that the webhook body repeats. Its properties hold the kind, the subject, the summary, the evidence link, and the outcome of the problem and the webhook parts. An outcome is done, skipped, or failed, with a reason. A failed entry is a server-class Activity row. When the webhook keeps failing, the outer loop counts those rows and files that as its own problem.

​
Problem

The alert records the fingerprint release|{kind}|{target}|{sha}, with the source release. Unlike a recurring signal, it is ready on its first occurrence. The next hourly problems:file run files it as a Backlog task in the orbit Project, ahead of every other ready fingerprint. Outer loop describes the sample, the brief, the daily cap, and the mute.

​
Webhook

Set both values in the Gateway’s own .env to receive alerts: When either value is unset or empty, the webhook part is skipped with the reason not_configured, and the Activity entry and the problem are still recorded. The Gateway never returns the secret. It never stores the URL, the secret, or the response body, because a chat webhook URL is itself a credential. Unset both values on a disposable Gateway clone, so its alerts do not reach the operator. The Gateway posts one JSON object. It signs {unix timestamp}.{raw body} with HMAC-SHA256 and sends X-Orbit-Timestamp, X-Orbit-Signature: sha256={hex}, and Content-Type: application/json. This is the same signature as the Coder webhook. The connection timeout is 3 seconds and the whole request has 10 seconds. The Gateway follows no redirect and does not retry. The body serves two kinds of receivers. A receiver that verifies the signature reads the typed fields, such as an agent’s intake webhook. A Slack incoming webhook shows text and ignores the signature, because its URL is the credential. The Gateway still needs a secret to send, so set any random value for such a receiver.

​
Fleet rollout

After each verified Gateway release, the Gateway brings every managed workload Node to the same release, one Node at a time. Each Node runs orbit self-update, the same command an operator runs on a Mac. A fleet failure never rolls the Gateway back. fleet shows and resumes the rollout.

​
Turn it on

The rollout is off until you set ORBIT_FLEET_ROLLOUT=true in the Gateway’s environment. While it is off, orbit-fleet-converge.service exits at once, and Doctor reports no node.release_lag. fleet:rollout:status shows Rollout off.

​
Units

php artisan orbit:gateway-web and the runtime handoff of each release install two units through local sudo. A failed install in the handoff only logs a warning; it never fails the release. After a release is recorded verified, the release command runs sudo systemctl start --no-block orbit-fleet-converge.service and returns. The service runs the new release’s code, so it resolves the desired state of the commit that now serves. A refused start only logs a warning; the timer starts the service within 5 minutes. The release command starts the service only for a release with a desired fleet state: verify saw the Gateway serve the release’s exact commit, and the release ships orbit:fleet-converge. Phase 1 of adopt never starts it. Its verified record only shows that the Gateway serves, often as dev, so a rollout never follows that record alone. The rollout follows the phase 2 deploy. A run that is still busy when a newer release goes current does not take the start; systemd joins it to the running job. So before the run plans and before each Node, it compares the release directory that the current path links to with the directory it runs from. When they differ, the run stops, visits no further Node, and asks systemd for a fresh run 15 seconds later with sudo systemd-run --on-active=15. A run never applies an older desired state than the current one. It compares directories, not versions, so a leftover APP_VERSION never makes it restart over and over.

​
Rollout set and order

The rollout visits a Node when all of these hold:
  • it is not a disposable task sandbox;
  • it is active and runs Linux;
  • the Gateway manages it over SSH: it has a WireGuard address and a pinned SSH host key;
  • it holds at least one active role other than gateway;
  • it is not the Gateway’s own machine: it holds no gateway role and is not the serving host;
  • its /usr/local/bin/orbit is not a CLI that Orbit did not install.
Roleless Nodes, such as operator machines, and macOS Nodes stay out. Their operators run orbit self-update. The Gateway’s own Node stays out too. Each release’s runtime handoff updates its agent, and its CLI is the release’s own apps/cli. fleet:rollout:status lists every Node it leaves out, with the reason sandbox, inactive, platform, unmanaged, gateway, roleless, or foreign_cli. A task sandbox is an app-dev Node that a task VM owns, on an Incus host or at UpCloud. The rollout leaves out the Node of every task VM that is not destroyed, with the reason sandbox. The Gateway creates a task VM for one task group and destroys it when the group ends. Provisioning links its Node to the task VM before any convergence, so the Node never gets the Orbit CLI. The rollout and the catch-up never visit a task sandbox, provisioning installs no Orbit CLI on it, and Doctor reports no node.release_lag for it. The sandbox reason comes first, so a sandbox shows it in every state. A Node leaves the rollout set as foreign_cli when the CLI install finds a link, a script, or another program at /usr/local/bin/orbit. That visit is skipped, never failed, and Doctor reports node.cli_foreign. Every later run probes the Node and, when it answers, inspects the path again, so the Node rejoins once an operator moved the file aside. The order goes lowest risk first. Each role has a group. A Node with several roles goes in the latest group of its roles, so it waits until every lower-risk Node is done. Inside a group, the Node ID decides. Orbit has no agent or s3 role yet. The order already places agent in group 1 and s3 in group 2.

​
Desired state

One run resolves the desired state of the commit the Gateway serves: the CLI release cli-v0.N.0 with the SHA-256 of each binary, the agent pin with the SHA-256 of each binary, and the footprint digest of each Node. A footprint digest covers only what Orbit renders from its own code and pins, never a user’s sites, Routes, or DNS records. It stores the state on a rollout record for that commit. The record belongs to the commit’s verified release record, and gateway:release:show shows it under fleet_rollout, with each Node’s result. A Gateway in the release layout rolls out only a commit whose release record is verified. A run during a release, before the record exists, does nothing (release_unverified). When the configured layout cannot be read, or its link names no complete release, the run rolls out nothing (release_layout_unreadable). An in-place Gateway, whose current path is a checkout, has no release records, and its running commit is the desired state. A Gateway version that is not a commit, such as dev, rolls out nothing. CI publishes the CLI release a few minutes after the commit’s checks pass, and the Gateway usually deploys the commit first. Until the release exists, the rollout is waiting and changes nothing on any Node. Once the release appears, the catch-up starts the sequential visit, which brings the footprint and the CLI to each Node together, with its verify and halt. A Node whose orbit self-update reports the release as pending is waiting too. When the commit’s release is still missing after 30 minutes, the desired state falls back to the newest published release of an ancestor commit with the same CLI code. The rollout then starts with that release and raises rollout_cli_fallback once, so a missing release is never silent. When the commit’s own release appears later, the catch-up visits each Node again and brings it to that release. A started rollout visits no Node while its CLI release cannot be confirmed, for example while GitHub fails after the confirmed release left the cache. Each Node would otherwise get a release it cannot install. The catch-up continues once the release is confirmed again. A rollout that waits longer than 2 hours raises rollout_stalled once. It means the fleet is stuck: CI never published the release, and no ancestor’s release could stand in. Publish the release, or merge a newer commit.

​
One Node

For each Node, the run takes the Node’s role lock, the lock every role converge takes, and runs these steps. The cli and footprint steps also hold the Node’s update lock, /run/lock/orbit-self-update.lock; self-update takes that lock itself, in between. So a Gateway step and a self-update never run on one Node at the same time. A self-update that keeps the lock longer than 5 minutes leaves the Node deferred with node.update_busy. A step that fails while the Node still answers SSH runs once more, because a dropped channel or a timeout is usually transient. When the Node does not answer a new probe, it is unreachable. A self-update that answered with its JSON verdict is not retried. A step that meets a busy lock (node_role.node_busy, node.update_busy, self_update.busy, agent.converge_busy, agent.converge_lock_lost, node.lock_lost, tool.operation_locked) leaves the Node deferred. The caddy-package artifact uses the shared Caddy package step. Its source rename changes that artifact’s digest once. The next footprint run repeats the step without reinstalling a Caddy at or above the floor. When the baseline shows an owned issue, the footprint step re-applies the artifact that repairs it even though its digest matches: agent for an agent issue, caddy for Caddy drift. A Caddyfile that the render refuses, because a site cannot be built, is skipped: the live Caddyfile stays, Doctor keeps reporting role.caddy_build_drift, and the verify tolerates that issue on this Node. A Caddyfile that caddy validate refuses can also be a fault in Orbit’s own template or Caddy pin. When the Caddy digest changed, so Orbit’s own Caddy code is new, a refusal on the first Node of a rollout, before any Node converged, fails the Node with node.footprint_caddy_failed and halts. A refusal on a later Node, or while the step only repairs live drift that the baseline showed, is skipped like a render refusal. The first skip in a rollout raises rollout_caddy_skipped once, so a kept Caddyfile is never silent. After a Gateway rollback, the desired CLI release is older than the Nodes’ CLI, and self-update refuses with self_update.downgrade_refused. When the desired state belongs to a verified release record, the step runs self-update once more with --allow-downgrade-to=<desired version>, which allows a downgrade to exactly that version and to no other. An in-place Gateway has no release records, so it never passes the option. The verify fails when Doctor reports an issue that was not there in the baseline, or any issue the rollout owns: node.ssh_unreachable, node.agent_missing, node.agent_binary_mismatch, node.agent_inactive, node.agent_secret_mismatch, node.cli_foreign, or role.caddy_build_drift. It checks Doctor three times, 10 seconds apart, before it fails, because a restarted agent or a reloaded Caddy settles within seconds. An older issue the rollout does not own, such as node.disk_low, is kept as evidence. The verify ignores the task VM host codes node.task_vm_orphaned, node.task_vm_missing, node.task_vm_inspection_failed, node.task_vm_parked, and the stray codes node.task_vm_stray_*, because task VMs come and go during a rollout. The presence check fails only when the agent reports another version. Without an active websocket role, or without any report from the Node’s agent, Doctor’s binary check stands in for it.

​
Halt and resume

At the first failed Node, the rollout stops. The Nodes after it stay as they were. The rollout record becomes halted with the Node, the error code, the message, and the step evidence: the CLI install result, the self-update report or its exit code and output, the footprint result, and the verify issues. The Gateway raises the rollout_halted release alert once, with the target fleet and the release id fleet-rollout-<id>, and stores the receipt on the record. A Node whose self-update stays incomplete for 6 visits in a row, 30 minutes of catch-ups, raises rollout_stalled once, with the reasons in the step evidence. A deferred or unreachable visit in between keeps the count. It does not halt the rollout. A halted rollout blocks every later rollout and the catch-up. Fix the cause, then resume:
fleet:rollout:resume sets the failed Node back to pending, starts the service, and returns. A skipped Node stays skipped in this rollout; the next desired state visits it again. When the Gateway moved to a newer commit since the halt, the halted rollout becomes superseded, and a rollout of the newest desired state opens with the skip carried over.

​
Catch-up

After the rollout, every run visits some Nodes again, one at a time and with the same rules: So a Node that was offline is converged within 5 minutes after it returns. Doctor reports a lagging Node of the rollout set as node.release_lag while the rollout is on. A Node outside the set, such as a task sandbox, never gets it. observed says why: no rollout yet, not in the rollout, the Node’s outcome, or drifted. Doctor reads only the desired state that a run already resolved; it never asks Git or GitHub.

​
Nodes being updated

Each Node in GET /api/v1/nodes and GET /api/v1/nodes/{node} has an updating field. It is null unless the Gateway is updating that Node now: The other id field is null. A visit clears its finished_at when it starts, so a catch-up visit of a converged Node counts too. The rollout visits one Node at a time, so at most one Node is fleet_rollout. The Gateway Node is never in the rollout set, so one Node is never both. A queued release record does not count, because nothing changed yet. A visit counts only while a run holds the fleet lock, so a long visit stays updating for as long as it runs. A run that dies leaves its visit open. Nothing renews the lock then, so it runs out within 20 minutes and the Node stops reading as updating. The next run ends every open visit before it visits a Node, and broadcasts each of those Nodes. A dead running release record ends as interrupted, as Release records describes. The list reads every Node’s state with two queries, one for the visits in progress and one for a running release. It reads the fleet lock only when a visit is open. The Gateway broadcasts node.updated with the new value when a visit starts and ends, and when a release record starts running and ends. The web app shows the Node as updating.

​
Limits

Automatic releases and the fleet rollout have these limits.

​
Release limits

A release that fails after its migrations ran pauses automatic releases until an operator acts. A deploy never switches back automatically after its migrator ran. Orbit cannot prove that older code runs on the resulting schema. Pre-migration snapshots stay on the Gateway host, in ORBIT_HOME/backups. Orbit does not copy them off the host. Keep a complete state set elsewhere. The release steps up to the switch run the code of the release that was current. A change to those steps takes effect from the release after the one that ships it. The deploying code also writes the release record and raises its alerts. So when the release that adds the Gateway Node’s agent update fails that update, its record shows the failure, but no alert is raised. Later releases alert.

​
Rollout limits

The rollout updates only the Nodes of the rollout set. Each release’s handoff updates the Gateway Node’s agent. An operator updates an operator machine or a macOS Node with orbit self-update, and the CLI tells the operator about a newer release. The rollout visits one Node at a time, so it takes longer as the fleet grows. A halted rollout blocks every later rollout until an operator resumes it. The Gateway pushes the rendered footprint over SSH, such as the Caddyfile or the tmpfiles rule of an app-dev Node. Adding or removing an artifact changes the footprint digest of each Node it applies to. The next rollout converges the artifacts in the current footprint list. orbit self-update replaces only the CLI and the agent. Caddy, cAdvisor, the FPM exporter, Prometheus, Grafana, Plausible, and Reverb keep their own update paths.

​
Not built

Each of these needs its own decision:
  • copies of the pre-migration snapshots off the Gateway host;
  • an expand and contract policy for Gateway migrations;
  • orbit self-update that applies the rendered footprint on the Node, so the Gateway stops pushing configuration over SSH;
  • automatic updates on operator Macs;
  • a macOS Node agent;
  • a self-update that an agent signal triggers instead of SSH (The agent only observes).

​
Why it works this way

These reasons explain the design. Check them before you propose a change.

​
The Gateway pulls from GitHub

The Gateway is private, so GitHub cannot reach it, and the GitHub App has no webhooks. A CI job that pushes to a deploy endpoint with a GitHub OIDC identity was rejected: it needs a new trusted path into the Gateway and new credential handling, and an earlier attempt stalled on exactly that. A CI deploy job that connects to the Gateway over SSH was rejected too: the CI runners are kept off the Gateway on purpose, so CI jobs cannot reach the control plane (Self-hosted Gateway runner). Pulling through the existing GitHub App adds no network path, credential, or Node grant.

​
Code decides what ships

Whether a commit passed its checks, and whether a release serves, are deterministic questions. Required checks, verify, and smoke answer them, not an agent’s judgment. A deploy bot was planned for this duty and rejected for that reason. Judgment is needed only after a pause, and the operator gives it. Every step is a gateway:release:* command that an agent can run and a person can read. An update in place serves a half-changed tree while git checkout and composer install run, because the Gateway’s PHP-FPM pool checks every script for changes on each request. It also needs a maintenance window. A release that is built beside the live one and switched with one rename never serves a mix of two commits. The in-place procedure is gone, not kept beside the release commands, so there is one way to update a Gateway.

​
Releases install no development packages

A release runs the Gateway, its artisan commands, and the CLI for the smoke test. None of them needs a development package. The Gateway and the CLI register Boost only when its classes exist, and the CLI binary already ships from a no-dev install. Leaving those packages out cut the two vendor/ directories from about 277 MB to 115 MB and the install from about 8 s to 5 s on a warm Composer cache, measured on a build host on 8 Oct 2026. A release cannot run composer test or artisan boost:*; use a checkout for those. Copying the current release’s vendor/ into the next one when composer.lock is unchanged was measured and rejected. The copy, the write bit it needs back, and the composer install that still runs to rebuild the autoloader took 6.6 to 7.2 s against 4.6 to 5.1 s for a fresh no-dev install. Hard links are not an option, because Composer rewrites autoload files in place.

​
No PHP-FPM restart

A PHP-FPM restart or reload ends the requests in flight, and a Gateway request may run 600 seconds. Caddy resolves the release link for each request instead, so a running request finishes on its release and the next one runs the new release.

​
Pause after the migrator

Older code on a newer schema is untested, and it can damage live data without a visible error. So a release that fails after its migrations pauses, and a person chooses between a forward fix, a code rollback, and a restore from the snapshot. Refusing automatic releases for commits with migrations was rejected: most commits that change the Gateway ship a migration, so automation would rarely run. The snapshot and the pause cover the risk.

​
Laravel owns migrations

A Gateway deploy follows a Laravel production deploy: take the pre-release database snapshot, run php artisan migrate --force, and switch code. The snapshot implementation, retention, permissions, and protection of a paused release’s clean copy stay the safety net. Rollback switches code; restoring a database snapshot is an explicit operator decision. This supersedes the migration-crossing part of ADR 0201. Release 200 (fa72c0f1e) was blocked because nckrtl/horizon-new-dawn registers a migration with loadMigrationsFrom(). Laravel had applied it at batch 145 during release fbeeb79d5, but the guard looked only in the Gateway’s own migration directory. Identical package code in every release therefore looked incompatible. Extending the guard to vendor directories is rejected: it duplicates Laravel’s migration discovery and still cannot prove that code and schema are compatible. Deploy leaves migration discovery to Laravel. Every deploy runs the migrator, including a release with only vendor migrations or none pending. A failure after the migrator runs pauses the release. A person chooses recovery.

​
The Gateway is not an Orbit Instance

The Instance pipeline deploys to Nodes over SSH and runs one PHP-FPM pool per Instance. The Gateway is the control plane on its own host. Its release cannot depend on a healthy Gateway to run it.

​
The rollout runs one command on each Node

orbit self-update is the one node-local updater, on a Node and on an operator’s Mac. A rollout in which the Gateway runs each update step over SSH would work, but it keeps the update logic in the Gateway. With one node-local command, a later move to another trigger changes only the trigger.

​
One Node at a time, and a halt

Converging Nodes in parallel is faster, but a bad release would break many Nodes at once. A failed Node halts the rollout, because the next Node would most likely fail the same way, and one broken Node is easier to repair than a fleet. A fleet failure never rolls the Gateway back, because the Gateway already verified and serves. An unreachable Node changed nothing, so it does not halt the fleet; the catch-up converges it. Halting on it would let one offline Node block every other. A missing CLI release is the normal state for a few minutes after each deploy, so it waits instead of failing. The rollout is off by default, so a Gateway release that adds the units does not reach Nodes before an operator prepared them.