Skip to main content

​
Gateway queue

The Gateway scheduler only dispatches. Each scheduled command that does work, such as tasks:tick, becomes a queued job, and Laravel Horizon runs the jobs. A long build therefore never delays a tick, and a slow tick never delays the next minute of the schedule.

​
Connection

The queue connection gateway uses the Redis driver. Its Redis server is a Redis Database connection that Orbit already has, such as the Valkey on the services Node (services-valkey). Set these keys in /home/orbit/shared/gateway.env, then run a release. The release caches them and creates the horizon Process. Do not only apply the env change: the scheduler would then dispatch to the queue before Horizon runs. The Gateway reads the host, port, and password of that connection from its records when a process first uses Redis. When the key names no Redis connection, the Gateway logs an error and every dispatch to the queue fails. Every key starts with orbit-gateway:, and Horizon’s own keys with orbit-gateway-horizon:. The Redis client is Predis, so the Gateway needs no PHP Redis extension. services-valkey listens on the WireGuard address of services and has no password. Only WireGuard peers reach it. A Valkey ACL user with a password for the Gateway is a follow-up.

​
Queues

Laravel Horizon runs three queues on the connection. Each has its own supervisor, so a long build never holds up a short job: The task-vms jobs run one at a time and in order, so no two jobs change the same VM at once. A task-vms job runs at most 1500 seconds, and any other job at most 1700 seconds. A job that a worker lost, for example when its process was killed, runs again after 1800 seconds. An idle worker waits on the queue with a blocking read, so a job starts within about a second when a process is free. One builds process runs one job at a time in the fleet. Base image steps and template builds take turns. The image build row allows one running base image build per host. Syncs and template builds keep their own host locks until they have their own chains.

​
Scheduled work

The scheduler dispatches each of these as a unique job. While the job waits or runs, the scheduler dispatches no second one, so a slow tick never stacks up: The job runs the command in its own PHP process, as the scheduler did, so a command keeps no state from an earlier run. A command that exits nonzero is logged as an error. tasks:tick keeps its own lock of 300 seconds. The “unique for” time only matters when a worker is lost: the lock then frees itself after that time. The task VM jobs that the scheduler queues, the host sweep, the pool refill, and the nightly refreshes, go to the same connection. The document cleanup commands stay in the scheduler, because their permit belongs to the scheduler’s start.

​
Without a queue

While ORBIT_GATEWAY_QUEUE_CONNECTION is empty, the scheduler runs the commands above itself, with their overlap locks, as before, and the Gateway creates no horizon Process. Task VMs need the queue. While task VMs are enabled, or the UpCloud token file is set, each due run of the host sweep, the pool refill, or a nightly refresh fails with an error that names the key, instead of being skipped. A task VM claim fails the same way.

​
Horizon

Horizon runs as the Gateway Node’s Process named horizon, beside the scheduler Process. It is a systemd Process with restart policy always. It runs php artisan horizon with the scheduler’s PHP binary, as orbit, in the scheduler’s directory, the stable path /home/orbit/orbit/apps/gateway. Read its output with orbit process:logs <id>. The serving phase of a release’s runtime handoff creates the Process and starts it, when ORBIT_GATEWAY_QUEUE_CONNECTION is set and no Process of the Gateway Node runs artisan horizon. So the first release after you set the key starts Horizon. When the handoff cannot create or start the Process, it fails with gateway.release_units_failed. The Gateway never removes the Process. After you empty the key, destroy it with orbit process:destroy <id>.

​
Dashboard

Open https://gateway.orbit/horizon in a browser on a machine that is a WireGuard peer with Gateway access, the same access that the Orbit CLI needs for Gateway commands. The dashboard is Horizon New Dawn, a React interface for Horizon. Any other address gets 403. The dashboard can retry, cancel, and clear jobs, and pause queues, so anyone with Gateway access can change the queue. The Gateway keeps no browser session, so the dashboard has no CSRF token. Laravel’s request forgery check runs in origin-only mode instead: a request that changes something passes only when the browser sends Sec-Fetch-Site: same-origin. Any other such request gets 403. Bulk actions run as jobs on the tasks queue. Each release publishes the dashboard’s compiled assets during composer install in prepare, with php artisan horizon-new-dawn:assets. The first release with the dashboard runs its migration, which adds the table horizon_new_dawn_batch_metadata. The scheduler runs horizon:snapshot every five minutes for the dashboard’s metrics. The Redis server needs maxmemory-policy noeviction, so that it never drops a queued job to free memory. services-valkey has it. It has no maxmemory, so it can use the memory of services; a limit for the Gateway’s use is a follow-up.

​
Releases

A release moves Horizon in the serving phase, right after the switch, and the old database jobs in the scheduler phase, after the scheduler’s drain. The handoff records the Horizon step as horizon: horizon:terminate reaches only a running master, so a stopped Horizon is started instead. The handoff never restarts an active unit, because a restart ends its running jobs. horizon:terminate sends SIGTERM to the Horizon master. Horizon does not use fast termination: the master waits until each worker has finished its running job, on the release that job started on, and then exits. The Process’s restart policy then starts Horizon again, on the new release. A new task-vms worker therefore never starts while an old one runs, and task VM jobs stay in order. The cost is the wait. A release during a base image build waits for one short step, not the whole build. The build is a chain of jobs, and the next step runs on the new release. Until the running jobs end, no new job starts on any queue, and tasks:tick waits too. Other operations, such as syncs and workspace template builds, still run as whole jobs until they have their own chains. The master waits at most the longest supervisor timeout, 1700 seconds, and then exits. The release does not wait for the new master. A failure of the step fails the handoff with gateway.release_units_failed, and the release switches back. The scheduler phase restarts every other Gateway Process in the Gateway application with systemd, but never the horizon Process. Do not restart Horizon with orbit process:restart, and do not stop it with orbit process:stop while a job runs. Both stop its unit, and systemd kills a job that still runs 90 seconds after the stop, such as a build. To restart Horizon, run php artisan horizon:terminate. The old database queues are task-vms, whose jobs move to task-vms, task-vm-git, whose jobs move to tasks, and task-vm-templates and task-vm-images, whose jobs move to builds. A delayed job keeps its time, and a job keeps its unique lock. Its attempts count from zero again. No job is lost. The handoff records the count as queue_moved. When jobs wait there and ORBIT_GATEWAY_QUEUE_CONNECTION is empty, the move fails with gateway.queue_move_failed and keeps them.

​
Switch back to an older release

A release from before Horizon has no horizon command. After a switch back to it, the older release’s handoff restarts the horizon Process, which then fails at each start until systemd gives up. The Process shows the failure in orbit process:list. Stop it with orbit process:stop <id>. The older release’s scheduler runs its own workers on the database queues meanwhile, and jobs left in Redis wait. The next release with Horizon starts the Process again, and Horizon works the jobs left in Redis. A job left in Redis keeps its unique lock in the Gateway cache. The older release then queues no second copy of that job, for example no second destroy of the same VM, until the lock expires; a lock without an expiry stays. To let the older release queue it again, remove the lock as orbit in /home/orbit/orbit/apps/gateway:
The key is laravel_unique_job: followed by the job class and its unique id, the task VM id for task VM jobs. When the newer release goes live again, it works the jobs left in Redis.

​
Jobs across releases

A job queued by one release can run on the next one, and during a release, on the one before. A change to a queued job’s class, its constructor properties, or a table that a job reads, must work with both releases for one release. Remove the old form in a later release.

​
When Redis is down

The queue depends on its Redis server, but the Gateway keeps running without it:
  • The scheduler logs each failed dispatch as an error and frees the job’s unique lock, so the next run dispatches it again. The scheduler keeps running.
  • A running Horizon reports each failed read and keeps running. It works the queued jobs again as soon as Redis answers.
  • A Horizon start fails while Redis is down.
  • A request that queues a job, such as a task VM claim, fails with an error.
systemd starts a failed Horizon again after 2 seconds. After several failed starts in a row, systemd stops trying, and the Process shows the failure. Start it again with orbit process:start <id>; the next release also starts it. A job that Redis lost runs again only when its owner queues it again. For example, the tick queues a publish again after its backoff.

​
Operations

These commands inspect and control Horizon on the Gateway. Run each php artisan command as orbit in /home/orbit/orbit/apps/gateway.

​
Not built

Doctor does not report a stopped Horizon or an old last_tick_at yet. Until it does, check the horizon Process and orbit tasks:status when task groups stop moving.

​
Why it works this way

The scheduler used to run its work, and the task VM workers, as foreground commands. A foreground command holds the scheduler’s run for that minute, so a slow tick or a VM job could hold a publish for minutes. With dispatch only, the scheduler’s run takes milliseconds, and each queue’s workers decide when a job starts. Horizon needs Redis. The fleet already runs Valkey on the database role, so the Gateway uses that registered connection instead of a second server on its own host. Orbit’s records keep its address, as for every other Database connection.