Skip to main content

​
Remote file writes on uutils coreutils

​
Problem

A convergence that writes a file on a Node succeeds once and fails on every later run. The remote command prints only install: No such file or directory, although the directory exists. In Metrics, the codes are metrics.configuration_publish_failed and metrics.exporter_configuration_failed.

​
Cause

Ubuntu 26.04 ships uutils coreutils, not GNU coreutils. Its install refuses an existing destination when the source is /dev/stdin:
Creating a new file works, and overwriting from a regular file works. Only an overwrite from standard input fails, so the failure shows one convergence after the code that causes it.

​
Solution

Never point a remote install from standard input at a live path. Remove any stale <path>.orbit-candidate, write the candidate, and then mv -fT it onto the target. The move is atomic, which running containers and systemd units need anyway. Metrics SSH publishers, MetricsExporterSshExecutor::publishConfiguration(), and NodeAgentSshExecutor::publishFile() use this sequence. RemoteProcessRuntimeManager writes each unit to /etc/orbit/systemd-candidates, verifies the candidate with systemd-analyze, and then moves it into place. Metrics exporter SSH lifecycle operations now delegate file publication to the shared publisher. Before staging a systemd candidate, RemoteProcessRuntimeManager reads the existing unit and checks its Process ownership marker. If its content matches a recorded successful activation, or belongs to a legacy Process with no pending activation, and its active or inactive state matches the desired state, convergence returns without writing a candidate, reloading systemd, or restarting the Process. Changed unit content still follows the candidate-and-verify sequence; an unchanged specification does not justify skipping recovery of an incomplete update. The annotator server publishes a set of files rather than one configuration file. AnnotatorServerInstallation sends the Gateway files through standard input as JSON to a fixed Python program. The program takes a Node-local file lock, writes a new immutable release directory, and atomically switches /opt/orbit/annotator/current to it. It never overwrites a live file with install /dev/stdin, and it retains earlier releases for running servers. The Gateway ships the vendored annotator files, a compressed injection bundle, and a manifest generated by bin/annotator-build; installation verifies them before publishing a release, so /inject.js cannot be silently omitted. The systemd unit still uses the candidate-and-verify sequence above. Sibling environment projection uses that sequence without activation, preserving sleeping or running services.

​
Limits

The problem affects only /dev/stdin sources. install -d, mode changes, and regular-file sources work. Node agent converge closes permissions on an existing Instance .env with a mode change, not file publication. For a nested Laravel app, that operation targets the application directory, such as <checkout>/apps/site/.env; it does not need the candidate-and-move sequence. A numeric container identity, such as Grafana’s 472, also needs a separate chown, because install -o and -g refuse an identity without a passwd entry. The Node agent uses this sequence only on Linux. A macOS install or repair returns node.agent_unsupported before SSH, and removal returns without a remote command.

​
Verification

Converge the same role twice over existing configuration, which is the case that fails. Unit tests cover the sequence in apps/gateway/tests/Unit/Infrastructure/Metrics/MetricsSshExecutorTest.php and MetricsExporterSshExecutorLifecycleTest.php; apps/gateway/tests/Feature/Infrastructure/Processes/RemoteProcessRuntimeManagerTest.php checks candidate verification and atomic publication of systemd units, and checks that repeated convergence of an unchanged running or stopped Process writes no candidate and issues no restart. apps/gateway/tests/Feature/Infrastructure/Processes/AnnotatorServerInstallationTest.php installs the annotation files twice, verifies that an identical install reuses its release, and verifies that a changed install switches the symlink while keeping the old release.