Operations

Every command on this page names its box. That is not decoration. An audit of the fleet's own registry on 2026-08-16 ran the seventy-two of its then-seventy-three self-checks that were safe to run from anywhere, from the Architect box: twenty-six failed — and all twenty-six passed when re-run on the server. The registry was accurate and location-blind, so following its instruction from the wrong machine manufactured twenty-six false failures.

Two conventions used throughout:

• # on ETH — the Architect's WSL box, 100.103.62.104.
• # on novaserver — 100.71.25.43. If you are not already on it, prefix with ssh nova@100.71.25.43.

Never write the bare alias — ssh with nova as the whole host, no user and no IP. There is no ~/.ssh/config on the Architect box to expand it, so it fails with Name or service not known. Thirteen files in the fleet's internal docs still write it that way; none of them work.

Reaching a box

# from anywhere on the tailnet — the server
ssh nova@100.71.25.43

# from anywhere on the tailnet — a specific master's account
ssh nova@100.71.25.43 'sudo -u cloud -H bash -lc "whoami; pwd"'

nova here is a unix account that happens to share a name with one of the masters — it is the SSH entry point for the box, and sudo -u <account> is how you become any of the others.

The account is not the lane, and this page used to say it was. A master's board lane, its GitHub identity and its unix account are three separate names that usually agree and are not required to. Which accounts exist and which lanes the scheduler routes to are both read from the fleet's own files, here:

There is also a browser terminal per master, read-only, served by ttyd on the tailnet:

# on novaserver — confirm the four viewports are listening
ss -ltn | grep -E '100\.71\.25\.43:781[0-3]'

Dispatching work

Dispatch is how a master receives an issue. Starting the run is the assignment — there is no separate claim step.

# on ETH, from the knowledge repo
bash _infra/scripts/dispatch.sh <master> <issue#> [owner/repo]

The repo argument defaults to utopia-models/knowledge. The script runs three pre-flight guards, syncs that master's clones, launches a detached transient systemd unit, and returns. The guards, their exit codes and their escape hatches are on The dispatch runtime. The run posts ▶ started on the issue, and its exit trap posts ✔, ✘, ⚠ or ⛔ with the pull-request link.

Before you dispatch: is the master dark?

# on ETH — the whole table
bash _infra/scripts/dispatch.sh --quota

Real output, 2026-09-04 — shown as a sample of the shape, not as a current reading:

win    clear
nova   clear
cloud  clear
dust   clear

A refused master replaces clear with the reason it was refused. That row, observed on 2026-08-16, is the shape to recognise:

nova   dark   resets=unknown provider=claude source=entitlement-capture

# on ETH — machine-readable: just the dark names
bash _infra/scripts/dispatch.sh --quota-dark

# on ETH — would this specific master be refused? exit 0 = dispatch, 9 = refused
bash _infra/scripts/dispatch.sh --preflight-quota cloud; echo "exit=$?"

# on ETH — dispatch anyway (e.g. the account was just rotated)
DISPATCH_IGNORE_QUOTA=1 bash _infra/scripts/dispatch.sh cloud 1234

resets=unknown is the state with no recovery path: the provider named no reset time, so nothing brings that agent back on its own. A reset time in the past counts as clear.

Why is nothing being dispatched?

If the board has open work and the scheduler is dispatching none of it, the scheduler will tell you why, per issue:

# on novaserver, from the knowledge clone
bash _infra/scripts/sweep-dispatch.sh --explain

Each rejected issue prints its own reason — lane-not-in-masters, has-hold-label, is-a-grouper:has-open-subissues, last-marker-is-terminal, parent-closed, is-parked. A large open count with zero dispatchable is almost always routing, not capacity: an issue with no lane, or on a lane no master serves, can never be picked up however idle the fleet is.

Is an agent actually alive?

getent passwd nova is not this check. It succeeds on an agent that cannot run a single turn. To answer the question you must exercise the artifact:

# on novaserver — the real probe. Anything other than a normal completion is the finding.
sudo -u nova -H claude -p 'reply with the single word: alive'

# on novaserver — what the agent itself last recorded
sudo -u nova -H cat /home/nova/.claude/state/quota-status.json

An api_error_status: 403 here means entitlement was withdrawn at the account, not that anything on the box is broken — the binary, the credentials file and the unix account are all still perfectly healthy. "resetsAt": null in that state file means there is no timer that restores it.

Watching a run

Runs are transient systemd units under the master's user manager. The name is built in dispatch.sh, so it is read from there rather than described from memory:

# on novaserver — what is running right now for a given master
sudo -u cloud -H XDG_RUNTIME_DIR=/run/user/$(id -u cloud) systemctl --user list-units 'task-*'

# on novaserver — the log for one run
sudo -u cloud -H journalctl --user -u task-5882-1786804561-2943 --no-pager

The board is the other half of this, and usually the better half: the run's own ▶ and ✔/✘ comments are on the issue, and they are the fleet's definition of liveness.

Secrets

# on any box — read one value (never echo it)
V=$(bash ~/.claude/scripts/infisical-get.sh fleet-infra SOME_KEY prod); echo "len=${#V}"

# on any box — list every key in a project, so "it does not exist" is a real claim
bash ~/.claude/scripts/infisical-get.sh --keys fleet-infra prod /

# on any box — write a value; the value goes on STDIN, never in argv
printf '%s' "$NEW_VALUE" | bash ~/.claude/scripts/infisical-set.sh fleet-infra SOME_KEY prod

Passing a secret as an argument is rejected by a guard: argv lands in shell history and in the process table. Before concluding a credential is absent, list the whole project — the fleet's recurring failure is not a missing credential but a grep for the wrong key name.

Checking services and timers

# on novaserver — system-level failures
systemctl --failed

# on novaserver — a master's own user-manager failures
sudo -u nova -H XDG_RUNTIME_DIR=/run/user/$(id -u nova) systemctl --user --failed

A clean --failed is weaker evidence than it looks. A dead timer never fails. Enumerate the timers themselves and look at the last and next columns:

# on novaserver — every scheduled timer for a master, with last/next fire times
sudo -u nova -H XDG_RUNTIME_DIR=/run/user/$(id -u nova) systemctl --user list-timers --all

Two traps that produce a clean bill of health on a broken box:

1. An enabled timer whose service is not-found never runs and never fails.
2. Cron is a second scheduling layer. Masters carry crontab entries with no systemd timer at all. One sudo -u line per account the fleet actually dispatches to, generated so that none of them can name an account that does not exist:

Deleting a scheduled unit — a two-half operation

Removing a unit from an agent's brain repo does not stop it running: ~/.config/systemd/user/ holds copies, not symlinks. A real deletion cost the fleet a spurious P0 when the orphaned timer kept firing against a script that no longer existed and exited 203/EXEC. Both halves:

# on novaserver, as the owning master — the on-box half
systemctl --user stop <unit>.timer <unit>.service
systemctl --user disable <unit>.timer
systemctl --user reset-failed <unit>.service
rm -f ~/.config/systemd/user/<unit>.{service,timer}
systemctl --user daemon-reload

...and then delete it from the brain repo and push, or a sync job will put it back.

Checking a public surface

A status code alone is never the answer — see Surfaces for why a 404 has at least three different meanings.

# from anywhere — the code, and whether anything redirected
curl -s -o /dev/null -w '%{http_code} redirects=%{num_redirects} final=%{url_effective}\n' -L https://<host>/

# from anywhere — headers, to see which edge served it
curl -sI https://<host>/ | grep -iE 'server|x-vercel|cf-ray|x-matched-path'

An x-vercel-error: DEPLOYMENT_NOT_FOUND on a 404 means no deployment is attached to that hostname, which is what both a deliberately retired surface and an accidentally dead one look like. The difference is a decision record, not a header. A 530 with Cloudflare error 1033 means a tunnel origin is unreachable — that one is never deliberate.

A 200 does not prove a page is public. An SSO gate redirects to a login page that itself returns 200, so assert redirects=0 and check the final URL, not just the code.

Shipping: merging when a check is red

Nothing gates main. Branch protection and rulesets both return 403 on this plan tier, in every repo, so every check is information, never a stop.

# from anywhere with gh auth
gh pr merge <n> --squash --admin

If that fails, read the actual error:

• CONFLICTING / DIRTY — rebase and force-push your own topic branch by name:

  git fetch origin && git rebase origin/main
  # resolve, then:
  git push --force-with-lease origin HEAD:<your-branch>

  Never a bare git push --force: it takes whatever HEAD happens to be. One such command once moved 501 commits off a shared clone's main, and nothing local showed it.

• A real build, typecheck or deploy failure — that failure is real. Fix the code.

• Anything else — retry with --admin.

When the secret scanner cannot run

The secret scan is the one check treated as binding. If the workflow itself cannot start, run it locally against the same pinned version rather than skipping it:

# on any box, from the repo root
VERSION=8.24.3
curl -sL "https://github.com/gitleaks/gitleaks/releases/download/v${VERSION}/gitleaks_${VERSION}_linux_x64.tar.gz" \
  | tar xz -C /tmp gitleaks \
  && /tmp/gitleaks git . --no-banner --redact --config .gitleaks.toml \
     --log-opts "--no-merges <base>..<head>"

Clean means merge. Findings mean fix, rescan, then merge.

Verifying a claim about the fleet

The habit that produces the fewest wrong answers:

1. Read the source, do not recall it. Every exact value — a path, a port, a key name, a count — comes from the file or the probe this session, never from memory.
2. Exercise the artifact, not the build. A green build proves the source compiles. Curl the deploy, run the binary, render the page.
3. State the box. A result without the machine it came from is not reproducible.
4. Prefer the negative control. Confirm your check can fail — run it against something you know is broken. A check that passes on everything is measuring nothing.