Operating

Operations

Runbooks for the Utopia fleet. Every command runs exactly as written and names the box it runs on, because a command that works from one machine and fails from another is not a runbook.

Updated

Prompt 1

Learn to verify a fleet the way its runbooks do

Reads this page, explains why every command names its box and why a clean status is weaker than it looks, then runs the public-surface probe live. Writes nothing.

Prompt 2

Make your own runbooks box-honest and falsifiable

Audits your runbooks and scheduled jobs for location-blind commands, liveness that is only existence, orphaned timers and secrets in argv, then proposes the rewrites. Writes only after you confirm.

Every command on this page names the box it runs on, for a measured reason. On 2026-08-16 we audited the fleet's own registry. At the time it had seventy-three self-checks, and seventy-two of them were listed as safe to run from anywhere. We ran those seventy-two from the Architect box, and twenty-six failed. All twenty-six passed when we ran them again on the server. The registry was accurate but never said where to run anything, so following it from the wrong machine produced twenty-six false failures.

The page uses two labels throughout:

  • # on ETH means the Architect's WSL box. This site writes its tailnet address as 100.x.x.x.
  • # on novaserver means the fleet's server, written <tailnet-ip> below. If you are not already on it, prefix the command with ssh nova@<tailnet-ip>.

Never write the bare alias, meaning ssh with nova as the whole host and no user or IP. The Architect box has no ~/.ssh/config to expand it, so the command fails with Name or service not known. Thirteen files in the fleet's internal docs still write it that way, and none of them work.

Reaching a box

# from anywhere on the tailnet — the server
ssh nova@<tailnet-ip>

# from anywhere on the tailnet — a specific master's account
ssh nova@<tailnet-ip> 'sudo -u cloud -H bash -lc "whoami; pwd"'

Here nova is a unix account that happens to share its name with one of the masters. It is the SSH entry point for the box, and you use sudo -u <account> to become any of the other accounts.

The account is not the lane, although this page used to say it was. A master has a board lane, a GitHub identity and a unix account. These are three separate names. They usually match, but nothing requires them to. The table below reads both the existing accounts and the lanes the scheduler routes to from the fleet's own files:

The scheduler routes to 4 lanes: win · nova · cloud · dust — read from MASTERS="win nova cloud dust".

Accounts that exist on a box and are not in that set: poppi (personal-agent) · scout (always-on-agent) · peaky (always-on-agent) · ronin (always-on-agent) · wingman (always-on-agent) · atlas (always-on-agent) · herald (always-on-agent) · shred (personal-agent). An account is not a lane: an issue on a lane no master serves is structurally undispatchable however idle the fleet is.

Generated at request time (2026-11-01T04:43:51.985Z) from utopia-models/knowledge @ main — _infra/scripts/sweep-dispatch.sh (the MASTERS= assignment — the scheduler rejects any Todo issue whose Lane is not in this set); wi/registry/organization.yaml (the agent records those lane names resolve to). Nothing here is stored: this page holds no copy of these values.

Each master also has a read-only browser terminal, which ttyd serves on the tailnet:

# on novaserver — confirm the four viewports are listening
ss -ltn | grep -E ':781[0-3]'

Dispatching work

Dispatch is how a master receives an issue. Starting the run assigns the issue, and there is no separate claim step.

# on ETH, from the knowledge repo
bash _infra/scripts/dispatch.sh <master> <issue#> [owner/repo]

The repo argument defaults to utopia-models/knowledge. The script runs three pre-flight guards, syncs that master's clones, launches a detached transient systemd unit, and returns. The dispatch runtime documents the guards, their exit codes and how to override each one. The run posts ▶ started on the issue. When it exits, its exit trap posts ✔, ✘, ⚠ or ⛔ with the pull-request link.

Before you dispatch, check whether the master is dark

# on ETH — the whole table
bash _infra/scripts/dispatch.sh --quota

This is real output from 2026-09-04. It shows the format, not a current reading:

win    clear
nova   clear
cloud  clear
dust   clear

For a refused master, the row shows the reason instead of clear. We saw this row on 2026-08-16, and it is the format to recognise:

nova   dark   resets=unknown provider=claude source=entitlement-capture
# on ETH — machine-readable: just the dark names
bash _infra/scripts/dispatch.sh --quota-dark

# on ETH — would this specific master be refused? exit 0 = dispatch, 9 = refused
bash _infra/scripts/dispatch.sh --preflight-quota cloud; echo "exit=$?"

# on ETH — dispatch anyway (e.g. the account was just rotated)
DISPATCH_IGNORE_QUOTA=1 bash _infra/scripts/dispatch.sh cloud 1234

resets=unknown means there is no recovery path. The provider gave no reset time, so nothing brings that agent back by itself. A reset time in the past counts as clear.

Why is nothing being dispatched?

If the board has open work and the scheduler dispatches none of it, ask the scheduler. It gives the reason for each issue:

# on novaserver, from the knowledge clone
bash _infra/scripts/sweep-dispatch.sh --explain

Each rejected issue prints its own reason, for example lane-not-in-masters, has-hold-label, is-a-grouper:has-open-subissues, last-marker-is-terminal, parent-closed or is-parked. When many issues are open and none can be dispatched, the cause is almost always routing, not capacity. No master will ever pick up an issue that has no lane, or whose lane no master serves, however idle the fleet is.

Is an agent actually alive?

getent passwd nova does not answer this. It succeeds for an agent that cannot run a single turn. To answer the question, make the agent do real work:

# on novaserver — the real probe. Anything other than a normal completion is the finding.
sudo -u nova -H claude -p 'reply with the single word: alive'

# on novaserver — what the agent itself last recorded
sudo -u nova -H cat /home/nova/.claude/state/quota-status.json

An api_error_status: 403 here means the provider withdrew the account's entitlement. Nothing on the box is broken. The binary, the credentials file and the unix account all still work. If the state file shows "resetsAt": null, no timer will restore the entitlement.

Watching a run

Each run is a transient systemd unit under the master's user manager. dispatch.sh builds the unit name, so the page reads it from that script instead of describing it from memory:

if [ "${DISPATCH_ALLOW_DUPLICATE:-0}" = "1" ]; then
  UNIT="task-${ISSUE}-dup-$(date +%s)-${RANDOM}"
else
  UNIT="task-${ISSUE}"
fi

An ordinary run’s unit is task-${ISSUE} — the last assignment above, and the one to search for. The longer form exists only behind its own flag. A unit name guessed from a process id or a timestamp will never match an ordinary run.

Generated at request time (2026-11-01T04:43:52.006Z) from utopia-models/knowledge @ main — _infra/scripts/dispatch.sh (every UNIT= assignment that names the transient systemd unit). Nothing here is stored: this page holds no copy of these values.

# on novaserver — what is running right now for a given master
sudo -u cloud -H XDG_RUNTIME_DIR=/run/user/$(id -u cloud) systemctl --user list-units 'task-*'

# on novaserver — the log for one run
sudo -u cloud -H journalctl --user -u task-<issue> --no-pager

The board is the other place to watch, and usually the better one. The run posts its own ▶ and ✔/✘ comments on the issue, and the fleet defines liveness by those comments.

Secrets

# on any box — read one value (never echo it)
V=$(bash ~/.claude/scripts/infisical-get.sh fleet-infra SOME_KEY prod); echo "len=${#V}"

# on any box — list every key in a project, so "it does not exist" is a real claim
bash ~/.claude/scripts/infisical-get.sh --keys fleet-infra prod /

# on any box — write a value; the value goes on STDIN, never in argv
printf '%s' "$NEW_VALUE" | bash ~/.claude/scripts/infisical-set.sh fleet-infra SOME_KEY prod

A guard rejects any secret passed as an argument, because argv ends up in shell history and in the process table. Before you conclude a credential is absent, list the whole project. The fleet rarely lacks a credential. It often greps for the wrong key name.

Checking services and timers

# on novaserver — system-level failures
systemctl --failed

# on novaserver — a master's own user-manager failures
sudo -u nova -H XDG_RUNTIME_DIR=/run/user/$(id -u nova) systemctl --user --failed

A clean --failed proves less than it seems to, because a dead timer never fails. List the timers themselves and read the last and next columns:

# on novaserver — every scheduled timer for a master, with last/next fire times
sudo -u nova -H XDG_RUNTIME_DIR=/run/user/$(id -u nova) systemctl --user list-timers --all

Two traps make a broken box look healthy:

  1. An enabled timer whose service is not-found never runs and never fails.
  2. Cron is a second scheduling layer. Masters have crontab entries with no systemd timer at all. The lines below have one sudo -u per account the fleet actually dispatches to. They are generated, so none of them can name an account that does not exist:
# on novaserver — the other scheduler, for every account the fleet dispatches to
sudo -u win -H crontab -l
sudo -u nova -H crontab -l
sudo -u cloud -H crontab -l
sudo -u dust -H crontab -l

Generated at request time (2026-11-01T04:43:52.009Z) from utopia-models/knowledge @ main — _infra/scripts/sweep-dispatch.sh (the MASTERS= assignment — the accounts to probe); wi/registry/organization.yaml (each lane’s unix account). Nothing here is stored: this page holds no copy of these values.

Deleting a scheduled unit takes two steps

Removing a unit from an agent's brain repo does not stop it running, because ~/.config/systemd/user/ holds copies of unit files, not symlinks. One real deletion cost the fleet a false P0. The orphaned timer kept firing against a script that no longer existed, and it exited 203/EXEC. Do both steps:

# on novaserver, as the owning master — the on-box half
systemctl --user stop <unit>.timer <unit>.service
systemctl --user disable <unit>.timer
systemctl --user reset-failed <unit>.service
rm -f ~/.config/systemd/user/<unit>.{service,timer}
systemctl --user daemon-reload

Then delete the unit from the brain repo and push. Otherwise a sync job puts it back.

Checking a public surface

A status code alone never answers the question. Surfaces explains why a 404 has at least three different meanings.

# from anywhere — the code, and whether anything redirected
curl -s -o /dev/null -w '%{http_code} redirects=%{num_redirects} final=%{url_effective}\n' -L https://<host>/

# from anywhere — headers, to see which edge served it
curl -sI https://<host>/ | grep -iE 'server|x-vercel|cf-ray|x-matched-path'

An x-vercel-error: DEPLOYMENT_NOT_FOUND on a 404 means no deployment is attached to that hostname. A site we retired on purpose and a site that died by accident both look like that. Only a decision record tells them apart, not a header. A 530 with Cloudflare error 1033 means a tunnel origin is unreachable, and that is never deliberate.

A 200 does not prove a page is public. An SSO gate redirects to a login page, and that login page returns 200. Assert redirects=0 and check the final URL as well as the code.

Shipping: merging when a check is red

Nothing gates main. Branch protection and rulesets both return 403 on this plan tier, in every repo. So every check gives you information and never stops a merge.

# from anywhere with gh auth
gh pr merge <n> --squash --admin

If that fails, read the actual error:

  • If the PR is CONFLICTING or DIRTY, rebase and force-push your own topic branch by name:

    git fetch origin && git rebase origin/main
    # resolve, then:
    git push --force-with-lease origin HEAD:<your-branch>

    Never run a bare git push --force, because it pushes whatever HEAD happens to be. One such command once moved 501 commits off a shared clone's main, and nothing local showed it.

  • If a build, typecheck or deploy really failed, fix the code.

  • For anything else, retry with --admin.

When the secret scanner cannot run

The secret scan is the one check the fleet treats as binding. If the workflow itself cannot start, run the scan locally with the same pinned version. Do not skip it:

# on any box, from the repo root
VERSION=8.24.3
curl -sL "https://github.com/gitleaks/gitleaks/releases/download/v${VERSION}/gitleaks_${VERSION}_linux_x64.tar.gz" \
  | tar xz -C /tmp gitleaks \
  && /tmp/gitleaks git . --no-banner --redact --config .gitleaks.toml \
     --log-opts "--no-merges <base>..<head>"

If the scan is clean, merge. If it finds something, fix it, scan again, then merge.

Verifying a claim about the fleet

These habits produce the fewest wrong answers:

  1. Read the source, do not recall it. Take every exact value, such as a path, a port, a key name or a count, from the file or the probe in this session, never from memory.
  2. Exercise the artifact, not the build. A green build proves the source compiles. Curl the deploy, run the binary, render the page.
  3. State the box. Nobody can reproduce a result that does not say which machine it came from.
  4. Prefer the negative control. Confirm your check can fail by running it against something you know is broken. A check that passes on everything measures nothing.

On this page