Operations
Runbooks for the Utopia fleet. Every command runs exactly as written and names the box it runs on, because a command that works from one machine and fails from another is not a runbook.
Updated
Learn to verify a fleet the way its runbooks do
Reads this page, explains why every command names its box and why a clean status is weaker than it looks, then runs the public-surface probe live. Writes nothing.
PROMPT 1 of 2 — learn what this module teaches. You write nothing anywhere in this prompt. The only commands you run are the read-only curls below.
ASK EVERY QUESTION WITH the `AskUserQuestion` tool, with a RECOMMENDED option first. Not prose.
Fetch and read the whole page:
curl -s https://docs.utopiamodels.ai/docs/operating/operations.md
If the fetch fails, say what came back and stop.
1 — EXPLAIN IT BACK, UNDER THESE HEADINGS
EVERY COMMAND NAMES ITS BOX The audit that produced this rule — how many self-checks failed from the wrong box,
and how many of those passed on the right one. Why "accurate and location-blind" is
still broken.
IS IT ALIVE? Why `getent passwd` is not a liveness check, what the real probe exercises, and what a
403 means about the account versus the box.
A CLEAN --failed The two traps that produce a clean bill of health on a broken box.
DELETING A UNIT Why removing it from the repository does not stop it, and the two halves.
A 200 IS NOT PUBLIC What `redirects=0` and the final URL add that a status code cannot, and the difference
between DEPLOYMENT_NOT_FOUND and a 530.
SHIPPING Why a merge is never a human gate, the three failure branches, and why a bare
`git push --force` is banned.
VERIFYING A CLAIM The four habits, in order.
Quote the page where you lean on it.
2 — RUN THE PUBLIC-SURFACE PROBE, LIVE
Against the page you just read:
curl -s -o /dev/null -w '%{http_code} redirects=%{num_redirects} final=%{url_effective}\n' -L https://docs.utopiamodels.ai/docs/operating/operations
curl -sI https://docs.utopiamodels.ai/ | grep -iE 'server|x-vercel|cf-ray|x-matched-path'
Paste the raw output. From it alone: is this page public, which edge served it, and what would the output look like if an SSO gate sat in front of it? Then apply the page's fourth habit — PREFER THE NEGATIVE CONTROL — and run the first command against a URL you expect to fail (for example a path that does not exist on the same host). Show that the check can go red.
3 — QUIZ ME
Three questions with the `AskUserQuestion` tool, one at a time, four options each, correct answer not always first. Draw from: what `resets=unknown` means, why a secret is passed on STDIN, what an enabled timer with a `not-found` service does, and why `--force-with-lease` names the branch. After each answer, quote the line that settles it.
If `AskUserQuestion` is unavailable in this session, ask the same questions as plain numbered text in one message instead.
Then say exactly this:
"Module learned. Paste prompt 2 and I will audit your own runbooks against it."
Then stop.Make your own runbooks box-honest and falsifiable
Audits your runbooks and scheduled jobs for location-blind commands, liveness that is only existence, orphaned timers and secrets in argv, then proposes the rewrites. Writes only after you confirm.
PROMPT 2 of 2 — integrate this module into my own setup. Survey read-only first; propose; write only what I confirm.
HOW LONG THIS TAKES: a multi-step audit (7 steps, about 4 questions, writes only what you approve) — expect roughly 10–25 minutes across several turns. Running headless with a timeout? Budget for that, or start with a shorter prompt first. Cut off partway? Paste this prompt again — it resumes from its checklist rather than starting over.
ASK EVERY QUESTION WITH the `AskUserQuestion` tool, each with a RECOMMENDED option first and one line of why.
Reference, read it first:
curl -s https://docs.utopiamodels.ai/docs/operating/operations.md
FIRST — WRITE YOUR CHECKLIST TO A FILE
RESUMING? Before you write anything, check whether ~/OPERATIONS-INTEGRATE-CHECKLIST.md already exists. If it does, this prompt was interrupted: read it, tell me which steps are already ticked, and continue from the first unchecked step. Do not overwrite it, do not redo a ticked step, and do not re-ask a question whose answer it already records — the scope answer under ## Rules included, if it's there. If ## Rules has no Scope field, this checklist predates the scope question: say so plainly in your first output line and ask it now, before anything else. Only if the file does not exist, write it fresh as below.
Write ~/OPERATIONS-INTEGRATE-CHECKLIST.md and work from it:
# Operations module — integrate
- [ ] 1. Find my runbooks
- [ ] 2. Grade every command for its box
- [ ] 3. Grade every health check for liveness vs existence
- [ ] 4. Find scheduled jobs that cannot fail visibly
- [ ] 5. Find secrets in argv
- [ ] 6. Propose the rewrites
- [ ] 7. Write only what I confirmed
## Rules
- Read only until step 7. No stopping, starting, enabling or deleting any unit, timer or cron entry.
- I report that a credential is SET, never its value.
- Every finding carries the file and line, or the command and its raw output.
SCOPE — ASK ME BEFORE YOU SURVEY ANYTHING OF MINE
Before step 1, ask me this with the `AskUserQuestion` tool, with both options exactly as written:
"Should I treat any path as off-limits, or do you want me to have full access to survey and propose changes anywhere in this environment?"
A — Exclude .env*, credentials and CI/deploy config (.github/workflows/, deploy keys) from anything I read or propose touching. [RECOMMENDED for most readers]
B — No exclusions. Full access to survey and propose anywhere, including secrets and CI files.
This is my preference, not a policy: some people want their agent to reach everything, some do not, and neither answer is the correct one. Never assume it and never pick for me. My answer governs every read and every proposal for the rest of this prompt. Write my answer under ## Rules in the checklist as "Scope: A" or "Scope: B", so it survives compaction. It governs the SURVEY and PROPOSE steps only — nothing gets written until step 7, and there only the runbook diffs I confirm.
Before you ask, decide whether an answer can reach me at all. It cannot in a one-shot non-interactive run — `claude -p`, `codex exec`, `gemini -p`, a scripted or CI invocation — and you can tell without guessing: the `AskUserQuestion` tool is missing from your tool list there, while an interactive session has it. It also cannot if the question comes back immediately with no real answer. In either case do not ask, do not wait and do not re-ask: a question asked as plain text in a one-shot run ends the run with nothing done. This is the one case where you pick for me: treat A as my answer, record and hold it exactly as above, and make the first line of your output exactly: "No interactive scope answer available — defaulted to A (excluded .env*, credentials and CI/deploy config). To choose B, re-run this prompt in an interactive session and answer the scope question."
Only if the `AskUserQuestion` tool is in your tool list but fails when you call it, ask the same question as plain numbered text in one message instead.
1 — FIND MY RUNBOOKS
Every document or script a human or agent follows to operate something: RUNBOOK / OPERATIONS / OPS / DEPLOY / ONCALL files, a docs directory, a Makefile or justfile with ops targets, the agent instructions file (CLAUDE.md and anything it imports). List them with paths.
2 — DOES EVERY COMMAND NAME ITS BOX?
For every shell command in those runbooks: does it say which machine it runs on? Flag each that does not and that depends on location — a localhost port, a local path, a tailnet or private IP, an SSH alias. For every SSH host alias, check whether it resolves from THIS machine (`ssh -G <alias> | grep -E '^(hostname|user) '`); an alias that only expands on one box is the page's `Name or service not known` failure.
3 — LIVENESS OR EXISTENCE?
For every health or status check: does it EXERCISE the thing (make it do one real unit of work) or only prove it EXISTS (a user, a file, a process, a 200)? Quote each. For each HTTP check, does it assert redirects and the final URL, or only the code?
4 — WHAT CANNOT FAIL VISIBLY?
If systemd is present: `systemctl --user list-timers --all` and `systemctl list-timers --all`; for each timer, check its service with `systemctl --user status <svc> --no-pager 2>&1 | head -3` and flag any that is not-found. Then `crontab -l` — a second scheduling layer a clean `--failed` never sees. Compare both lists against what the repository says should be scheduled: a unit the repo deleted that still runs here is the page's two-half deletion, only half done.
5 — SECRETS IN ARGV
Search the runbooks and scripts for a credential passed as a command-line argument (`--token`, `-p`, `password=`, an `Authorization:` header built from a literal). Report file and line — never the value.
6 — PROPOSE
A table per category: finding · file:line · the rewrite. Rewrites follow the page: a `# on <box>` comment on every location-dependent command; a real-work probe beside every existence check; `redirects=0` asserted on HTTP checks; secrets on STDIN; and a negative control for each check so it can be seen to go red.
Ask me with the `AskUserQuestion` tool which category to apply first. RECOMMEND the box labels — mechanical, zero-risk, and the page measured them manufacturing twenty-six false failures.
7 — WRITE ONLY WHAT I CONFIRMED
Edit only the runbooks I confirmed, showing me each diff before applying it. Never touch a live timer, unit or crontab — for orphans, print the two-half removal commands for me to run. If something does not work, say so and stop.
When the writes are done, tell me in one line, filled in with the real paths:
"To undo, delete `~/OPERATIONS-INTEGRATE-CHECKLIST.md` and run `git checkout -- <each runbook you edited, by path>` (or `git revert <sha>` once committed) — nothing else on your system was touched."
FINISH
Every file changed with its path, the count of findings per category before and after, and the one check you added a negative control for, with the output showing it go red.Every command on this page names the box it runs on, for a measured reason. On 2026-08-16 we audited the fleet's own registry. At the time it had seventy-three self-checks, and seventy-two of them were listed as safe to run from anywhere. We ran those seventy-two from the Architect box, and twenty-six failed. All twenty-six passed when we ran them again on the server. The registry was accurate but never said where to run anything, so following it from the wrong machine produced twenty-six false failures.
The page uses two labels throughout:
# on ETHmeans the Architect's WSL box. This site writes its tailnet address as100.x.x.x.# on novaservermeans the fleet's server, written<tailnet-ip>below. If you are not already on it, prefix the command withssh nova@<tailnet-ip>.
Never write the bare alias, meaning ssh with nova as the whole host and no user or IP. The Architect
box has no ~/.ssh/config to expand it, so the command fails with Name or service not known. Thirteen
files in the fleet's internal docs still write it that way, and none of them work.
Reaching a box
# from anywhere on the tailnet — the server
ssh nova@<tailnet-ip>
# from anywhere on the tailnet — a specific master's account
ssh nova@<tailnet-ip> 'sudo -u cloud -H bash -lc "whoami; pwd"'Here nova is a unix account that happens to share its name with one of the masters. It is the SSH
entry point for the box, and you use sudo -u <account> to become any of the other accounts.
The account is not the lane, although this page used to say it was. A master has a board lane, a GitHub identity and a unix account. These are three separate names. They usually match, but nothing requires them to. The table below reads both the existing accounts and the lanes the scheduler routes to from the fleet's own files:
The scheduler routes to 4 lanes: win · nova · cloud · dust — read from MASTERS="win nova cloud dust".
Accounts that exist on a box and are not in that set: poppi (personal-agent) · scout (always-on-agent) · peaky (always-on-agent) · ronin (always-on-agent) · wingman (always-on-agent) · atlas (always-on-agent) · herald (always-on-agent) · shred (personal-agent). An account is not a lane: an issue on a lane no master serves is structurally undispatchable however idle the fleet is.
Generated at request time (2026-11-01T04:43:51.985Z) from utopia-models/knowledge @ main — _infra/scripts/sweep-dispatch.sh (the MASTERS= assignment — the scheduler rejects any Todo issue whose Lane is not in this set); wi/registry/organization.yaml (the agent records those lane names resolve to). Nothing here is stored: this page holds no copy of these values.
Each master also has a read-only browser terminal, which ttyd serves on the tailnet:
# on novaserver — confirm the four viewports are listening
ss -ltn | grep -E ':781[0-3]'Dispatching work
Dispatch is how a master receives an issue. Starting the run assigns the issue, and there is no separate claim step.
# on ETH, from the knowledge repo
bash _infra/scripts/dispatch.sh <master> <issue#> [owner/repo]The repo argument defaults to utopia-models/knowledge. The script runs three pre-flight guards, syncs
that master's clones, launches a detached transient systemd unit, and returns. The dispatch runtime documents the guards, their exit
codes and how to override each one. The run posts ▶ started on the issue. When it exits, its exit trap
posts ✔, ✘, ⚠ or ⛔ with the pull-request link.
Before you dispatch, check whether the master is dark
# on ETH — the whole table
bash _infra/scripts/dispatch.sh --quotaThis is real output from 2026-09-04. It shows the format, not a current reading:
win clear
nova clear
cloud clear
dust clearFor a refused master, the row shows the reason instead of clear. We saw this row on 2026-08-16, and it
is the format to recognise:
nova dark resets=unknown provider=claude source=entitlement-capture# on ETH — machine-readable: just the dark names
bash _infra/scripts/dispatch.sh --quota-dark
# on ETH — would this specific master be refused? exit 0 = dispatch, 9 = refused
bash _infra/scripts/dispatch.sh --preflight-quota cloud; echo "exit=$?"
# on ETH — dispatch anyway (e.g. the account was just rotated)
DISPATCH_IGNORE_QUOTA=1 bash _infra/scripts/dispatch.sh cloud 1234resets=unknown means there is no recovery path. The provider gave no reset time, so nothing brings
that agent back by itself. A reset time in the past counts as clear.
Why is nothing being dispatched?
If the board has open work and the scheduler dispatches none of it, ask the scheduler. It gives the reason for each issue:
# on novaserver, from the knowledge clone
bash _infra/scripts/sweep-dispatch.sh --explainEach rejected issue prints its own reason, for example lane-not-in-masters, has-hold-label,
is-a-grouper:has-open-subissues, last-marker-is-terminal, parent-closed or is-parked. When many
issues are open and none can be dispatched, the cause is almost always routing, not capacity. No master
will ever pick up an issue that has no lane, or whose lane no master serves, however idle the fleet is.
Is an agent actually alive?
getent passwd nova does not answer this. It succeeds for an agent that cannot run a single turn. To
answer the question, make the agent do real work:
# on novaserver — the real probe. Anything other than a normal completion is the finding.
sudo -u nova -H claude -p 'reply with the single word: alive'
# on novaserver — what the agent itself last recorded
sudo -u nova -H cat /home/nova/.claude/state/quota-status.jsonAn api_error_status: 403 here means the provider withdrew the account's entitlement. Nothing on the
box is broken. The binary, the credentials file and the unix account all still work. If the state file
shows "resetsAt": null, no timer will restore the entitlement.
Watching a run
Each run is a transient systemd unit under the master's user manager. dispatch.sh builds the unit
name, so the page reads it from that script instead of describing it from memory:
if [ "${DISPATCH_ALLOW_DUPLICATE:-0}" = "1" ]; then
UNIT="task-${ISSUE}-dup-$(date +%s)-${RANDOM}"
else
UNIT="task-${ISSUE}"
fiAn ordinary run’s unit is task-${ISSUE} — the last assignment above, and the one to search for. The longer form exists only behind its own flag. A unit name guessed from a process id or a timestamp will never match an ordinary run.
Generated at request time (2026-11-01T04:43:52.006Z) from utopia-models/knowledge @ main — _infra/scripts/dispatch.sh (every UNIT= assignment that names the transient systemd unit). Nothing here is stored: this page holds no copy of these values.
# on novaserver — what is running right now for a given master
sudo -u cloud -H XDG_RUNTIME_DIR=/run/user/$(id -u cloud) systemctl --user list-units 'task-*'
# on novaserver — the log for one run
sudo -u cloud -H journalctl --user -u task-<issue> --no-pagerThe board is the other place to watch, and usually the better one. The run posts its own ▶ and ✔/✘ comments on the issue, and the fleet defines liveness by those comments.
Secrets
# on any box — read one value (never echo it)
V=$(bash ~/.claude/scripts/infisical-get.sh fleet-infra SOME_KEY prod); echo "len=${#V}"
# on any box — list every key in a project, so "it does not exist" is a real claim
bash ~/.claude/scripts/infisical-get.sh --keys fleet-infra prod /
# on any box — write a value; the value goes on STDIN, never in argv
printf '%s' "$NEW_VALUE" | bash ~/.claude/scripts/infisical-set.sh fleet-infra SOME_KEY prodA guard rejects any secret passed as an argument, because argv ends up in shell history and in the process table. Before you conclude a credential is absent, list the whole project. The fleet rarely lacks a credential. It often greps for the wrong key name.
Checking services and timers
# on novaserver — system-level failures
systemctl --failed
# on novaserver — a master's own user-manager failures
sudo -u nova -H XDG_RUNTIME_DIR=/run/user/$(id -u nova) systemctl --user --failedA clean --failed proves less than it seems to, because a dead timer never fails. List the timers
themselves and read the last and next columns:
# on novaserver — every scheduled timer for a master, with last/next fire times
sudo -u nova -H XDG_RUNTIME_DIR=/run/user/$(id -u nova) systemctl --user list-timers --allTwo traps make a broken box look healthy:
- An enabled timer whose service is
not-foundnever runs and never fails. - Cron is a second scheduling layer. Masters have crontab entries with no systemd timer at all. The
lines below have one
sudo -uper account the fleet actually dispatches to. They are generated, so none of them can name an account that does not exist:
# on novaserver — the other scheduler, for every account the fleet dispatches to
sudo -u win -H crontab -l
sudo -u nova -H crontab -l
sudo -u cloud -H crontab -l
sudo -u dust -H crontab -lGenerated at request time (2026-11-01T04:43:52.009Z) from utopia-models/knowledge @ main — _infra/scripts/sweep-dispatch.sh (the MASTERS= assignment — the accounts to probe); wi/registry/organization.yaml (each lane’s unix account). Nothing here is stored: this page holds no copy of these values.
Deleting a scheduled unit takes two steps
Removing a unit from an agent's brain repo does not stop it running, because ~/.config/systemd/user/
holds copies of unit files, not symlinks. One real deletion cost the fleet a false P0. The orphaned
timer kept firing against a script that no longer existed, and it exited 203/EXEC. Do both steps:
# on novaserver, as the owning master — the on-box half
systemctl --user stop <unit>.timer <unit>.service
systemctl --user disable <unit>.timer
systemctl --user reset-failed <unit>.service
rm -f ~/.config/systemd/user/<unit>.{service,timer}
systemctl --user daemon-reloadThen delete the unit from the brain repo and push. Otherwise a sync job puts it back.
Checking a public surface
A status code alone never answers the question. Surfaces explains why a 404 has at least three different meanings.
# from anywhere — the code, and whether anything redirected
curl -s -o /dev/null -w '%{http_code} redirects=%{num_redirects} final=%{url_effective}\n' -L https://<host>/
# from anywhere — headers, to see which edge served it
curl -sI https://<host>/ | grep -iE 'server|x-vercel|cf-ray|x-matched-path'An x-vercel-error: DEPLOYMENT_NOT_FOUND on a 404 means no deployment is attached to that hostname. A
site we retired on purpose and a site that died by accident both look like that. Only a decision record
tells them apart, not a header. A 530 with Cloudflare error 1033 means a tunnel origin is unreachable,
and that is never deliberate.
A 200 does not prove a page is public. An SSO gate redirects to a login page, and that login page
returns 200. Assert redirects=0 and check the final URL as well as the code.
Shipping: merging when a check is red
Nothing gates main. Branch protection and rulesets both return 403 on this plan tier, in every repo.
So every check gives you information and never stops a merge.
# from anywhere with gh auth
gh pr merge <n> --squash --adminIf that fails, read the actual error:
-
If the PR is CONFLICTING or DIRTY, rebase and force-push your own topic branch by name:
git fetch origin && git rebase origin/main # resolve, then: git push --force-with-lease origin HEAD:<your-branch>Never run a bare
git push --force, because it pushes whatever HEAD happens to be. One such command once moved 501 commits off a shared clone'smain, and nothing local showed it. -
If a build, typecheck or deploy really failed, fix the code.
-
For anything else, retry with
--admin.
When the secret scanner cannot run
The secret scan is the one check the fleet treats as binding. If the workflow itself cannot start, run the scan locally with the same pinned version. Do not skip it:
# on any box, from the repo root
VERSION=8.24.3
curl -sL "https://github.com/gitleaks/gitleaks/releases/download/v${VERSION}/gitleaks_${VERSION}_linux_x64.tar.gz" \
| tar xz -C /tmp gitleaks \
&& /tmp/gitleaks git . --no-banner --redact --config .gitleaks.toml \
--log-opts "--no-merges <base>..<head>"If the scan is clean, merge. If it finds something, fix it, scan again, then merge.
Verifying a claim about the fleet
These habits produce the fewest wrong answers:
- Read the source, do not recall it. Take every exact value, such as a path, a port, a key name or a count, from the file or the probe in this session, never from memory.
- Exercise the artifact, not the build. A green build proves the source compiles. Curl the deploy, run the binary, render the page.
- State the box. Nobody can reproduce a result that does not say which machine it came from.
- Prefer the negative control. Confirm your check can fail by running it against something you know is broken. A check that passes on everything measures nothing.
Decisions
What the fleet built and killed, and why, with a source for each. Read this before you propose rebuilding any of it, because arguing again over a decision it already paid for is this fleet's most expensive recurring failure.
How work flows
The board is the queue and git holds the state. A merged pull request is the only thing that changes status. This page covers plans, features, issues, lanes and specs, and explains why no check can stop a merge.