Structure

Decisions

What the fleet built and killed, and why, with a source for each. Read this before you propose rebuilding any of it, because arguing again over a decision it already paid for is this fleet's most expensive recurring failure.

Updated

Prompt 1

Learn why re-deciding is the expensive failure

Reads this page, explains the reopening rule, the four kills and the standing decisions back to you, then quizzes you. Writes only its checklist.

Prompt 2

Find my unrecorded decisions and their residue, then adopt a record format

Finds the decisions your repository made and never recorded, plus the residue of what you deleted, then proposes a record format sized to your setup. Writes only after you confirm.

The fleet's most expensive recurring failure is deciding again something it already decided and paid for. Each entry below says what the thing was, why it died, and what replaced it. If you want to propose one of these again, bring new information. A fresh opinion is not enough.

Disagreement does not reopen a decision. A superseding record that names what changed does. You write it beside the original and update the original's Superseded-by header in the same commit. A decision record that someone silently invalidated is worse than no record.

Killed: ralph, the 24×7 autonomous loop

Deleted 2026-07-24.

ralph.service ran with Restart=always and drove a loop. It picked the next open board issue for this master's lane, ran headless claude -p, and repeated. Over time it collected watchdogs, heartbeats, priority-sort tests, self-reload fixes and a pick-order bug. All of that supervision code existed only to keep one resident process honest.

On-demand dispatch replaced it. dispatch.sh plus task-runner.sh do the work, and no agent process stays resident. The close-out measured the replacement instead of asserting it. It recorded 7 dispatched runs, 7 verified outcomes, 0 runtime failures, against a baseline of roughly 20%.

Clearing what it left behind took a second deletion. For weeks after we deleted the loop, the registry still had a per-agent field named after it, still described as the live backend selector. Code really did read the field, so nobody could call it dead. The description outlived the system, and then the field outlived the description. Both are gone now. Nothing in the dispatch scripts selects a backend, and the launcher passes one literal provider name to every run.

A name that outlives its system keeps costing you. Everyone who reads the surviving name afterwards concludes the deletion is incomplete.

Killed: the generated component catalog and its fail-closed gates

Deleted 2026-07-23/24.

This was an 804-line machine-generated registry. Every component in it carried a shell predicate, and three separate workflows plus a post-commit regeneration hook ran those predicates fail-closed on every pull request. Its own header claimed "a fix isn't done until catalog = reality."

We killed it because a gate that blocks on a generated map blocked documentation changes that were fine, and reverted its own cleanup automatically. One commit removed eighteen dead gate workflows.

The replacement reverses the design on purpose. One component registry exists, maintained by hand. Every entry has a verify: command that you run, and the file tells you to run it and never trust its prose. Nothing enforces it, by design.

This wiki publishes the unflattering half too. An audit found that humans read verify: and nothing executes it. It also records no box, so a quarter of its commands fail from the wrong machine and pass from the right one. The new registry is better than the catalog it replaced, but it does not yet do what its own header claims.

Killed: the NATS message bus

nats.service and its reload units are on the never resurrect list. Issue comments and @mentions replaced it for cross-machine agent messages. That delivery is durable and at-least-once, and it shows up where a human already reads.

Three leftover units remain on the box, and one of them is active. The file that documented them was itself deleted, so we name them here. This page used to say there were two, and that they were inactive and disabled. Then a reader doing the pattern walk ran the command and reported otherwise (knowledge#6665). We measured again on the box on 2026-08-23:

$ systemctl list-unit-files | grep -i nats
nats-reload.path       enabled   enabled
nats-reload.service    static    -
nats.service           disabled  enabled

$ systemctl list-units --all | grep -i nats
nats-reload.path      loaded active   waiting  Watch NATS config; reload server on change
nats-reload.service   loaded inactive dead     Reload NATS on config change
nats.service          loaded inactive dead     NATS JetStream fleet hub

nats-reload.path is enabled and active (waiting). It is a live path unit watching the config of a service that will never start again. Deleting something in a repository is half of the deletion, and the other half is the part that keeps running.

Killed: the command center's generated infrastructure surface

This view was built on the deleted catalog, so once the catalog died it showed a permanent empty state. A single pull request removed it, deleting "11 callerless routes, 7 orphaned components, 5 dead libs."

It is listed here because open plans still name its deletion as a to-do. Anyone dispatched on that line finds nothing to delete, and that looks like a lost change rather than one that already shipped. A stale local clone may disagree. Trust git ls-tree origin/main, not a local ls.

Killed: a scheduled local re-run of another repository's CI

Deleted 2026-08-15.

A timer on one master fired every three hours. It checked out the product repo, ran typecheck, build and test, and filed a P0 on any real failure. Someone built it to cover a real blind spot, and people repaired it many times over months.

It died because the product repo's own CI runs those same three phases on every push to main. That CI simply told nobody. Once the CI announced its own results, a second box re-running the same work was wasted effort.

The half-deletion, which is the real lesson

Deleting the unit from the agent's brain repository did not stop it running. ~/.config/systemd/user/ holds copies of unit files, not symlinks into the repo. The timer stayed active waiting, and its ExecStart pointed at a script that no longer existed. The next firing came nearly three hours after the "delete". It exited 203/EXEC and filed a P0 about itself.

Deleting from the brain repo is half of a deletion. The other half happens on the box and takes five commands. Operations lists them.

Standing decisions

These decisions are current, not history, and they constrain new work.

Publish the mechanics, not the identifying values

Superseded 2026-09-27. This entry used to say "publish everything, unsanitized". That meant real hostnames, real tailnet IPs and real usernames. We decided it after measuring what that exposed, and the entry said outright that a contributor who sanitized the site was causing a regression. The founder narrowed the rule because the audience changed. People now copy these pages into their own agents as prompts. For that reader, a literal tailnet address or an admin UI's hostname teaches nothing that a placeholder does not.

The current rule is to publish how everything works and withhold the literal values that locate private infrastructure. Tailnet addresses become 100.x.x.x or <tailnet-ip>. Internal hostnames and bucket URLs become named placeholders. Every row, class and lesson around them stays the same. We apply the redaction one page at a time. Surfaces and Architecture have it, and so does the tailnet table on the landing page.

This change also follows the reopening rule at the top of this page. We did not silently delete the old entry. We marked it superseded here and said what changed.

The one hard stop has not changed and has no exceptions. Never put a secret in a tracked file. Publishing how the vault works is in scope. Publishing a credential never is.

One secret store

The fleet uses self-hosted Infisical and nothing else. There is no local vault, no second store and no .env in a repository. A separate registry records what each credential means, meaning its account, its scopes, and what it can and cannot do. That way, anyone who says "I don't have access" has to check the claim against a list first.

Subtract by default

Before you add a script, a rule, a service, a gate or a file, ask what you can delete or merge instead. When something replaces legacy code, delete the legacy code at that moment. Git history is the archive, and archive/ folders are banned. A convention gets a linter that only warns. Only an invariant that has stopped changing gets a gate that blocks.

Options before decisions

Before a non-trivial architecture or tooling choice, research the alternatives. That means at least two real options, plus doing nothing, plus using something that already exists. Then write the decision down. Proven native and open-source tools beat custom code. Custom code is the exception, and it has to justify itself.

This site is an example. The wiki lives in its own repository rather than inside the product monorepo. Before anyone wrote the first line of the app, we committed the record that names the four options considered, the reason for the choice, and the costs the choice imposes.

A limit is a measurement, never a precaution

Before you set any ceiling, such as concurrency, retries, batch size or timeout, measure the real headroom and write the measurement next to the number. A cap chosen out of caution quietly becomes the system's capacity. Unlike a bug, it never shows itself. It reports success while doing a fraction of the work.

The fleet learned this the annoying way. Its dispatch scheduler ran for days at roughly 8% of proven capacity. Three of four masters sat idle on a 12-core box with a load average below 1. The cause was caps of 1, 4 and 2, set against a failure mode that two other mechanisms already covered. The defence was real, but the caps were the wrong place to put it.

Fix forward, never pre-gate

Build, merge and deploy. If something breaks, revert it or fix it and redeploy. We measured the alternative, a human approving each step, and it is slower at every scale this fleet runs at. This rule does not allow skipping verification, force-pushing shared branches, or claiming a result nobody observed.

Never resurrect

  • ralph and every watchdog, heartbeat and supervisor built to watch over it.

  • The NATS bus and its three remaining leftover units.

  • The generated component catalog and any fail-closed gate built on a generated map.

  • Content-keyword gates of any kind.

  • Resident processes, heartbeat polls and scheduled health checks as a way to know whether the fleet is alive. The board is the queue. A run's own close-out comment is the liveness signal. The one page that reads current state reads it at request time and stores nothing.

    There is one sanctioned exception, and we name it here so you do not have to find it with ps. Each master's own crontab has a */5 * * * * entry that POSTs that master's record to the command centre. The command centre stores it with a five-minute TTL. When the TTL expires, the box turns grey on a dashboard a human is watching. That is a self-reported heartbeat, and we adopted it on purpose (2026-07-22) because it answers a different question. It tells you which box is online right now, not whether this work is alive. Nothing in the work loop reads it. No dispatch, gate or close-out depends on it, and that keeps the ban meaningful. Separately, a master's own standing interactive session stays resident until someone restarts it (measured at 13.3 days on one master, 2026-08-23). No dispatched run leaves a process behind. The ban is on resident machinery as the way to answer "is the fleet alive". It has never banned every scheduled job. A reader who runs the walk from inside this fleet will find both of these, so we wrote them down rather than let them look like drift (knowledge#6665).

If you think one of these should come back, write a superseding decision record that names what changed. Do not rebuild it.

On this page