Decisions

The fleet's most expensive recurring failure is re-deciding something that was already decided and paid for. Every entry below says what the thing was, why it died, and what replaced it. If you are about to propose one of these, the burden is new information — not a fresh opinion.

A decision is not reopened by disagreement. It is reopened by a superseding record that names what changed, written beside the original, with the original's Superseded-by header updated in the same commit. A silently invalidated decision record is worse than none.

Killed: ralph — the 24×7 autonomous loop

Deleted 2026-07-24.

ralph.service, with Restart=always, drove a loop: pick the next open board issue for this master's lane, run headless claude -p, repeat. Around it accreted watchdogs, heartbeats, priority-sort tests, self-reload fixes and a pick-order bug — an entire body of supervision machinery whose only job was keeping a resident process honest.

Replaced by on-demand dispatch: dispatch.sh plus task-runner.sh, no resident agent process at all. The close-out measured the replacement rather than asserting it — 7 dispatched runs, 7 verified outcomes, 0 runtime failures, against a roughly 20% baseline.

The residue was instructive, and it took a second deletion to clear. For weeks after the loop was deleted, the registry still carried a per-agent field named after it, still described as the live backend selector — and the field really was read, so nobody could call it dead. The framing outlived the system, then the field outlived the framing. Both are gone now: nothing in the dispatch scripts selects a backend, and the launcher passes one literal provider name to every run.

A name outliving its system is how a deleted thing keeps costing you, and the cost is specifically that the surviving name makes the deletion look incomplete to everyone who reads it afterwards.

Killed: the generated component catalog and its fail-closed gates

Deleted 2026-07-23/24.

An 804-line machine-generated registry where every component carried a shell predicate, run fail-closed on every pull request by three separate workflows plus a post-commit regeneration hook. Its own header claimed "a fix isn't done until catalog = reality."

It was killed because a gate that blocks on a generated map false-blocked documentation changes and auto-reverted its own cleanup. Eighteen dead gate workflows went in one commit.

The replacement inverts the design deliberately. There is one hand-maintained component registry, and every entry carries a verify: command you run, with the instruction "run it — never trust this file's prose." Advisory by construction.

Then the honest footnote, because this wiki does not publish only the flattering half: an audit found that verify: is read by humans and executed by nothing, and that it records no box, so a quarter of its commands fail from the wrong machine and pass from the right one. The mechanism is better than what it replaced and it is not yet what its own header claims.

Killed: the NATS message bus

nats.service and its reload units are on the never resurrect list. Cross-machine agent comms were replaced by issue comments and @mentions — delivery that is durable, at-least-once, and visible on the surface a human already reads.

Three zombie units survive on the box, and one of them is active. Their residue was documented in a file that was itself deleted, which is why they get named here. This page said two, and said they were inactive and disabled, until a reader running the pattern walk ran the command and reported otherwise (knowledge#6665). Re-measured on the box, 2026-08-23:

$ systemctl list-unit-files | grep -i nats
nats-reload.path       enabled   enabled
nats-reload.service    static    -
nats.service           disabled  enabled

$ systemctl list-units --all | grep -i nats
nats-reload.path      loaded active   waiting  Watch NATS config; reload server on change
nats-reload.service   loaded inactive dead     Reload NATS on config change
nats.service          loaded inactive dead     NATS JetStream fleet hub

nats-reload.path is enabled and active (waiting) — a live path unit watching for changes to the config of a service that will never start again. A deletion in a repository is half a deletion, and the surviving half is the half that keeps running.

Killed: the command center's generated infrastructure surface

Built on the deleted catalog, so it returned a permanent empty state once the catalog died. Removed in a single pull request that deleted "11 callerless routes, 7 orphaned components, 5 dead libs."

This one is listed because open plans still name deleting it as a to-do. Anyone dispatched on that line finds nothing to delete, which reads like a lost change rather than an already-shipped one. A stale local clone will happily disagree; git ls-tree origin/main is the truth and a local ls is not.

Killed: a scheduled local re-run of another repository's CI

Deleted 2026-08-15.

A timer fired every three hours on one master: check out the product repo, typecheck, build, test, file a P0 on any real failure. It was built to close a genuine blind spot and was repeatedly repaired over months.

It died because the product repo's own CI runs those same three phases on every push to main — it just told nobody. Once it announced for itself, a second box re-running identical work was duplicated effort.

The half-deletion, which is the real lesson

Deleting the unit from the agent's brain repository did not stop it running. ~/.config/systemd/user/ holds copies, not symlinks into the repo. The timer stayed active waiting with its ExecStart pointing at a script that no longer existed, and the next firing — nearly three hours after the "delete" — exited 203/EXEC and filed a P0 about itself.

A brain-repo deletion is half of a deletion. The other half is on the box, and it is five commands. They are written out in Operations.

Standing decisions

These are not history. They are current, and they constrain new work.

Publish everything, unsanitized

This wiki carries real hostnames, real tailnet IPs, real usernames and real credential mechanics. That was decided after measuring exactly what it exposes, not by default and not by oversight. Of the fleet's internal documents, most name hosts or users, many carry tailnet addresses, and a substantial share describe how the vault works.

Do not redact, do not generalize, and do not add a scrub pass. A contributor "helpfully" sanitizing this site is a regression, not a fix.

The one hard stop is unchanged and absolute: never a secret in a tracked file. Publishing the vault's mechanics is in scope; publishing a credential never is.

One secret store

Infisical, self-hosted, and nothing else. No local vault, no second store, no .env in a repository. A separate registry carries each credential's meaning — account, scopes, what it can and cannot do — so that "I don't have access" is a claim that must be checked against a list before it can be made.

Subtract by default

Before adding a script, a rule, a service, a gate or a file, the question is what can be deleted or merged instead. Legacy that is replaced is deleted at the moment of replacement. Git history is the archive; archive/ folders are banned. A convention gets an advisory linter; only a stable invariant gets a blocking gate.

Options before decisions

A non-trivial architecture or tooling choice gets researched alternatives — at least two real options, plus do-nothing, plus integrate-something-existing — before the decision, and the decision is written down. Proven native and open-source tools beat custom machinery; custom is the exception that must justify itself.

This site is its own example. The wiki lives in a standalone repository rather than inside the product monorepo, and the record naming the four options considered, the reason, and the costs the choice imposes was committed before the first line of the app.

A limit is a measurement, never a precaution

Before setting any ceiling — concurrency, retries, batch size, timeout — measure the real headroom and write the measurement next to the number. A cap chosen out of caution silently becomes the system's capacity, and unlike a bug it never announces itself: it reports success while doing a fraction of the work.

The fleet learned this in the most annoying possible way. Its dispatch scheduler ran for days at roughly 8% of proven capacity — three of four masters idle on a 12-core box under a load average below 1 — because caps of 1, 4 and 2 had been set against a failure mode that two other mechanisms already covered. The defence was real; it was simply aimed at the wrong axis.

Fix forward, never pre-gate

Build, merge, deploy. If it breaks, revert or fix and redeploy. The alternative — a human approving each step — was measured and is slower at every scale this fleet operates at. What this explicitly does not license: skipping verification, force-pushing shared branches, or claiming completion that was not observed.

Never resurrect

• ralph and every watchdog, heartbeat and supervisor built to babysit it.

• The NATS bus and its three remaining zombie units.

• The generated component catalog and any fail-closed gate built on a generated map.

• Content-keyword gates of any kind.

• Resident processes, heartbeat polls and scheduled health checks as a way of knowing whether the fleet is alive. The board is the queue; a run's own close-out comment is the liveness signal; and the one page that reads current state does so at request time and stores nothing.

  One sanctioned exception, named here so a reader does not have to discover it with ps. A */5 * * * * cron entry on each master's own crontab POSTs that master's record to the command centre, which stores it under a five-minute TTL; the TTL expiring is what turns a box grey on a dashboard a human is looking at. That is a self-reported heartbeat and it was adopted deliberately (2026-07-22) because it answers a different question: which box is online right now, not is this work alive. Nothing in the work loop reads it — no dispatch, no gate and no close-out depends on it, which is what keeps the ban meaningful. Separately, a master's own standing interactive session is resident for as long as nobody restarts it (measured 13.3 days on one master, 2026-08-23); no dispatched run leaves a process behind. The ban is on resident machinery as the ANSWER to "is the fleet alive". It has never been a ban on every scheduled job, and a reader who runs the walk from inside this fleet will find both of these — so they are written down rather than left to look like drift (knowledge#6665).

If you believe one of these should come back, the path is a superseding decision record naming what changed. Not a rebuild.