# Architecture (/docs/structure/architecture) Five layers, in the order they depend on each other: the **hosts** that exist, the **agents** that run on them, the **runtime** that starts an agent, the **services** that stay up between runs, and the **secret store** everything authenticates against. ## The vocabulary [#the-vocabulary] The fleet's structural choices are **architectural patterns** — there are 14 of them, and the term is the fleet's own, decided rather than drifted into. Everything below is one of two things: * a **component** — something that *runs* or *holds state*. The box, the sweeper, the secret store. The fleet keeps a registry of them, one document each, and that registry is what this page describes. * an **architectural pattern** — one structural choice, stated so a reader on different substrate can **adopt it or reject it**. Several of them do not run at all: decision discipline and liveness-vs-existence are policy, not processes. That second half is why this site does not call them *mechanisms*, *components*, *Fleet Architecture* or *Fleet Infrastructure* when naming them as a set — each of those fits the runtime half and is wrong for the policy half. The adopt-or-reject test is the one a name here has to pass, and only *pattern* passes it. Elsewhere on these pages "mechanism" keeps its ordinary English meaning; the rule binds the set. The fleet's canonical inventory of the 14 lives in its knowledge base, at `world-infrastructure/reference/architectural-patterns.md`. The pages here are the exposition of them, and [/feedback](/feedback) walks that same list — one pattern at a time, each explained inside the question before it asks you whether you would adopt it. Nothing on this page is a health reading. Whether a host is online, whether an agent can run *today*, and how full the disk is are all volatile and belong on `/live`. Read that page for them — and note what it does with the ones a public page cannot reach: it prints **the exact command and the box that answers it**, rather than a number it did not measure. ## Hosts [#hosts] ### novaserver — the fleet's box [#novaserver--the-fleets-box] One machine does essentially everything. It is an HP EliteDesk 805 G6 running Ubuntu 24.04, reachable on the tailnet at `100.71.25.43` and on the LAN at `10.0.0.231`. | | | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CPU | AMD Ryzen 5 PRO 4650GE — 6 cores / 12 threads | | RAM | 30 GiB, plus 8 GiB of swap | | GPU | Integrated Radeon RX Vega 6. **No NVIDIA** — `nvidia-smi` is not installed, and nothing on this box does GPU inference beyond what Ollama does on CPU | | Root filesystem | 295 GB, and it is the fleet's most persistent capacity worry. `disk-guard.timer` fires hourly and reclaims a few megabytes per pass — it is running, and it is not winning. The figure is not readable from a public page: `/live` names the command (`ssh nova@100.71.25.43 'df -h /'`) rather than showing a number it cannot measure | | Reached by | `ssh nova@100.71.25.43` | **The bare alias does not resolve.** Writing `ssh` with just `nova` as the whole host — no user, no IP — exits with `Name or service not known`, because there is no `~/.ssh/config` on the Architect box to expand it. The working address is always the user at the tailnet IP, which is exactly what the dispatch script itself uses (`NOVA="${DISPATCH_NOVA_IP:-100.71.25.43}"`). This is the single most common error in the fleet's internal docs — thirteen files still write the bare alias, one of them inside a copy-pasteable runbook block, so a reader following them fails on their first command. ### ETH — the Architect's box [#eth--the-architects-box] A WSL2 Ubuntu 24.04 guest on the founder's Lenovo, tailnet `100.103.62.104`, unix user `kxdev`. WSL is capped by `.wslconfig` at 12 GB of memory, 4 GB of swap and 6 processors; inside the guest that reads as roughly 11 GiB of RAM. This box plans, files issues, and dispatches. It does not host any service the fleet depends on. ### utopiamodels — a host that is not a machine [#utopiamodels--a-host-that-is-not-a-machine] `100.92.117.107` is an application that joined the tailnet using `tsnet` — an LLM gateway, not a computer. It appears in the node list beside real hardware, serves on port 80, and has no shell. Worth knowing before you try to SSH to it. ## Agents [#agents] **Who the agents are is generated, not written.** The table below is read out of the fleet registry on every request. It used to be typed here, and it was wrong in the way that matters most: it called the DMV-portfolio master `dmv`, a retired alias for the agent the registry records as **`dust`**. That name has never been a unix account on any box — `id -u dust` returns `1001`, and `id -u dmv` still returns *no such user* — so every `sudo -u dmv` command this site published failed for every reader who copied one, and no page can catch that about itself. > GENERATED — The unix accounts that run agents > > This block could not be read and therefore renders nothing: GitHub 404 Not Found on /contents/world-infrastructure/fleet/registry/components.yaml?ref=main *Generated at request time (2026-09-14T16:06:26.794Z) from `utopia-models/knowledge` @ `main` — `world-infrastructure/fleet/registry/organization.yaml` (every `agents:` record — its class, its box, its unix account, its GitHub identity, and whether it carries a subscription block at all); `world-infrastructure/fleet/registry/components.yaml` (the `uid` inside each runtime component’s `where:` line). Nothing here is stored: this page holds no copy of these values.* Each agent has its own credentials, its own `~/.claude` brain directory (each one a version-controlled repo of its own), and its own GitHub identity. They are peers. There is no supervisor process. The **binary** is deliberately not in that table. `/usr/bin/claude` is where it resolves on novaserver and nowhere else — on the Architect's WSL box it resolves elsewhere — so "every agent runs the same binary at the same path" is a claim about one box wearing the clothes of a claim about the fleet. The command that settles it for a given account is `sudo -u -H command -v claude`, run on the box that account lives on. ### Existence is not liveness [#existence-is-not-liveness] **The registry's own check for an agent is `getent passwd `, and that check passes on an agent that cannot execute a single turn.** This is the most important distinction on this page. A master can hold a valid unix account, a present credential file, a clean systemd state, working `gh` auth, and still be unable to run — because entitlement lives with the account at the provider, not on the box. When that happens the agent looks *busier* than a healthy one: transient units keep starting, keep dying in seconds, and keep announcing themselves. Measured 2026-08-16: **three of the four masters could actually run**, and the fourth reported `{"exhausted": true, "resetsAt": null}` in its own state file. A `resetsAt` of `null` is the important half — it means the provider named no recovery time, so no timer brings that agent back. The dispatch scheduler correctly excluded it and said so in its own log line every five minutes. The current per-agent answer is on `/live`. It is derived from each agent's own measured entitlement — a recorded refusal, never `getent` — and that page names the `claude -p` command, and the box, that settles it definitively. ### There is one backend, and choosing between backends was deleted [#there-is-one-backend-and-choosing-between-backends-was-deleted] `dispatch.sh` passes a **literal** backend name into every run — `--setenv=TASK_BACKEND=claude`. There is no lookup, no registry read and no per-master resolution: the function that once chose between providers was deleted along with the alternate provider it chose between. This page said the opposite for a while, and the wrong version was the more flattering one — a fleet that resolves each agent's provider at runtime sounds more capable than one that hardcodes a single name. It is worth stating the plain version instead, because the plain version is what a reader would find in the script. The variable itself survives for a different job. It is the fleet's *"am I inside a dispatched run"* signal, which is what every board-keyed gate exempts on. A flag whose original purpose was deleted can be worth keeping for the second job it accidentally does — provided somebody writes down which job it is now doing. ## The dispatch runtime [#the-dispatch-runtime] Two scripts and systemd. That is the entire agent runtime — no scheduler daemon, no queue, no watchdog. The overview is below; the guards, their exit codes and the race that only a file lock can close are on [The dispatch runtime](/docs/operating/dispatch-runtime). | Script | What it does | | ----------------------------------------------------- | -------------------------------------------------------------- | | `_infra/scripts/dispatch.sh [repo]` | The Architect hands a master its issue **by starting the run** | | `_infra/scripts/task-runner.sh [repo]` | The run itself, on the master's own box | What one dispatch actually does, in order: 1. Launch under the one backend the fleet has — a literal, not a lookup (see above). 2. `agent` runs locally; the novaserver masters are reached with one `ssh nova@100.71.25.43` plus `sudo -u `. 3. **Pre-flight refuses a quota-dark master.** A dispatch into an agent that is already dark burns the run and charges the issue an attempt for it. The check fails *open* — an unreadable state file, a missing file, or a reset time that has already passed all mean "not dark", because a false block stops the fleet and that is worse than a wasted run. 4. **Pre-flight refuses a second run for an issue that is already running.** One issue, one run. The guard lives in the entry point both the scheduler and a hand-typed dispatch share, because a guard in only one consumer cannot see the other path. 5. **Pre-flight syncs the master's clones**: fetch, stash anything dirty, checkout main, hard-reset to `origin/main` for every repo in the run's repo list. 6. Launch a **detached transient unit** — `systemd-run --user --unit=task-- --collect` — so the run survives the SSH connection dropping and the laptop going to sleep. 7. `task-runner.sh` self-updates once, re-checks that the issue is *still work*, posts **▶ started** before anything that can fail, runs `claude -p` under a turn cap, and on exit its trap posts **✔ / ✘ / ⚠ / ⛔** plus the pull-request link. The cap in that last step is the kind of number that goes stale the moment someone tunes it, so it is read rather than written: ```bash MAXT="${TASK_MAX_TURNS:-200}" ``` The cap is a **default of 200, not a fixed number** — any caller may raise or lower it by exporting `TASK_MAX_TURNS`, and `_infra/scripts/dispatch.sh` does exactly that when it launches the unit. A run that consumed all its turns reports the cap it actually ran under, which is why the number in a failure message is evidence and the number on this page is not. *Generated at request time (2026-09-14T16:06:26.798Z) from `utopia-models/knowledge` @ `main` — `_infra/scripts/task-runner.sh` (the `MAXT=` assignment passed to `claude -p --max-turns`). Nothing here is stored: this page holds no copy of these values.* ### The last-moment disposal re-check [#the-last-moment-disposal-re-check] An issue can stop being work while its run is starting, and every gate that asks about it runs too early. Measured on one real case: the pre-flight read the issue at 14:36:01 and it was open and correct; the issue was labelled a throwaway at 14:36:03 and closed at 14:36:05; systemd started the unit at 14:36:12; ▶ was posted at 14:36:15. A master then spent a full session on board residue. The fix asks the same two questions from **inside** the run, immediately before ▶ — the genuine point of no return. Only two signals count: the issue is CLOSED, or it carries the throwaway-fixture label. Anything else is a judgement a human may legitimately be overriding at the command line. ### The sweeper [#the-sweeper] The thing that carries a chain of work forward is **not a service and not a cron job**, which means every probe that goes looking for a `sweeper.service` falsely reports it missing. It rides as the second `ExecStart` line of a systemd drop-in on one master's `reconcile-board.service`, fired by `reconcile-board.timer` every five minutes: ```text /home/nova/.config/systemd/user/reconcile-board.service.d/10-sweep-dispatch.conf ExecStart= (clear) -/usr/bin/bash %h/.claude/ops/reconcile-board-status.sh --apply -/usr/bin/bash %h/dev/utopia-models/knowledge/_infra/scripts/sweep-dispatch.sh --apply -/usr/bin/bash %h/.claude/ops/mute-agent-notifications.sh --apply ``` Its own log line is the best one-line health summary the fleet produces. One real line, read out of `~nova/.claude/logs/sweep-dispatch.log` and emitted 2026-09-04T08:48:34Z — quoted as a sample of the shape, not as a current reading: ```text sweep: 39 open · 0 dispatchable · in-flight 3 (win=2 nova=0 cloud=1 dust=0 ) ``` A refused master adds a trailing segment naming who and why — `· rate-limited: (exhausted nova)` — which is absent above because on that pass every master was clear. Read that carefully, because it is a lesson in reading your own instrumentation. A large open count against zero dispatchable looks like a capacity failure and is not one — it is a **routing** failure. Issues with no lane, or on a lane no sweeper reaches, are structurally undispatchable no matter how idle the fleet is. The sweeper had been printing the cause every five minutes while investigations looked for a broken mechanism. ## Services on the box [#services-on-the-box] The first two groups below run on novaserver and stay up between agent runs; the third is state the box depends on and does not host. The dispatch runtime is the part with no resident process; the *box* has plenty. **System units:** `tailscaled` · `cloudflared` (the tunnel that publishes selected local services to the public web) · `ttyd` plus one read-only browser viewport per master on ports 7810–7813 · `ollama` on 11434, serving small local models · `occ-redirect` on 3939, a small Python responder that 301s a legacy hostname to the apex · `grafana-server` on 3000 · `prometheus` on 9090 with node-exporter on 9100 · two self-hosted GitHub Actions runners, a full lane and a light lane. **Docker:** **Infisical** — backend, database and redis — which is the fleet's only secret store, and a full **SigNoz** observability stack (ClickHouse, a Postgres metastore, keeper, ingester). **Managed Postgres, off the box:** two hosted databases hold state that nothing here can reconstruct, so they belong in this list even though no process for them runs on novaserver. **Neon** — the `utopia-auth` project, Postgres 17 in `aws-us-east-1`, holding the product's auth and user rows; a nightly `pg_dump -Fc | gzip` through the direct unpooled endpoint copies it to cold object storage at 35-day retention, because Neon's free point-in-time recovery reaches back only about six hours. The same Neon organisation holds three other projects that are not fleet infrastructure, and conflating them with this one is the easy mistake. **Supabase** — the `peptide-shop` project in `us-east-1`, which holds the database, auth, and the order and payment records behind `dmvresearch.shop`; those rows are the irreplaceable asset, and they are dumped nightly to the same bucket by two independent jobs, one a `pg_dump` and one a row-level export through the REST API. **Scheduling is single-homed.** One master's user manager carries essentially every scheduled timer in the fleet (measured 2026-09-04 with `systemctl --user list-timers --all` under each master account: 37 timers on nova against 4 on win and 3 each on cloud and dust). The consequential ones publish the command center's feeds, re-render it and check it, sync creative assets, watch token health, refresh the tailnet snapshot, guard the disk hourly, and self-heal the default route every sixty seconds. **Cron is a second, parallel scheduling layer**, and it is easy to miss. Masters carry crontab entries for monitors, self-reports and usage feeds that have no systemd timer at all. An audit that reads only `systemctl --user list-timers` under-reports every master's real cadence. **A dead timer never fails, so `systemctl --failed` reports clean.** The box has held an *enabled* timer whose service unit was `not-found`, and dormant timers that had not fired in a week, while every failure query returned empty. Absence of failure is not presence of function. ## Secrets [#secrets] **Infisical is the only secret store**, self-hosted in Docker on novaserver. There is no local vault, no `.env` checked in anywhere, and no second store. | Operation | Command | | --------------------------- | ----------------------------------------------------------------------- | | Read | `bash ~/.claude/scripts/infisical-get.sh [env]` | | List every key in a project | `bash ~/.claude/scripts/infisical-get.sh --keys prod /` | | Write | pipe the value on **stdin** to `infisical-set.sh [env]` | Projects are scoped by domain: fleet infrastructure, the product, per-portfolio research, and engineering. Writing a value as a command-line argument is rejected by a guard, because an argv secret lands in shell history and in the process table. A second file — a credential registry — carries each credential's *meaning*: which account it belongs to, what scopes it has, what it can do and what it explicitly cannot. Reading that registry before declaring a credential missing is a standing rule, because the fleet's recurring failure mode is not a missing credential but a grep for the wrong key name. Two absolute rules: * **Never print a secret, or the presence of one, to any display sink.** Capture into a variable and check its length. * **Never write a secret into a tracked file.** This is the fleet's single hard stop — the one thing that is not fix-forward, not advisory, and not overridable. A secret scanner runs on every pull request; it is the only check whose result is treated as binding. Publishing the *mechanics* of all of this, as this page does, is in scope and deliberate. Publishing a value never is.