The always-on agent

experimental

Sourced from the fleet's own component document for agent-always-on. Called experimental because there is exactly one instance of it; because it exists under an amendment to a rule that forbids its entire class; and because its service units live in no repository, so nothing compares what is running against a committed artifact.

Everything else in this fleet is event-driven. Work arrives on a board, a run announces itself on an issue, a merge moves a status. A resident process is forbidden for anything the board, a dispatched run or a git event already announces — heartbeat polls and scheduled health checks are exactly that, and their failure mode is silence, which is indistinguishable from health.

Then there is one exception, and this page is about what earned it.

What a permanent loop has to prove

The amendment permits a resident process only to observe an external system that emits no signal anyone can subscribe to, and only where latency below the polling interval is the actual point. It adds one requirement that does most of the work:

🔴 The process must name the external event it exists to catch, in its own documentation.

That single requirement is the transferable part. A daemon that cannot name the unsubscribable event it watches is a daemon that is polling something it could have been told about, and writing the name down is where that becomes obvious — to the author, before anyone else has to review it.

The one process here names two:

• A role appearing on an applicant-tracking board. No webhook, no feed and no subscription exists for "a role opened at any company in this region". Being early is the entire edge, and two minutes and twelve hours are different applications.
• A chat-membership update at bootstrap. The moment a human adds the bot to a group. This one is finished: the poll stopped permanently once the identifier was captured, and the published feed reports it as captured rather than as running. A watch that has caught its event and stopped is the right shape; a watch that keeps polling for something it already has is the wrong one.

The loop

service, Restart=always            one cycle
  ├─ detect     watch the external boards
  ├─ enrich     fill in what was seen
  ├─ mailbox    inbound replies
  ├─ notify     outbound, successes only
  └─ apply      act on what detect wrote   ← LAST, and deliberately

each duty runs every executable in its directory, sequentially,
logging the name and the exit code of each

handlers WRAP the canonical tools in the main repository — never a fork

every cycle merges a liveness block into a state file
  → collected into a committed feed
    → rendered on the command center, where a human already looks

Three properties that are load-bearing and easy to undo

A duty is registered in the loop, not discovered from its directory

Which directories run is a list of literal calls in the loop body. A handler dropped into a directory that is not on that list is dead code that looks installed — present, executable, and never invoked.

Discovery-by-directory is the tempting alternative and it trades a small convenience for a silent failure mode. An explicit list means adding a handler requires one edit that a reviewer can see.

The last duty runs last for a measured reason. It is the only handler that can spend minutes rather than seconds — median 19 seconds, 90th percentile 98, maximum 246, across 45 records — and the duties run sequentially. Placed ahead of the watch, it would spend the very latency budget the process exists to protect.

Order matters in a sequential loop, and the ordering argument belongs next to the order.

The restart limit is what makes death visible

Restart=always on its own restarts a crash loop forever. The unit never enters a failed state, the failure handler never fires, and an always-on agent dies silently while its service manager reports it as active.

Adding a start-limit interval and burst is what converts an endless crash loop into a unit that eventually gives up, enters failed, and triggers the notification path. Without it, the most important property of the whole design — that failure announces itself — is quietly absent for the one component that has no other liveness signal.

A schedule is a timer. Only the watch is resident

The daily rollup that summarizes what the loop did is a timer, not a sixth duty. Three reasons, and each one is a test a future addition should be run against:

1. A rollup of rows already written has no latency edge, so it fails the criterion that permitted the resident process in the first place.
2. The duties are sequential, so a rollup inside the loop spends the watch's budget.
3. A surface reporting what the system did today must not go dark when the thing it reports on does.

Anything that is a schedule is a timer here; only the watch is resident. That is the line a proposed sixth thing gets tested against, and it is a much easier test than "is this important enough."

Its isolation is enforced by code that never reads the registry

The scheduler's list of dispatchable agents is a hardcoded string in two scripts. This agent is in neither, so no board issue can route to it — not by misconfiguration, not by a lane typo, not by someone assigning it in good faith.

Adding it to either list would make an always-on agent dispatchable, which is the one property the class denies. The isolation is therefore not a policy anyone has to remember; it is the absence of a name in two string literals, which is the kind of guarantee that survives everyone forgetting about it.

There is a second half worth stating plainly: this agent is never foreground. All human interaction with it proxies through the planning agent. A resident process with its own conversational surface would be a second place to give it instructions, and a second place to give a system instructions is a second tracker wearing different clothes.

Where this one is currently weakest

• The unit files are not in any repository. They are installed from a different repository by a script, and the main repository holds neither — so nothing compares the running schedule against a committed artifact. Read the live one on the box; do not trust a written copy, including this page.
• The interpreter is a virtual environment on the box that nothing in git describes. One duty needs a real browser under a virtual display, and the system interpreter has no package directory of its own on this distribution. The environment is created by three commands documented in the agent's own brain repository, and the handler's pre-flight names it concretely when it is missing rather than failing as a bare non-zero exit. Probe it; do not assume it.
• A probe run as the wrong user reports absence and non-existence identically. These are per-user service units, invisible to a probe from another account, and the failure text does not distinguish "not there" from "not yours to see." Run the probe as that account with the runtime directory set, or fall back to the committed feed — which carries its own timestamp, and is the only reading available from an account that cannot reach the box at all.

That last point generalizes past this fleet: a health check that cannot tell "absent" from "not permitted" is a health check that will eventually report a working system as dead.

Take it further

You are going to inventory every process that runs continuously or on a schedule on this machine, and find the ones that could die without anyone noticing.

Read the reference first:

    curl -s https://docs.utopiamodels.ai/docs/structure/agent-always-on.md

Do not start, stop, enable, disable or edit anything. Every command you run must be a read. If a check would require changing state, describe what you would run and why, and do not run it.

## Step 1 — enumerate, from every layer

There is never only one scheduler, and an audit that reads one layer under-reports every other. Cover all of these, and say explicitly which ones do not apply to this machine:

    systemctl list-units --type=service --state=running --no-pager
    systemctl list-timers --all --no-pager
    systemctl --user list-units --type=service --state=running --no-pager
    systemctl --user list-timers --all --no-pager
    crontab -l
    ls -la /etc/cron.d/ /etc/cron.daily/ 2>/dev/null
    launchctl list 2>/dev/null
    docker ps --format '{{.Names}}\t{{.Status}}\t{{.Command}}' 2>/dev/null

Two notes that will change your results. A per-user service manager is invisible from another account, so a probe run as the wrong user reports "not there" and "not yours to see" with the same text — say which account you ran as. And cron is a genuinely separate layer from the service manager: a machine can carry scheduled work in both, and reading only one of them is the most common way this audit comes out wrong.

## Step 2 — separate the resident from the scheduled

For everything you found, sort it into two piles and state the criterion you used:

- **Resident** — a process that stays up between pieces of work.
- **Scheduled** — a process that starts, does a thing, and exits.

Anything in the resident pile that does its work on an interval is in the wrong pile. That is the single most common finding: a timer wearing a daemon's clothes, paying the cost of always being up for none of the benefit.

## Step 3 — find the ones that can die silently

For each resident process, answer three things from its configuration, not from assumption:

1. **What is its restart policy?** If it restarts unconditionally with no start-limit interval and burst, it can crash-loop forever without ever entering a failed state — so nothing that watches for failure will ever fire. Quote the actual directives.

2. **Is there a failure path?** An `OnFailure` unit, an alert, anything at all that runs when it gives up. If there is none, say so plainly: this process's death is silent.

3. **Where does it announce that it is alive?** Not "the logs" — somewhere a person actually looks. If the only evidence of life is a log file nobody opens, its liveness is theoretical.

## Step 4 — the trap, checked explicitly

Run this and read it carefully:

    systemctl list-units --type=timer --all --no-pager
    systemctl --user list-units --type=timer --all --no-pager

For every timer, check that the service unit it activates actually EXISTS. A timer whose service is not-found never fails — it simply never runs, and `systemctl --failed` reports clean the whole time. Report any you find, with the timer name and the missing unit.

Absence of failure is not presence of function, and this is the cheapest place to prove that on any machine.

## Step 5 — report

One table: name, layer it lives in, resident or scheduled, restart policy, failure path, where it announces liveness.

Then three findings, ranked: the process most likely to be dead right now without anyone knowing, the one that should be demoted from resident to scheduled, and any timer pointing at a service that does not exist.

If you cannot determine something because a probe needs a privilege you do not have, say exactly which command would answer it and which account has to run it. Do not report an unreadable thing as absent — those are different results and conflating them is the specific failure this reference page warns about.

Then ask me which of the three findings to fix first with the `AskUserQuestion` tool, and recommend one with your reason — the silent-death finding is usually the right default, because it is the one where I currently have no way of learning I was wrong.