My shared automation runner failed sixteen times without a trace, and every health check was green

By Nan Liang · 2026-10-29

Everything here was measured on my own machines in October 2026: a Mac mini running a shared Perplexity research runner (browser automation over CDP, port 9226) and a scheduling side on a laptop. The Perplexity page fields I cite will rot when their UI changes. The failure model and the evidence rules don’t.

For four days my health checks said the channel was fine, and they were mostly right. The channel was fine. What had been dying was the work flowing through it. On the afternoon of October 28, my shared research runner logged sixteen uncaught-exception failures in two clusters, each with an empty note field and no error text anywhere on disk. The account had quota. The page loaded. The evidence simply did not exist.

This post is the post-mortem: the three-layer model of what my checks actually covered, the per-day numbers, root causes ranked by how sure I am, and the two code changes that make the next death explainable. It is the sibling of my earlier post about the job that ran on time and did nothing. That one was a single job with no one reading its log. This one is a shared channel where every layer’s health check passed while the pipeline was dead.

The cracks were between the layers, not in them

One shared runner script drives a logged-in browser to run deep research. Three scheduled jobs plus ad-hoc research calls all funnel through it, serialized by a single file lock. Health is policed by a daily verify job that opens the site and checks the page structure. Three layers, each checking itself:

The verify job ran green on October 27 and 28 even though the account’s login state was gone on both days: it printed login:false and still returned OK, because a page that renders unauthenticated is, to that check, a healthy page. All four documented exit codes carry named reasons — CDP unreachable, lock wait or quota, not logged in, output too short. Code 1 carries nothing: the ledger recorded it with an empty note, no screenshot, no stderr capture.

Two properties that read like small log-hygiene details are the whole bug: failures left no body, and no ledger row said whose job it was.

The channel never went down. It stopped leaving bodies.

My ledger, per day:

The last successful deep run on October 28 was 13:38. After that the channel went quiet: one call on the 29th, succeeded at 02:24, then nothing. Two checks that would have caught the rot early cost almost nothing. Tailing the ledger and counting empty-note exit-1 rows per day — five minutes, no new infrastructure. And confirming the subscription was active: the quota ledger showed one deep run used on the 28th, and the account was active, so this was never a quota story.

One honest note on the 18:06 cluster: it sits inside a dead window on my laptop’s proxy, 17:31–19:31. The mini’s network is independent, so I marked that overlap as unproven rather than a cause. It still looked tempting. That is the difference between a cause and a correlation, and I had to write the difference down.

Every root cause I could prove was about missing evidence

Ranked by how sure I actually am:

1. Confirmed — failures left no evidence. The exit-1 rows had empty note fields, no stderr on disk, no screenshot. This is the exact mechanism behind the complaint that started the investigation: “the jobs stopped working, and I can’t say why.”
2. Confirmed — the shared ledger has no owner. 145 calls on the 25th, 26 on the 28th: I cannot say which job issued any of them, so I cannot even say whose backlog died.
3. Confirmed — the verify job’s blind spot is structural. It failed on October 29 only because my laptop’s network dropped for six minutes around 10:02 and its 3600-second probe timed out. It sailed through two days of lost login state. It alarmed when it shouldn’t and stayed silent when it should have. Both directions, same root cause.
4. Inferred, not confirmed — the retirement mechanism. A done marker for one job had existed since the 27th at 11:41, yet its retry job kept spinning on 30-minute cycles. Two jobs were disabled at 00:17 with no reason recorded. I cannot separate “the gate failed” from “a different consumer was issuing those calls” — which is exactly the consequence of root cause 2. So I stopped calling it a root cause.

The ranking is the deliverable. Three confirmed, one explicitly marked inference. A claim I cannot ground in something on disk is a hypothesis, and the incident write-up writes it down as one.

The fix targets the next death, not this one

Three changes, two shipped and one deliberately queued:

1. Catch the uncaught path. The runner now installs process-level handlers for uncaught exceptions and unhandled rejections: on a crash, a one-line summary of the error lands in the ledger’s note field instead of the field staying empty. The next exit-1 row reads uncaught: <message>. Someone will be able to say what died.
2. Put an owner on every row. The ledger record gained a by field — the calling job’s name, passed in by each consumer. A failure row now says who died, not just that something died.
3. Leave the verify split as the next ticket. Splitting the page-structure check from a login assertion — so login loss gets its own alarm — is two lines. It stays in the queue: as of writing, the daily verify still prints login:false → OK. Naming the debt beats pretending the observability is complete.

And one decision that is not code: I am not re-enabling any runner. Nothing is queued. Restarting a consumer to keep a channel warm is how the 145-call lock-storm day happened. The channel earns its keep on demand.

A dashboard that cannot distinguish “no work pending” from “the worker is dead” is a green light on a dead channel. The rule I keep from this incident: when a failure cannot be explained with evidence on disk, the defect is in the evidence path — and fixing the path, not guessing the cause, is what ends the incident.