Two nights of silent failure in my unattended automation, and the five-minute check that would have caught it

2026-10-17

This post describes the behavior of specific tool versions on my machine — macOS with bash 3.2 and the Claude Code CLI as of October 2026. Argument-parsing details change; verify against your own versions before copying any of this.

For two nights in a row, my nightly automation fired exactly on schedule and exited one second later, having done nothing. The scheduler was never the problem. The prompt I fed to the agent CLI happened to start with a line of three dashes — the YAML frontmatter of my runbook file — and the CLI parsed the entire 12-kilobyte prompt as one long command-line option, rejected it, and quit with exit code 1. Here is the failure, the one-line fix, and the check I now run on every unattended job.

The job fired on time and failed in one second

I run a small unattended pipeline on my Mac. Every night at 21:00, a launchd agent executes a shell script. The script checks that a logged-in browser is reachable, then hands a runbook file to an AI coding agent in print mode, which executes the night’s tasks: a few community replies, a read-only check of one account, and a blog post for this site.

On the morning of the third day I opened the logs. Both nights looked like this (the two Chinese lines translated — the script logs in Chinese, everything else verbatim):

===== nightly-yanghao start 2026-10-16 21:00:05 =====
[Fri Oct 16 21:00:05 CST 2026] CDP preflight OK (localhost:9223)
error: unknown option '---
created: 2026-10-15
type: runbook
...
[Fri Oct 16 21:00:05 CST 2026] claude exit rc=1
===== nightly-yanghao end 2026-10-16 21:00:06 rc=1 =====

The timeline table, both nights:

The scheduler did its job perfectly. The script did its preflight correctly. The agent CLI then rejected its instructions as a malformed flag, and the whole run was over before a human heartbeat’s worth of time had passed. Two nights, six tasks per night, twelve missed executions, and the system was green on every dashboard I had — because I had no dashboard, only a log file nobody read.

One thing worth crediting: macOS itself kept receipts I never looked at. launchd records the last exit status of every job it manages — launchctl list shows it in the Status column, where a 1 had been sitting since the first night. And the plist format has native StandardOutPath and StandardErrorPath keys that persist a job’s output to files without any wrapper script. My wrapper did capture stderr into its own log, so nothing was lost — but between launchd’s exit-status table and my log, the failure was recorded twice and read zero times. The observability was there. The observing was not.

The prompt was valid Markdown and an invalid command line

My runbook is a Markdown file, and like most of my Markdown files it starts with YAML frontmatter:

---
created: 2026-10-15
type: runbook
---

My script passed the file’s contents to the CLI as a positional argument, roughly like this:

claude -p "$(cat runbook.md)"

The CLI’s own help text describes print mode plainly: use -p/--print for non-interactive output. What the help text does not need to say — because it is a convention every CLI follows — is that argument parsing happens before your positional argument is interpreted. A positional argument that begins with a dash is not a positional argument at all. It is an option, by definition, at the layer where the shell’s words meet the program’s parser.

This convention has a name and a number. POSIX.1’s Utility Syntax Guidelines define -- as the delimiter that ends option parsing (Guideline 10): everything after it is an operand, even if it begins with a dash. Every mainstream argument parser — glibc’s getopt, Python’s argparse, the Node and Go flag libraries — implements some flavor of this rule. That means two things. First, my failure was not a quirk of one particular CLI; any standards-following parser would have rejected the input the same way. Second, there was always an escape hatch I did not know I needed:

claude -p -- "$(cat runbook.md)"   # parses fine — everything after -- is an operand

So the parser saw ---\ncreated: 2026-10-15\ntype: runbook\n..., tried to match it against its known flags, failed, and printed error: unknown option '---...' with the entire runbook echoed back as the offending “option”. Every layer of my pipeline was working as designed. The design was wrong.

My preflight checked everything except the thing that runs

This is the part that stung, because I had been careful. The script has a --check mode. Before I loaded the launchd job, I ran the preflight and watched it print PREFLIGHT-OK. I verified the schedule, the working directory, the browser probe, the log directory.

What the preflight validated: the environment around the task. What it never validated: the task itself. The one line that actually invoked the agent was exercised for the first time at 21:00 on the first night, in production, unattended. Loading a schedule is not running a job. A green preflight means the runway is clear; it says nothing about whether the plane can fly.

There is a subtler version of this trap worth naming. Even if my preflight had invoked the CLI, it would have invoked it with a fixed, known-good test string — while the real run reads its input from an external file. A preflight with hardcoded inputs proves the binary starts; it says nothing about the contract between the binary and the data you feed it at runtime. The honest fix is a smoke test with the real inputs, end to end: after enabling any unattended job, run it once by hand, exactly as the scheduler will. If that is too expensive, read the first real run’s log the next morning before you assume the system exists.

A notification is not monitoring

One detail deserves honesty, and a harsher framing than I first gave it. My script did not stay silent: on a non-zero exit it fired a macOS desktop notification — something went wrong, see the log. It fired on both nights. I never saw either one, because notifications on this machine are where focus-mode banners go to die.

My first draft of this section asked “why did the notification fail?” The better question is why a desktop banner was the observability layer of an unattended job in the first place. A banner is volatile, silenced by Do Not Disturb, and requires a human to be looking at the screen within seconds. Rob Ewaschuk’s classic essay My Philosophy on Alerting (reprinted in Google’s Site Reliability Engineering) argues that passive channels like email are of very limited value precisely because they get buried; an alert that does not demand immediate human action is noise. A transient GUI banner on an unattended machine is worse than email — it pages no one, ever. That is not a notification problem. It is an architectural anti-pattern: a desktop UI component carrying a production observability responsibility.

The deeper issue is the direction of the signal. Mine was push-on-failure: the system tries to announce its own death. But anything announced by the dying component shares a failure domain with that component — if the Mac loses power, hangs, or the process is killed before the trap runs, the announcement dies with it. Absolute silence, indistinguishable from success. The robust pattern inverts the direction: the job sends a success heartbeat to an external service when it completes, and the service — not the job — raises the alarm when the heartbeat stops arriving. This is the dead man’s switch, or inverted heartbeat, and it is the entire business model of services like Healthchecks.io, Cronitor, and Dead Man’s Snitch. Google’s SRE book devotes a chapter of its cron discussion to exactly this: for periodic batch jobs, “the expected output did not appear” is the symptom worth alerting on, not any signal the job itself emits.

Every mature scheduler has some version of this lesson built in. Classic cron’s MAILTO captures whatever the job printed and mails it — which is why > /dev/null 2>&1 in a crontab silently disarms it. systemd has OnFailure=, which activates a separate diagnostic unit when a service fails, so the reporting path is a different process from the failing one. macOS gave me the exit-status table I never read. My cheap version, until I build a real external heartbeat, is a rule instead of a tool: the first morning after any new unattended job goes live, reading its log is a scheduled task, not a vibe.

The fix was one pipe, with three caveats

The corrected invocation strips the frontmatter and feeds the prompt through standard input, where no option parsing applies:

awk '/^---$/{c++; next} c>=2' runbook.md | claude -p

stdin beats the positional argument here for two reasons, and only one of them is option parsing. The other is size: positional arguments live in the fixed-size buffer execve copies into the new process, capped by ARG_MAX (1 MB on macOS). A prompt that grows past that ceiling fails with E2BIG before the program starts at all. A pipe is a stream with no such ceiling, so the fix removes the parsing ambiguity and the size limit in one move.

Three caveats make this production-grade rather than a snippet:

1. Pipeline exit codes lie by default. A pipeline’s exit status is its last command’s. If awk crashed mid-stream and the CLI treated empty input as success, the failure would vanish. set -o pipefail makes the pipeline report any stage’s failure.
2. launchd does not load your shell’s locale. A job started by launchd runs without the locale environment your terminal has; byte-oriented tools like awk can misparse multi-byte UTF-8 content unless you set LC_CTYPE explicitly in the script. My runbooks contain Chinese — this one is not theoretical.
3. --- is also Markdown syntax. In CommonMark, three dashes are a thematic break — a horizontal rule. My one-liner counts every line of three dashes, so a horizontal rule in the runbook body would be silently deleted from the prompt: a parsing crash traded for silent data corruption. The hardened filter only strips a frontmatter block at line 1 and stops at its first closing delimiter:

awk 'NR==1 && /^---$/ {f=1; next} f && /^---$/ {f=0; next} !f' runbook.md

I tested the pipe with a two-word prompt before trusting it, which is what I should have done on day one. The next scheduled run executed all six tasks end to end.

The total cost of the bug was two quiet nights. The total cost of not noticing could have been weeks — nothing in the system would ever have gotten louder on its own. That asymmetry is the real lesson, and it is why the incident earned a permanent entry in the family’s incident log rather than a quick fix and a shrug: unattended automation fails silently by default, and silence only breaks where you have built a place for it to break.

What I changed after the incident

1. Strip metadata before handing files to a CLI. Any file that might carry frontmatter gets filtered before its content becomes an argument; stdin beats positional arguments for anything that can start with a dash; and where an argument is unavoidable, -- ends option parsing before it begins.
2. Smoke-test the invocation with real inputs, not just the environment. A preflight proves the surroundings are ready; only a real run with real data proves the pipeline runs.
3. Check the first-run log the next morning. New unattended jobs get a scheduled human review of their first execution, before trust is granted.
4. Treat silence as the alarm. Until a real external heartbeat exists, the interim rule is explicit: no news is bad news, and something — eventually an external service, for now a calendar entry — is responsible for noticing.
5. Write the incident down where future runs will read it. My nightly agent now greps the incident log before starting and appends new entries before finishing, so a mistake is expensive exactly once.

None of this is clever. That is the point. Unattended systems do not need cleverness; they need the dull discipline of checking that the thing you built is the thing that ran.

This article was created with the help of AI. AI was not used to write the content; it assisted only with translation and grammar checks.