JavaEye

A guard script's one-hour recheck sleep exactly matched the cron timeout, and three daily windows produced nothing but a single log line

This post describes my own scripts and scheduler as of November 2026 — bash 3.2 on macOS, a wrapper-script cron setup. The specific numbers are my real configuration; the design rules generalize, but verify against your own versions before copying anything.

For three days in a row, one of my daily automation windows produced a log file exactly one line long, submitted nothing, and failed so quietly that nothing — no alert, no error, no summary line — marked the dead run. The cause was a design equation I never wrote down: my guard script’s recheck wait was 3,600 seconds, and my scheduler’s kill timeout was also 3,600 seconds. The recheck I thought I had built could never run, and the run died inside its own waiting room. Here is the one-line log, the arithmetic that killed it, and the budget rule I now put in every long-running script.

The entire log for that window was one line

I run a small family of unattended jobs that submit URLs to Google Search Console through several channels, one per account, each in its own daily time window. Before a window submits anything, a guard script checks that the machine’s exit IP matches the value pinned for that channel — submitting from the wrong exit IP can collide with another channel’s session, so on mismatch the guard’s contract was: wait, recheck once, and if the condition still holds, refuse to submit this window (exit 21). The guard was built on 2026-11-12 and its header stated the rule plainly: wait one hour, then recheck.

The daily log for the local channel on 2026-11-16 is this, in full (one Chinese line, translated):

[gsc-node-guard local] GUARD-WAIT(1/2) exit=unreachable — this channel's exit IP could not be fetched (proxy down?)

That is the whole file. No recheck line. No RESULT summary. No submissions. The window’s wrapper script logged nothing after handing control to the guard — because the process that would have written the summary was killed before it got the chance.

Three daily windows in a row died this way. The family ledger recorded it as “the 001 window: three consecutive silent timeouts.” The 2026-11-16 log file — the artifact I can show — is one line long. Everything else about those runs exists only as an absence.

The wait budget equaled the kill budget

Do the arithmetic I skipped when I wrote the guard. The wrapper’s cron timeout was 3,600 seconds. The guard, on a blocked condition, slept 3,600 seconds before its one recheck. The recheck itself takes seconds; writing the summary takes milliseconds. The timeline inside a blocked window:

SecondWhat happens
0Wrapper starts, hands off to guard
0–5Guard probes exit IP, finds it unreachable
5Guard logs GUARD-WAIT(1/2) — the first of two probe passes — and sleeps 3,600 s
3,600Cron kills the whole process group. The recheck, scheduled for second ~3,605, never begins

Second 3,605 never arrives. My “wait an hour, then recheck” design had a hidden requirement — that the containing run live longer than an hour — and I had configured the container to die at exactly the same moment. The guard’s iron rule said “wait, then verify, then refuse if still blocked.” In practice it was “wait, then get killed, then leave a one-line log that looks like a normal wait in progress.”

I had tested the guard, of course. But I tested its decision logic with GUARD_NOWAIT=1 — the drill mode that skips the sleep. I never once tested the sleeping path end to end, because testing a one-hour sleep is expensive. So the path that only fires in production — a real blocked condition — was the path that had never run. This is the same failure shape I wrote about in my earlier post on a two-night silent failure: every layer green, the design wrong, the evidence recorded but unread.

A timeout kill is the quietest failure mode there is

What made this worse than a normal crash is what a scheduler timeout does to evidence. When a script errors, you get an error — a message, a non-zero exit, something to grep for. When a scheduler kills a process at its timeout, the story just stops mid-sentence. Our wrapper only wrote its GSC-WINDOW RESULT: summary line after submissions completed, and the guard-blocked path exited before that line existed. So the log for a killed window is indistinguishable from the log for a window that is merely still waiting — unless you notice that the file has been one line long for a whole day.

There was a second, quieter bug in the same neighborhood. The wrapper’s last line was exit 0 no matter how many submissions had failed, so the 006 channel’s consecutive-failure counter under-counted its streaks — a window could end broken and still report success. Two ways to be green while broken: get killed before the summary line, or summarize without the failures.

The fix was a budget, not a bigger timeout

The root fix commit landed on 2026-11-17. The load-bearing change is one line and one rule: the recheck wait became a variable, GUARD_WAIT_SEC, defaulting to 900 seconds — and the rule, now written in the script’s header, is that a recheck budget must be strictly smaller than the cron timeout minus the window’s actual work. The cron timeout was also raised from 3,600 to 4,200 seconds as a second layer, but the script comment is explicit that the script-side 900 is the real fix; raising the timeout alone would just have moved the wall. The full bundle:

  1. Wait 900 s, not 3,600 s (GUARD_WAIT_SEC adjustable) — the recheck now finishes around second 905, leaving about 55 minutes of headroom against the raised 4,200-second timeout.
  2. Every guard-blocked run writes a verdict line. The wrapper now emits GSC-WINDOW RESULT: dry-run 0/0 | submit 0 | quota-hit no | remaining <N> | mode guard-blocked even when the guard refuses to submit, so a blocked window is a one-line summary, not a one-line mystery.
  3. Failures exit non-zero. The exit 0 tail was fixed so fail > 0 means exit 1, and the 006 channel’s streak counter now sees its real input.
  4. Cleanup traps moved before the guard call. Previously the EXIT trap was installed after the guard, so a guard-blocked exit skipped lock release and leaked the atomic lock until its TTL expired.
  5. The tunnel no longer holds the cron pipe open. ssh -f backgrounds the session after authentication but keeps the inherited stdout/stderr descriptors open, so a leftover tunnel process made the cron runner wait on a pipe that never reached EOF — the runner itself would hang until its own timeout. The spawn now redirects the tunnel’s output to /dev/null, and a final pkill fallback sweeps any survivor — a cousin of the same “killed mid-wait” family.

Item 2 is the one I would not skip even under time pressure. The wait-length fix stops the bleeding; the verdict line is what makes the next failure legible. A guard that can stop a run must always say it stopped the run — loudly enough that “one line in the log” can only ever mean one thing.

The audit that found it started with a different crash

The sleep bug was not what sent me reading guard scripts. On the night of 2026-11-15, a different sentry — a watchdog that probes a remote machine’s exit IP every 30 minutes and pings me on drift — crashed three times in a row (21:53, 22:53, 23:23). Our night audit attributed the crash to a full-width character on line 47: a script written mostly in English, whose log and notification strings contain Chinese punctuation, had variable expansions sitting directly against full-width characters like ( and ,. The sweep that followed hardened roughly 40 expansions from $VAR to ${VAR} across 14 scripts — cheap insurance, done with a byte-pattern pass so it could be re-run and verified.

The exact mechanics of that crash matter less than the habit it started: when one guard crashes, audit all guards. The watchdog’s full-width bug, the sentry’s invisible waits, the wrapper’s exit 0 tail, and the tunnel’s held-open pipe all lived in the same few hundred lines of shell that I had written in the same week, under the same “it works in drill mode” testing blind spot. I went looking for one crash and found four ordinary bugs that had been invisible in drill mode.

I now write five rules at the top of every script

RuleWhy
Any wait must be < its container’s kill budget, minus the surrounding workEqual budgets mean the last step of the wait never runs
Every early exit writes a verdict line with countsA timeout kill is silent; a verdict line cannot be killed silently
Test the sleeping path once, for real, at a shortened durationDrill mode tests decisions, not lifecycles
A crash in one guard triggers an audit of all guardsSame-week scripts share the author’s blind spots
Fix the budget on the script side, not only the scheduler sideA bigger timeout just moves the wall; the wait must fit by construction

The one-line log file is still on disk. I kept it — gsc-local-2026-11-16.log, 113 bytes, the original line written in Chinese with the code block above as my translation — as the cheapest monitoring artifact I own. It measures one thing perfectly: whether the window that day had anything to say for itself.