Two nights of zero submissions ended in a script that claimed my domain list as work

By Nan Liang · 2026-10-25

On 24 October, two scheduled Search Console channels were supposed to move about twenty URLs between them. Both logged one line and stopped. The next day I fixed what I believed was the cause and then found two more defects underneath it: my queue picker globbed every text file in the queue folder, so it claimed a 19-domain site list and a 14-line submitted log as fresh work, and the submitter reported success for ten pages it had only inspected. The backlog that night was 160 URLs.

None of the three defects threw an error. That is the part worth writing down.

All three defects hid inside one claim-and-submit path

• 160 URLs sat queued across nine per-site files, from 89 on the largest site to 1 on the smallest.
• 19 bare domains and 14 already-submitted URLs lived in the same directory as the work, one glob() call away from being picked up.
• 2 nights produced zero submitted URLs; the second night’s skip was correct behaviour and the log could not tell me that.
• 10 lines claimed, 0 submitted on the first repaired run: every URL came back indexed, meaning the script inspected it, saw it was already in Google, and never clicked anything.

The lock error was real, and it was the least interesting part

The two channels are shell scripts. One runs on a Windows box at 20:50, one on this Mac at 22:00, and each claims URLs from a shared folder by deleting the lines it takes. To keep a manual run from colliding with a scheduled one, both take a lock at the top of the script. The original code was the usual idiom:

exec 9>/tmp/gsc.lock
if ! flock -n 9; then
  log "another instance is running → skip"
  exit 0
fi

The problem is that flock is not present on macOS. which flock returns nothing here — I checked again while writing this. When the command does not exist, the shell returns 127, ! inverts that to true, and the guard fires on every single run. The script exits 0 with a message that reads like a reasonable decision. The scheduler sees a successful run. Nothing in the pipeline is in a position to notice, because “I skipped because someone else is working” is a legitimate state.

A guard that cannot fail is worse than no guard, because it looks like protection and it converts every run into a no-op. I replaced the file-descriptor idiom with an atomic mkdir lock: the lock directory holds the owner’s pid, a live pid makes the second run give way, a dead pid or a directory with no pid file makes the lock reclaimable after five seconds of waiting, and a trap releases it on exit. That reclaim path is the weak part and I know it: mkdir and the pid write are two separate operations, so there is a window in which a second process can find a lock directory with no owner yet, wait five seconds, and take it away. I chose that over the alternative of a lock that never blocks anything, and the window is five seconds wide rather than the entire night.

I tested four cases in isolation before trusting the replacement — first acquisition, concurrent refusal, stale lock with a dead pid, lock directory with no pid file — and only then pointed the scripts at it.

Night two was contention, and the log named the wrong machine

On 24 October that channel’s log holds a single dry-run at 21:19 and then a skip message at 21:40, an hour after its scheduled slot; I have not established whether the 20:50 slot fired and logged nothing or never fired at all. The Mac channel logged the same sentence at 22:00, with a problem: the Mac script’s message said the other machine was holding the lock. The two scripts had been forked from one another and the string had never been updated, so a skip on this Mac pointed at a computer that was idle at the time. That string was corrected later that night.

The reason for the Mac skip was real, and it was me. I had a manual round running in an interactive terminal, holding this machine’s lock, submitting URLs one at a time. A scheduled window deferring to an interactive session is exactly what a lock is for. Deferring was correct.

What was not correct was the reporting. A skip, a failure, and an empty queue all reached my end-of-day check as “zero submissions”. Three states, one number. The check now requires the claim count to appear in the log (claimed=N), and treats a missing claim line as a different condition from a claim of zero.

The picker globbed every text file in the folder

This is the defect I did not go looking for. While verifying that the lock fix had landed, I read the claim step properly for the first time. Here is what it did:

for f in sorted(pathlib.Path(qdir).glob("*.txt")):
    if "claimed" in f.name:
        continue
    ...

The directory holds one file per site: gradeplot.com.txt with 89 URLs, expectcalc.com.txt, payofftable.com.txt and six more. It also holds two files that are not work at all. sites.txt is the list of my 19 domains, one bare hostname per line, no scheme. submitted.txt is the append-only audit trail of everything already submitted — 14 lines at that moment, 24 by the end of the next day.

The filter skipped any file with claimed in its name. It did not skip those two, because they do not contain that substring. So on every claim round, the picker was entitled to take bare hostnames as URLs and to re-take URLs that had already been submitted and logged. The separator between work and record was a filename convention that the code half-implemented.

I reproduced it in a sandbox copy rather than reasoning about it, with the queue directory pointed at a scratch folder holding both kinds of file. Before the fix, the claimed list contained bracketree.com, expectcalc.com and https://x.com/already-submitted/. After the fix, it contained only real queue URLs.

The fix is two rules, and only one of them is a whitelist:

1. Skip any file named sites.txt, submitted.txt or excluded.txt — a name blacklist, the weaker half.
2. Keep only lines that begin with http, so a bare hostname can never enter the loop regardless of which file it came from — a scheme whitelist.

I also renamed one of the two site lists in the same pass, and immediately discovered why name-based exclusion is fragile: a third file, excluded.txt, had to be added to the skip list the same afternoon. The list of names I remember is not the list of names that will exist next month. The structural fix is one directory for work and another for records, or a suffix convention like *.urls.txt that the picker accepts; that is on my list instead of another name.

Exit code zero covered two different outcomes

The submit step runs a Node script against a DevTools-attached browser. Zero means the run finished, not that a submission happened. The script first runs Search Console’s URL Inspection and reads the verdict; when the verdict is “URL is on Google”, it stops there and never clicks Request Indexing, because resubmitting an already-indexed URL spends indexing quota for nothing. The inspection itself draws on a separate daily inspection allowance, which is why the wrapper still runs it only once per URL. It reports that short circuit as status indexed. That logic is correct. The shell wrapper around it logged OK, counted it, and moved on.

The first repaired round made the distinction visible. Ten URLs were claimed from the expectcalc file, the round finished in three minutes nine seconds, and every one of the ten came back indexed, with the tool’s detail line saying the URL was already in Google. Ten OK lines, zero quota spent, zero clicks on the button. Those ten pages had been queued by an automatic lastmod-based top-up and were already inside the index, so the queue had been carrying ten tasks that could not produce anything.

Left alone, that failure mode repeats indefinitely: the URLs are appended to submitted.txt when they exit successfully, the submitted list was not being read by anything except me, and the queue file gets refilled by the same automation the next day. Every line that is both in a queue file and in submitted.txt is a line that will be retried tomorrow with no possible effect.

The wrapper now reads the status field out of the submitter’s output and counts indexed, dry-ready and submitted separately, then reports all three. The line “10 claimed, 0 submitted, 10 already indexed” describes that night; “ok=10” did not.

The same guard now exists in two copies

One more thing I found while verifying. The window script for the Mac channel exists twice on this machine — once under ~/.hermes/scripts/ and once under the scheduler profile’s scripts/ directory. At 19:48 today, after a separate incident where a browser port turned out to be logged into the wrong Google account, I added a ten-line login check to that script.

The two files no longer have the same md5. diff shows the login check in one and not in the other. Whichever copy the scheduler resolves, the other one is now a file that looks like the production script and behaves like last week’s. Duplicated guards drift, and the drift is invisible because both copies still run.

I have not consolidated them yet.

Five changes replaced name filters with structural separation

• Split records from work in the filesystem, not in a filter. The picker reads every .txt in that directory, so any new file placed there becomes work.
• Make the claim step announce its own numbers. claimed=10 at the top of a run is a cheap assertion; the absence of that line is now itself a failure signal.
• Treat skip, failure and empty as three states. My daily check reads the claim count before it reads anything else.
• Test a guard by trying to make it fail. The mkdir lock got four negative tests before it got a single real run. The flock version got none, and it never worked once.
• Do not trust exit 0 from a wrapper around a UI. Read the status the interface actually reported, and count outcomes, not attempts.

The queue holds 160 URLs tonight, and the two channels can move about 22 a day when nothing is queued ahead of them, which puts the backlog at roughly a week of windows. The open question is whether the remaining lines can produce anything at all: ten of last night’s were already in the index before they were queued, and the only way I have to find the rest is to diff the queue files against submitted.txt.

A note on scope: this describes two shell scripts and one Node script driving the Search Console web interface in October 2026. Interface labels and quota behaviour change without notice; treat every number here as a snapshot of my own account.