My review gate caught a number my own tests were hardcoded to defend

By Nan Liang · 2026-11-06

This is a field report from November 2026. The simulation code it describes runs at build time with deterministic seeds, so every number in it is reproducible — but the specific claims belong to one site in my portfolio at one point in time. Verify from the seeds, not from my memory.

Earlier this week, an automated review pass flagged a claim in a published data post on one of my tool sites: a history-aware team redraw that, per the post’s own FAQ, “collapsed” the repeat-pairing rate “from 4.7% to 0.0% across the entire simulation” — zero repeats in 9,999 consecutive draws. The flagged number was not merely wrong. It was impossible: with 24 players split into 4 teams of 6, zero repeated pairs across two consecutive draws cannot happen, and the mathematical floor is 8. The worse finding came after I opened the repo: my regression test had been asserting the impossible value as the expected output. This post is the re-measurement, the fix (commit b004552), and what I changed so the pipeline can’t certify a lie again.

The claim that could not exist

The site is lineupdraw.com, a seeded random team generator. Its blog carries data posts whose numbers are computed at build time by a small simulation module — 10,000 seeded draws of the same 24-name roster, every draw reproducible from its seed. Post 1, “Same-Team Odds in Random Splits, Computed,” answered three questions: how often a fixed pair lands on the same team, how often two independent draws in a row both group them together, and whether a history-aware redraw prevents repeat pairings.

The third answer was the problem. The published FAQ stated that with the previous draw passed in as history, “the engine re-rolls internally (up to 20 tries) until it finds a split with zero repeated pairings,” and that “measured over 9,999 consecutive draws, the repeat rate collapsed from 4.7% to 0.0%.” The comparison table rendered the same claim as data: 0 repeats in 9,999 draws = 0.00%, next to a column that read engine target: 0.

A design target and a measurement, printed side by side, reading as one fact.

Eight pairs is the floor, and the arithmetic fits in one line

Here is why zero was never on the table. Yesterday’s draw produced 4 teams of 6, which is 60 pairs in total (each 6-person team contributes 15). Today’s redraw scatters each of those 6-person groups across only 4 teams. Six people into four teams means at least one team receives at least two of them — pigeonhole — so each of yesterday’s four teams forces at least one repeated pair. Split 6 as evenly as possible (2, 2, 1, 1) and each yesterday-team contributes 2 repeated pairs, one per pair of members that landed together. Multiply by four teams: at least 8 of yesterday’s 60 pairs must recur in today’s draw, every single time.

No redraw policy, however clever, can go below 8. A claim of 0 repeats in 9,999 draws is not an optimistic measurement — it refutes the arithmetic it sits on. One of my favorite failure modes in all of engineering is the impossibility argument, because it costs one sheet of paper and ends every debate. I had skipped the sheet of paper.

My test asserted the lie, so CI stayed green

The post survived its own test suite, and that turned out to be part of the disease. The old regression test did not check whether the history-aware claim was plausible. It checked whether the page contained the expected string:

• assert.ok(html.includes('0.00%')) — with a Chinese comment declaring that the history-aware row must contain 0.00%.

Green CI proved the lie was present, faithfully rendered, byte for byte. It said nothing about truth. I have written before about checks that lie in the other direction — alarms that scream while the underlying state is fine. This is the mirror image and it is scarier: a check that blesses the exact thing it should forbid, because someone (me) typed the wrong expected value once and the test enshrined it.

A test that matches output against a hardcoded literal is a photograph of a claim, not a validation of it.

Three questions had been melted into one counter

The old simulation function computed everything in a single loop: the single-draw rate, the independent back-to-back rate, and the history-aware rate all came out of one counter structure with shared state carried between iterations. Three genuinely different questions — “are they together today,” “are they together twice in a row by chance,” and “does history-help beat chance” — were melted into one measurement stream, and the FAQ narrated the engine’s design target (“re-roll until zero”) as if it were the measured output.

The decision, written in the three-part form I owe every technical choice: I wanted one fairness number for the history feature. The engine actually has three distinct behaviors with three distinct denominators. So I split one simulation into three — independent single draws, independent consecutive draws, and history-aware consecutive draws — each with its own counter, its own seed namespace, and its own row in the table. No shared state, no ambiguity about what a row measures.

The re-measurement, with calibers separated

All numbers below come from the rewritten build-time simulation: 10,000 deterministic seeded draws of the same 24-player roster, 9,999 consecutive-draw transitions, same group-assigning engine the site serves in production. Absolute counts and percentages are both shown; the simulation recomputes identically on every build.

The honest story is better than the fake one. The history-aware engine cannot reach zero — nothing can — but it grinds the average down from 13.1 repeated pairs per draw to 8.6, within touching distance of the 8-pair floor, at a cost of about 8.6 internal re-rolls per draw (it tries up to 20 and keeps the split with the fewest repeats). For the tracked pair specifically, back-to-back collisions fall from 5.08% to 3.15%. Team balance never wavered: across all 10,000 draws the largest gap between any two teams was 0.

“3.15% instead of 5.08%, near the provable floor” is a real achievement. “0.0%” was a magic trick.

The review gate did what CI structurally cannot

The number was caught by an automated review pass — a large-language-model reviewer re-reading both data posts during a site handover check, flagging Post 1’s zero claim as mathematically suspect and confirming it with the pigeonhole argument. Post 2, reviewed in the same pass, needed only wording-level fixes and its numbers stood.

Notice what happened. The test compared the page against my expectation. The review gate compared the claim against reality — it ran the impossibility check I had skipped, arithmetic that costs nothing and tolerates no negotiation. A second agent with no stake in the original number was willing to say “this cannot be true” where my own tooling was designed to say “this matches what you typed last month.”

This is the strongest argument I have yet collected for keeping an LLM review gate in front of automated publishing, and I say that as someone whose gate had just indicted him. I have measured plenty of quota and indexing walls on this blog (one alarm at a time); this one is about judgment, not quotas.

What changed so it cannot recur

1. Calibers are separated in code, not only in prose. Three simulation functions, three seed namespaces, and a table row per measured question. If a future number looks off, its denominator is one glance away.
2. The test now forbids the failure instead of photographing it. The rewritten regression test asserts histRate > 0 with a comment stating the pigeonhole floor; asserts the history-aware rate is below the independent baseline (the real claim); asserts the page contains the phrase “eight-pair floor”; and asserts the literal string 0.00% appears nowhere in the post.
3. The copy states the floor before the feature. The FAQ now leads with the impossibility — “It prevents as many as mathematics allows — and the ceiling is lower than intuition suggests” — and reports 8.6 against the floor of 8, not against zero.
4. Design targets stay out of the measurement column. The table’s analytic column carries exact values or “no exact value — a constraint problem,” never the engine’s aspiration.

After the rewrite: 65/65 tests green, drift check 40/40 clean, deployed the same morning. The post now says something true and, I would argue, more useful — a hard floor you can quote beats a soft zero you cannot.

Next in this series: how the same review pass grades two posts in one batch, and why “only wording fixes” is a verdict worth logging on its own.