A ten-cent sample run proposed 99 claims from my own notes, and I accepted 95 of them

By Nan Liang · 2026-11-13

For $0.10, a large language model read 17 pages of my personal wiki and proposed 99 standalone claims. A human review pass accepted 95 and rejected 4. This post is about the two-queue pipeline behind those numbers, and about the three truncated API calls and one expiring access token that the ten cents also bought.

I ran this pipeline against my own notes in November 2026. Model prices, token limits, and API behavior change fast; treat every number here as a receipt from specific runs, not a benchmark.

My wiki holds 1,567 pages of accumulated working notes — decisions, source assessments, post-mortems. Buried in the prose are judgments: “this metric lies,” “that vendor’s number is marketing, not measurement,” “don’t touch this curated list.” Useful, but they were never first-class data. You could read a page and absorb its opinion. You could not review, deduplicate, or audit the opinions as a set.

Last week I started extracting them with a large language model.

The opinions were already in the wiki; they were just not data

Full-text search can find a note when I remember its wording. Vector search, which I added earlier, does better — it retrieves a page even when I ask with different words. But retrieval was never the real problem. The problem was that a judgment inside a 900-word page is not an object. It has no status, no reviewer, no way to say “we checked this one and it aged badly.”

I wanted every load-bearing opinion lifted out of the prose and parked somewhere reviewable, each one traceable to the page it came from. Bulk extraction, unsupervised, was the obvious approach. It was also the one I refused, for a reason that has burned me before.

Proposals stay untrusted until a human accepts them into the page

Unsupervised extraction has a quiet failure mode: the output looks like knowledge. It arrives formatted, confident, page-numbered. If it lands directly in the wiki, nobody can tell which sentences a human ever endorsed. I have written about this class of failure before — automation that writes conclusions into notes is expensive to audit afterward.

So the pipeline has two queues with different trust levels:

1. Propose. The LLM reads pages and writes candidate claims — the system calls them “takes” — into a pending queue. Proposals are untrusted by definition. They carry provenance (source page, section, session) but touch nothing.
2. Accept. A human reviews the pending queue one entry at a time. An accepted take gets appended to its source page inside a marked ## Takes fence, attributed, and only then becomes part of the wiki. Rejected ones are dropped with a reason recorded.

The design bet is simple: the expensive resource is not generation, it is judgment. Generation is $0.10. Judgment does not get cheaper by skipping it.

The sample run cost $0.10 and lost three calls to output truncation

The sample ran on the night of November 11, 2026, against a coding-tier model from Kimi through its OpenAI-compatible endpoint. I am not printing the exact model version or the token meter; the provider’s dashboard gives me the aggregate charge, not a per-call receipt I can quote cleanly. Here is what the records do support:

All three failures had the same shape: the model started emitting the structured JSON, hit its output-token limit, and stopped mid-array. The retry wrapper re-ran each failed batch to completion, which is where most of the 24 minutes went. The table counts initial calls; the retries are not metered separately in any view I have, so I cannot quote a total-attempt figure — three of the ten initial calls truncated, full stop. Whether the retried calls are billed separately I also cannot tell from the dashboard; the $0.10 is the total I was charged. With one ten-call sample, that is a reason to monitor truncation, not a reliable estimate of the rate on future runs.

Scaling linearly — same page density, same retry behavior — the full corpus extrapolates to roughly $9–10. The real full run is not one job anyway: it is split into 30-plus overnight rounds by a scheduled 05:00 job, with caching so a page is not re-read after its first pass.

Review accepted 95 of 99, and the four rejections named two traps

The acceptance rate was 95 of 99 — 96%. The four rejections sorted into two categories, and the recorded rejection reasons are worth more than the acceptances because each names a trap the pipeline cannot see on its own (reasons paraphrased; the queue stores the originals):

• Three rejections, one reason: “don’t touch the Vue learning path.” The extractor had pulled opinions out of a maintained learning-path document — a page whose whole point is a hand-ordered sequence. Each extracted claim read fine in isolation. Detached from the sequence, each one was still wrong to keep: the page is the content. The review verdict for the three was the same — leave the learning path alone — and the enforcement of that boundary lives outside the extractor, in the accept queue.
• One rejection: “don’t treat vendor data as a promise.” A take restated a tool vendor’s figure as if it were a durable fact about our own infrastructure. Extracted opinions inherit the epistemic status of their source; a marketing page does not become measurement because an LLM quoted it neutrally.

The 96% figure is a baseline, not a verdict on the pipeline. What it tells me so far: the proposer reads ordinary notes well enough that review is mostly confirmation, and the rejections cluster exactly where judgment — not language — is the bottleneck. If the rate moves a lot in either direction on the full run, the first thing I will check is which page types entered the sample, before I conclude anything about model quality.

An hourly token expiry killed long batches and masqueraded as an empty queue

The review queue is served over an MCP interface, and the access token for that interface lives for one hour. The overnight jobs are long enough to outlive a token, and they did: they started failing mid-run once we began running real batches through the queue. Before we caught it, that failure was indistinguishable from the legitimate state: the client asks for the queue, gets an auth error it does not distinguish from “nothing pending,” and reports an empty review list. Silent, and plausible.

The affected jobs were the overnight ones. The fix is a cron job that refreshes the credential at 50 minutes past every hour and pushes it out to the thirteen client profiles that talk to this server. It has run clean since: the nightly jobs complete, and the queue reads match what the proposer actually wrote. The narrower lesson, stripped of drama: when a failure signature matches a legitimate empty state, you will discover the failure from its consequences — a queue that should have entries and does not — not from an error message.

Generation is cheap; review is the product

For ten cents, the sample showed me where my note-taking conventions are ambiguous about what is claim, what is curation, and what is borrowed confidence. That map is the product. The 99 takes, the $0.10, even the 96% acceptance rate are byproducts — the number that matters longer-term is whether the review queue stays legible when the full 1,567-page run lands. A queue a human can actually clear is a feature. A queue that grows faster than review is a landfill with timestamps, and no extraction spend fixes that.