Three test rounds in one day with an open-source memory layer for my site-running agents

By Nan Liang · 2026-11-10

This field moves fast. gbrain was at v0.60.634.0 on the day I ran these tests — 10 November 2026 — and by the time you read this some of what broke may already be fixed. Treat every number here as a measurement from that one day, not a permanent property of the tool.

On 10 November 2026 I ran three rounds of tests against gbrain, an open-source memory layer (MIT license, garrytan/gbrain) that stores pages and facts for AI agents. The machine is an Apple-silicon Mac with 16 GB of RAM that runs a small fleet of scheduled agents operating my portfolio of static sites. By the end of the day I had a Chinese-capable semantic memory with zero API spend, five findings on record, and one usage rule my other agents now have to follow.

• 3 test rounds in a single day, each written up before the next started
• 65 MB single-file database (embedded PGLite), no API key, no daemon of its own
• 0.887 similarity score for a Chinese query that returned nothing before the fix
• 689 MB hot memory for the local embedding model — 4.2% of 16 GB, released after 5 idle minutes
• 5 findings: two fixed by local embeddings, one that turned out to be by-design behavior with a house rule, two still open

The first round exposed an English-only keyword index, not a broken memory

I stored a test page containing a two-character Chinese word meaning “increase revenue” (pinyin: zēngshōu), then searched for exactly that word: zero results. An English word from the same page (trekpelican) hit instantly. A mixed Chinese-English query hit too — but only because the English half matched and rank fusion carried it.

I have a gift for finding the one code path that doesn’t work in Chinese.

gbrain’s keyword index runs on Postgres tsvector with an English configuration — that is the project’s own diagnosis of the behavior I observed, and it fits: the index simply never sees the Chinese tokens. What I can swear to from my own runs is narrower: Chinese keyword page search returned nothing for words sitting right there in stored pages, while Chinese fact recall worked the whole time because facts and pages go through different matching paths. I did not test Japanese or Korean, so test search in your language on day one, before you store anything you care about.

Local embeddings fixed Chinese search for zero API cost

The fix had three decision beats. I wanted semantic search so reworded queries would hit. Hosted embedding APIs cost money and send my notes to someone else’s server. So I ran the embedding model locally: bge-m3 from BAAI (568M parameters, 1.2 GB download) on Ollama 0.40.15, served by brew services on the same Mac. The embedding requests go to localhost and nowhere else — that is the entire “data never leaves the machine” boundary, and it costs nothing per call.

The measured before/after, on the same stored pages:

Two setup details cost me time. brew install ollama timed out on a direct connection and only worked through my local proxy. And enabling embeddings needs the database unlocked — I had to stop the MCP server child process first, run gbrain embeddings enable --embedding-model ollama:bge-m3 (1024-dim vectors), backfill with gbrain embed --stale, then restart the server.

On memory, my main worry going in: 16 GB is not a lot. The hot load is 689 MB of unified memory (the GPU path), 4.2% of total. Ollama’s default keep-alive is 5 minutes — after the last call it unloads by itself, and I watched ollama ps count down to prove it. Cold start on a new call runs 1–3 seconds; inside the warm window the embedding call itself is milliseconds, with the database work on top. Note that Ollama is a resident service — the thing with no daemon is gbrain itself, whose MCP server runs as a child process of the agent session and dies with it.

The single-writer database forced a one-client house rule

PGLite is a single-file Postgres with one writer. The MCP server holds that writer for as long as the agent session lives. Every CLI command that touches the database during a session — read or write — comes back with pglite_busy. That is a clean, honest error, not a hang, and it taught me the house rule fast: inside a live agent session, all database access goes through the MCP tools; the CLI is the out-of-session maintenance channel. gbrain’s own docs point the same way.

There is one unresolved scar here. After gbrain embed --stale reported a successful backfill from the CLI, a newly spawned MCP server still claimed one chunk had no embedding. Same database file, two different counts. I would love to blame a stale cache in a long-lived server process, but the server was newly spawned, so that theory doesn’t fit cleanly. Round four, tomorrow, starts with a fresh session to see whether the count heals itself on first query — and until then I simply don’t trust either counter.

The database itself is a single 65 MB file under ~/.gbrain. Copy it only with no server running: a live copy of an open database is not a backup, whatever its size.

Semantic scores are uncalibrated, so margins matter more than hits

Round three was a batch of eight real scenarios. The one that mattered most: I stored a real operations document from my own fleet — a search-scope audit checklist, several thousand Chinese characters — and gbrain chunked it into four pieces. Asking “how do I decide which pages to delete or merge” (in Chinese, reworded from the document’s own phrasing) returned the right chunk first at 0.88. A separate query, asking for the audit’s bottom-line conclusion, hit that passage at 0.87.

But scoring worked on a different query too: an unrelated page about ad revenue — Ezoic, nothing to do with the question — inserted itself at roughly 0.80, only ~0.08 below where correct answers landed. gbrain’s doctor command flags that its fact-admission thresholds are uncalibrated for this model, and the page-search scores turned out to have the same character in practice. bge-m3 scores are relative, not absolute: 0.88 means “more similar than 0.80,” not “correct.”

The operating rule I took away — call it a personal heuristic, not a measured threshold — is to read the gap, not the hit. A ~0.08 margin above the noise floor was enough that day, but I would not trust a 0.02 margin for anything that feeds a real decision.

Loud degradation and guarded deletion earned my trust

Four small behaviors earned more confidence than any benchmark would have.

Degradation is loud. With embeddings offline, every search and recall result carries a [gbrain degraded] marker plus a discipline note telling the agent not to treat degraded output as “nothing found.” A memory tool that admits when it’s half-blind is worth more than a faster one that doesn’t.

Blind deletion is blocked on purpose. A bare delete of a revisioned page returns a conflict error, which looks like a bug the first time you hit it. It isn’t — the revision counter is an optimistic lock, and --force deletes cleanly, verified by listing and recalling after. I’d rather be annoyed once than lose a page to a typo.

Empty input doesn’t crash. An empty query returns a doctor hint instead of an exception. Small thing. It means an agent that hallucinates an empty string gets a useful answer instead of a stack trace.

Registration is one command. hermes mcp add wired all eight tools with a single invocation, auto-registered them into the correct agent profile (not the default one), and dropped a brain-first instruction block into that profile’s system file wrapped in replaceable marker comments. Eight-for-eight on the handshake, on a Hermes version (v0.22.5) newer than gbrain’s CI pins (v0.20.0).

Two defects and one policy decision are still open

Honest ledger, because a trial report that only lists wins is marketing.

• Setup self-check false positive. gbrain setup hermes --instructions-only reports turn_hook_not_loaded; the hooks doctor lists the hook as loaded. Re-running the suggested fix reproduces the warning. Function over form — logged, not fixed.
• The stale embedding count described above. Unresolved, reproducible, scheduled for tomorrow’s first query.
• Writeback policy. gbrain asks whether agents may auto-remember user-stated facts (salient), everything, or nothing. My family leans toward salient with provenance — record what the user explicitly says, keep terminal conclusions in the wiki — but that decision is mine to make, not the tool’s.
• Multi-agent concurrency. Two more agents (the ones that drain and the ones that handle outbound links) are supposed to share this brain. Same-brain concurrent behavior is round four material, not something I’ll assert today.

The full setup, pinned to the versions I tested

Versions matter more than adjectives for anything you want to reproduce a year later:

1. Install bun 1.4.4 (from npmmirror — the official installer couldn’t reach GitHub from my network), clone garrytan/gbrain, check out the v0.60.634.0 tag, bun install (283 packages, 79 s), bun link.
2. gbrain init --pglite → a 65 MB single file in ~/.gbrain. No API key, no daemon.
3. hermes mcp add per gbrain’s Hermes setup docs (the current command lives in the repo; at my version it was one line) → eight tools handshake; verified on Hermes v0.22.5.
4. brew install ollama (through a proxy, if your direct route times out) → brew services start ollama → ollama pull bge-m3 (1.2 GB).
5. Stop the MCP server to release the database lock → gbrain embeddings enable --embedding-model ollama:bge-m3 → gbrain embed --stale → start the server again.
6. House rule: in a live session, all database access goes through MCP tools only; CLI is for when no session is alive.

I don’t have a grand conclusion. One day converted me from “another memory tool” to “a shared memory is viable for a small agent fleet, if it degrades loudly, locks honestly, and you treat similarity scores as weather reports rather than verdicts.” Round four is the one that counts: a different agent, a fresh session, asking the brain what yesterday’s agent was doing. That is the entire point of the thing, and I haven’t earned it yet.

This fleet runs on scheduled jobs that have to survive unattended nights — the failure mode I care about most is the silent one, which is why every automation I ship gets a failure detector (https://javaeye.dev/blog/silent-failure-unattended-automation/). And the agents themselves will outlive any single machine, a lesson I paid for in an SSH tunnel setup that quietly chained my Mac mini’s browser to wherever my laptop happened to be (https://javaeye.dev/blog/ssh-tunnel-traps-laptop-to-mini/).