Memory you own

The previous post ended on agents: what they are, what autonomy means, and why the choice you actually make is the harness you put around them. This one is about the part of the harness everyone skips past, because it sounds like a solved problem and because it does not demo well.

Memory.

Not the context window. A context window is a table you set for one meal. Memory is the pantry, the inventory system, and the person who remembers what was thrown out. When an agent forgets a decision from last Tuesday, nothing was "lost" from the model — the model never had it in the first place. Something outside the model failed to hand it back.

That thing is an index, and the question this post asks is whether you own it.

Three problems wearing one word

Most "memory" conversations collapse three different problems into one, and then argue past each other.

Storage is where the text lives. Boring, mostly settled, usually a database or a folder of files.

Retrieval is how the right piece comes back. This is where the engineering is, and where the choices actually differ.

Continuity is whether any of it survives a week of not thinking about it — whether next Monday's session starts with your history or with a blank stare. That is the agent-memory problem, and it is the next post.

Continuity is the why. Retrieval is the how, and this post is about the how — which sounds narrower than it is, because every continuity system anyone sells you is a retrieval system with a scheduler bolted on.

Four arms, not one index

Almost every practical retrieval system pulls from the same four families. They fail differently, and that is the useful part.

Lexical — the words themselves. Best understood as BM25, a ranking function (a scoring rule that rewards documents containing the query's rare terms, discounted by length). It is unglamorous but exact. It handles names, error codes and part numbers without effort. Most importantly it is immune to the primary failure of embeddings: a vector cannot tell you why it scored a document highly. Lexical can. You can point at the term.

Vector — meaning rather than wording. Find documents that say the same thing in different words. It cannot find what you cannot paraphrase, and it blurs exactly the tokens — identifiers, version strings, surnames — that operations work turns on.

Multimodal — meaning across media. Embeddings that put text, images, audio and video in one vector space, so a question about a slide can find the video it appeared in.

Graph — relations and time. Entities and the edges between them, so "where does Alice work" is answerable from two facts that were never stored in the same sentence.

None of these is the right answer. Every serious system runs several of them and merges the results, which is why "RAG versus agents" was always a false fight.

What the numbers actually say

Two pieces of evidence from this fortnight, both worth reading closely — one because of what it measures, one because of how it measures.

The embedder got smaller and stranger

EmbeddingGemma 2 is a Google DeepMind model under Apache 2.0: 740 million parameters total, made of a 270M text model with modular vision (170M) and audio (300M) encoders. It maps text, code, images, video and audio into one shared 768-dimensional space — not four separate indexes with four separate scores. Eight-thousand-token context, described on the model card as covering "minutes of audio or video". Weights are 1.49 GB in BF16, which fits a laptop.

It also supports Matryoshka truncation, and the headline there is storage: one long vector you can cut down to 512, 256 or 128 dimensions and store only the short version, which the card puts at up to a sixfold reduction for minimal quality loss. If you are paying for vectors by the gigabyte, that is the number that matters.

"Minimal" deserves a number, and the model card's own benchmarks are the honest place to look. On multilingual text retrieval it is a wash against the previous version — 61.36 against 61.15 mean task score. On code retrieval it jumps: 78.68 against 68.76. The headline is not "smarter text". It is "multimodal, and much better at code".

The ownership angle is not patriotism about local inference. It is execution and licence: Apache 2.0 weights you can hold, on hardware you can unplug, with no API to be deprecated out from under the index you built on it.

Sources EmbeddingGemma 2 model card — https://huggingface.co/google/embeddinggemma-2 Technical report — https://arxiv.org/abs/2610.37364 Launch post — https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2

Someone measured the trade-off

Sam Witteveen built a video retrieval system on that model — fifteen of his own videos, split into 427 segments of about thirty seconds each, transcribed first so the cuts never land mid-sentence — and then tested it properly rather than demoing it.

Each segment is embedded five ways into the one shared space: the transcript, a single mid-segment keyframe, the clip's frames at one per second, the raw audio with no transcript at all, and a fused input with all three interleaved. He then generated 228 questions from the clips themselves, so he knew which segment held each answer, and sorted them into spoken and visual.

Recall@5 — the right segment in the top five — at full 768 dimensions:

Read the columns separately, because averaging them hides the story. On questions written from the transcript, text is near-perfect — 0.98 — and that comparison is biased in text's favour, which he says plainly. On questions about what is on screen, the multimodal views win: fused 0.78 and video 0.75 against text's 0.67.

Two findings worth more than the table.

Raw audio with no transcript reached 0.82 on spoken queries. Not better than transcription, but close enough to ask the uncomfortable question: if the embedder hears the sentence itself, what is the whisper pass for? His own answer is in the pipeline — the transcript is what makes sentence-respecting cuts possible, not what makes search work.

Rank fusion beat fused vectors at small widths. At 768 dimensions the fused view is the best multimodal option. Truncate to 128 and the picture changes: a hybrid that embeds the three views separately and merges their rankings by reciprocal rank fusion — a rule that scores each candidate by its position in each list rather than by its raw similarity — holds 0.72, beating the fused vector's 0.69 and every other view at that width. The very technique used to fuse multiple search types in complex agent-memory systems also happens to rescue the storage-saving option.

The Matryoshka claim holds up better than the model card implies: 256 dimensions keeps roughly 95 percent of full-width quality. It is 128 that costs, at 10 to 15 points, unless you fuse by rank.

Sources Sam Witteveen, "Building a Video RAG with EmbeddingGemma2", 15:49 — https://www.youtube.com/watch?v=Q8T9lvtGFNw Code, pipeline and full results table — https://github.com/samwit/basic-video-rag-with-embeddinggemma2 (15 videos, 427 segments, 228 synthetic queries; figures above are his, from his repo's README)

The lexical index got a real engineering story

Meanwhile, in the unglamorous arm: pg_fts is a BM25 index built as a PostgreSQL access method — the same structural slot as a B-tree or GIN index. That distinction is the whole ownership argument. A search service is a thing you run beside your data and trust; an access method is a thing your data already has, and you can read its plan.

The reason it belongs in a post about ownership is not the feature list. It is the benchmark directory. Two million, one hundred and eighty-eight thousand Wikipedia articles; one indexed column; local NVMe rather than RAM; eight runs per query with the median of the last five; and every result gated behind a parity check that compares the index's answer against an exact sort over the raw rows. A result does not get published unless it matches.

Then the part that makes it trustworthy. They document their own mistakes. An earlier timing protocol dropped only one warm-up run, and they published the consequence: a competing engine measured at 6.86ms under that protocol whose steady state is 2.13 ms — a threefold error, in the direction that flattered their own project. They re-measured by hand. Elsewhere they report the numbers that hurt: turning on phrase positions speeds phrase queries by 36 times and grows the index 1.85 times, and the default is off. A normal build on the default 64 MB maintenance_work_mem leaves eight segments needing a 216-second merge and lands at 8,613 MB; give it 1 GB and it builds in one segment at 4,605MB. Half the size for two more minutes of build time.

That is what an inspectable index looks like in practice. Not a claim about being fast — a record of how the claim was tested, and where it used to be wrong. You can EXPLAIN the query, read the raw sort behind it, and see for yourself whether the ranking did what the README says.

Sources pg_fts — https://github.com/gburd/pg_fts (v1.17.0, 9 Nov 2026); benchmarks and methodology in bench/BENCHMARK_SUMMARY.md pgvector — https://github.com/pgvector/pgvector (v0.8.7), the vector half of the same database

The graph question

Graph memory is the arm with the loudest marketing and the weakest published evidence, so it gets its own section rather than a line in a table.

The sceptical case is not hypothetical. Work through 2025 and 2026 on graph-based retrieval keeps finding that GraphRAG frequently fails to beat plain vector retrieval on ordinary question answering, while costing considerably more to build. A paper titled When to Use Graphs in RAG says so directly, citing LightRAG's own studies among the reports.

So we will not tell you graphs are the next thing after loops. We will tell you what we think the line is, and you can check us on it.

Graphs win when the question is relational or temporal, and lose when it is topical.

"Which services depended on the certificate that broke, and what else did that team own?" is a traversal. Vector search fails on it not because the embeddings are bad but because the answer is never in a single passage; lexical search fails because it has no connective tissue for "who belongs to what". Neither failure is fixable by tuning, which is what makes the traversal a genuinely different shape of question.

"What did we decide about the cache layer?" is topical, and a decent lexical index with a competent embedder beats a knowledge graph built to answer it.

That predicts where graph memory should earn its keep: infrastructure, incident history, dependency maps, anything where "what else" and "when" are part of the question. That is also, not coincidentally, the shape of a personal wiki and of an agent's own history — which is why we are building one, and why we will report what it costs rather than what it promises.

Sources When to Use Graphs in RAG — https://openreview.net/pdf?id=i9q9xDMjG7 WildGraphBench, graph retrieval on wild-source corpora — https://arxiv.org/html/2602.02053v1 LightRAG — https://github.com/hkuds/lightrag

Where each fits, and for whom

The same rule as the runtime post: no single winner, only fit.

One machine, one wiki, one person — files, a lexical index, and an embedder. That is it. Markdown in a directory, BM25 for the names, vectors for the paraphrase, and the ability to grep the whole thing when the index lies to you. Our own wiki is exactly this: plain files, frontmatter, an index file regenerated by a script, no server.

A team's internal docs — same four arms, more care about permissions. The lexical arm carries error codes and ticket numbers; the vector arm carries the question someone actually typed. Both in the database you already back up.

Media archives — this is where multimodal earns its cost and nothing else will do. If the answer is in a frame or a spoken sentence, text-only retrieval is not a cheaper option, it is a different question.

Incident and dependency history — the one place we would argue for a graph, and the place to measure it honestly rather than assume it.

And one convergence worth noting for the next post: the current generation of agent-memory systems, built to their own designs, describe retrieval the same way — semantic, keyword, graph and temporal searches run in parallel, then fused by rank rather than by score, cut to a token budget instead of a top-k. Four arms, one architecture. Nobody serious is picking one.

What's in your control

• The index format. A BM25 index inside your own PostgreSQL is a table you can inspect, a query you can explain with EXPLAIN, and a backup you already take.
• The embedder. At 1.49 GB and Apache 2.0, the vectoriser runs on the machine in front of you. The licence matters more than the size: you can retrain it, ship it, and keep using it after someone deprecates an API.
• The chunking. Transcribe before you split. Thirty seconds is arbitrary; not cutting mid-sentence is not.
• The k you retrieve and the width you store. Those two dials are a measured trade — smaller vectors plus a wider k is a better deal than it looks.

What's not in your control

• Benchmark numbers, including ours. Vendor benchmarks measure the vendor's harness. Every figure in this post that came from a model card or a project site is a claim about itself until someone outside reproduces it. Our own numbers come with our hardware in the caption or they do not go in.
• How good retrieval can get when the corpus is dirty. No index repairs a wiki that contradicts itself.
• Whether the models you embed with keep existing — only whether you can swap them without re-architecting.

What we're testing next

We run this stack. A markdown wiki with a regenerated index, PostgreSQL underneath, a hybrid of local and hosted inference, and an agent that reads all of it. The next post takes the other half of memory — continuity, what an agent keeps between sessions, and what it costs to keep it — including the parts of our own setup that are currently broken.

Because the honest version of "memory you own" isn't a claim of perfection. It's a claim of agency: when the system breaks, you can read the index and find out why.