Agent memory layers don't need an LLM deciding what to remember

Most agent memory setups run a model call on the way in. Something reads the turn, decides whether it's worth keeping, rewrites it into a "memory", tags it with a type and an importance score. That's a second inference pass on every interaction and I think it's the wrong place to spend one. Cost isn't the issue. The judgment being unauditable is. When the agent fails to recall something you can't tell whether retrieval missed it or the extractor decided six days ago it wasn't worth keeping. Two different bugs, same symptom, and separating them means reading logs that mostly don't exist. Take that stage out and you have storage, an embedding, retrieval. Which is what a memory layer is. I run memU partly for this reason. Core is around 500 lines doing those three things, exposed as commit, list, retrieve. What it keeps is readable Markdown, embedded and indexed into a local sqlite db. Apache-2.0. You lose the distillation, obviously. Raw turns are noisier than summarised ones so retrieval has to be better to compensate. I'll take that trade because a noisy store you can read beats a tidy one you can't inspect, but I'd take the other side seriously from anyone who has actually measured it. Being straight about the cost since someone will find it anyway: self-hosting does not get you off the hook for an embedding provider, you still need a key for that. It's single machine too, syncing across boxes is their hosted mode. And recall quality is entirely downstream of whichever embedder you point it at. With a small one the recalls go vague on anything abstract. Facts and procedures come back fine. A preference I mentioned once in passing, much less reliably. Anyone still running the extraction step in their harness and finding it earns its keep? Would like to know what it's buying that retrieval isn't. submitted by /u/derspenti [link] [comments]

Source: r/LocalLLaMA | 2026-08-05

Related