I benchmarked classic vector RAG vs Google's new OKF format vs both combined — same corpus, same 7 questions, all local (Ollama + ChromaDB)

Google Cloud published OKF (Open Knowledge Format) on June 12th — a spec for storing curated knowledge as a directory of markdown files with YAML frontmatter. One concept per file, linked to each other, with an index.md for progressive disclosure. The only required field is type. I wanted to know whether it actually fixes anything, so I built a test corpus and measured. Everything runs locally: qwen3:8b + nomic-embed-text + ChromaDB, no external APIs. SETUP - Corpus: 60 markdown files of fake-but-realistic company docs (wiki, table schemas, ADRs, 40 support tickets). 85 chunks at 800/100. - OKF bundle: 9 curated concepts covering the same ground. - 7 questions, each designed to trigger a different retrieval failure mode. RESULTS (7 questions) RAG OKF OKF+RAG correct 2 3 4 tokens 6341 8625 8435 Nothing passes. The combined layer gets twice what plain RAG does, at ~33% more tokens. THE ONE THAT SURPRISED ME Question: "how do we calculate revenue?" The corpus has a 2023 doc (deprecated, verbose, 4000 chars) and the current 2026 spec (terse, 500 chars). The deprecated doc splits into 7 chunks, the current one into 1. Three of the top-5 retrieved chunks came from the deprecated doc. The correct document ranked 15th out of 85 — behind a glossary, a customer table schema, and a support ticket about shipping costs to the Canary Islands. Raising k to 15 doesn't help: you'd pull in 6 chunks saying the wrong thing against 1 saying the right thing. A reranker can't fix it either — there's nothing in the chunk text indicating which is current. The date isn't in the chunk. OTHER FAILURE MODES THAT FIRED - Chunker split an 18-column schema table. The right file WAS in context; the table wasn't. Model said "I don't know" at both k=3 and k=5. - Composition: a metric definition needs 3 rules living in 3 separate files. RAG retrieved 2 of 3 and answered confidently, citing sources, never hinting anything might be missing. - Interesting pattern: it said "I don't know" when it had almost nothing, and said nothing when it had almost everything. It goes quiet exactly when it's most expensive. WHERE OKF LOSES Long-tail questions. "Was there an incident with duplicate orders in March?" — plain RAG nailed it over 40 messy, unreviewed tickets. Curating those by hand would be absurd. OKF alone failed it. TERMINOLOGY CAVEAT I'm using "RAG" as shorthand for the classic vector implementation. Strictly, an agent navigating an OKF index is also a RAG pipeline — just with structured retrieval instead of vector retrieval. The precise framing is "classic vector RAG vs structured retrieval over OKF". Someone rightly called me out on this. Full code, corpus, bundle and the raw results.txt: https://github.com/JoaquinRuiz/rag-vs-okf git clone + uv sync and you can reproduce it. Curious whether anyone gets different numbers with a bigger model — question 4 was unstable across runs for me. submitted by /u/jokiruiz [link] [comments]

Source: r/LocalLLaMA | 2026-08-03