Can an OKF bundle do the job of a RAG pipeline? A small experiment
I wanted to know whether the things we usually solve with a RAG pipeline can be solved with an OKF bundle instead, at least in principle, and how that performs: in the quality of the answers, in tokens, and in runtime. So I built both for the same set of documents, asked the same 20 questions, and measured.
This is an engineer’s experiment, not a study. It is small, but the numbers are real, and it gave me more understanding and confidence for using OKF in appropriate scenarios.
Two ways to give a model custom context
RAG (retrieval-augmented generation) splits documents into chunks, computes an embedding for each chunk and stores them in an index. For a question, the pipeline retrieves the most similar chunks and puts them into the prompt. The model answers from these chunks.
An OKF bundle is a folder of Markdown files, one concept per file, with a small YAML header and links between the concepts (I described the format in an earlier post). The model does not get a fixed set of chunks. It starts at an index, opens the concepts that look relevant and follows links. Retrieval is a series of small decisions by the model itself.
The motivation: why I care about OKF here
Performance is only half of my interest. A bundle has two properties that a vector database does not have:
It is readable by humans and machines. I can open a concept, read it, correct it, and review changes in Git. An embedding index is opaque. When an answer is wrong, I cannot look into the index and see why.
Access control can use tools that already exist. A bundle is a directory tree. Subdirectories can have their own read and write permissions, the bundle can live in a repository with its own review rules, and different agents or people can be given different parts of the tree. For a vector database, the usual approach is metadata filters that every query has to apply correctly.
I want to be clear about the second point: it is an argument, not a result. I did not test access control in this experiment. At the end I will come back to it, because the experiment taught me one thing about it that I had not expected.
The setup
Documents. I wanted material that is boring, fact-dense and and not contained in the model’s training data. I took five open access papers (CC-BY) on the Neolithic in Europe, published after the model’s training cutoff: settlements, radiocarbon chronology, burial practices. About 56,000 words, which is roughly 159,000 tokens. The papers share sites, cultures and dating methods, so some questions need facts from several papers.
Questions. 20 questions, with the ground truth written from the paper texts, not from the bundle or the index:
8 single facts from one paper (“How many soil samples were analysed …?”)
6 questions that need facts from two to four papers
3 open questions that need a summary, one of them deliberately vague
3 questions that cannot be answered from the papers, where the right answer is “this is not reported”
Four approaches, all with the same model (Claude Sonnet 5.5) and the same instruction to answer concisely and not to guess:
No context: the model alone.
RAG: hybrid retrieval (local embeddings plus BM25 keyword search), the top 8 chunks go into the prompt.
OKF: the model works in the bundle with file tools (read, grep, glob), starts at
index.mdand navigates by itself.Full context: all five papers in the system prompt. This only works because the dataset is small, so I treat it as a ceiling for this experiment, not as a competitor.
Answers were graded against the key facts of each question by a second model (Claude Opus 5.5) that did not know which approach produced the answer. The model calls ran through the Claude Code command line in headless mode, which reports usage per call. That adds a constant overhead per call (about 600 tokens without tools, 3,200 with file tools), which is small against the numbers below.
Building the two knowledge bases
RAG index |
OKF bundle |
|
|---|---|---|
Wall clock |
47 seconds, on a CPU |
14 minutes |
LLM calls |
none (local embedding model) |
6: five extractions in parallel, one merge |
Tokens |
104,000 embedder tokens, no LLM tokens |
234,000 output, 507,000 fresh input |
Cost at list price |
$0 |
$4.68 |
Result |
222 chunks, 0.9 MB |
160 concepts, 31,000 words, 118,000 tokens |
Two observations. First, the bundle is a lossy but modest compression in words: 31,000 words in the OKF bundle against 56,000 in the papers, or about 78% of the 40,000 words once the reference lists are left out. In tokens, the bundle is about 118,000 tokens, against 100,000 for the papers without references. Markup, YAML headers, links, footnote labels and paper ids cost more tokens per word than running text. Putting the whole bundle into a prompt would save little compared with putting in the papers. Second, the merge step, which combines concepts that appear in several papers, took 543 tool turns and three times as long as the extraction of all five papers together. That will be the bottleneck when the dataset grows. But this was a naive approach without any optimizations.
Results
Approach |
Score (out of 20) |
Median time |
Input tokens per question |
Cost per question |
|---|---|---|---|---|
No context |
6.0 |
9.3 s |
0.7k |
$0.009 |
RAG |
14.5 |
6.6 s |
6.9k, one turn |
$0.032 |
OKF |
17.0 |
9.7 s |
6.9k fresh + 16.9k cached, 5.3 turns |
$0.039 |
Full context |
18.5 |
4.9 s |
8.5k fresh + 150k cached |
$0.054 |
A correct answer counts 1, a partially correct one 0.5. The time for RAG includes retrieval, which took 1.9 seconds, mostly the start-up of my command line script. Cached tokens are billed at a fraction, which is why full context is cheap here despite 150,000 tokens per question.
By question type:
Approach |
Single (8) |
Cross-paper (6) |
Open (3) |
Not answerable (3) |
|---|---|---|---|---|
No context |
0.5 |
2.0 |
1.0 |
2.5 |
RAG |
7.5 |
2.5 |
1.5 |
3.0 |
OKF |
8.0 |
4.5 |
1.5 |
3.0 |
Full context |
8.0 |
5.5 |
2.0 |
3.0 |
Interpreting the numbers
Quality. For single facts, RAG and OKF are close: 7.5 and 8 out of 8. The difference appears on questions that need several papers. RAG scored 2.5 of 6, OKF 4.5. The clearest case was a question about the end date of the LBK, the first farming culture, in four papers. All eight chunks that RAG retrieved came from the same paper, and the model concluded, correctly given what it saw, that the other dates were not reported. A fixed list of the eight best-matching chunks does not guarantee that different documents are represented. An OKF bundle organizes knowledge by entity, and a merged concept carries the attribution of every paper that contributed, so the model can find the fact in multiple documents.
Tokens. The fresh input is about the same for RAG and OKF, 6.9k tokens per question. But OKF needs 5.3 turns instead of one, and each turn carries the growing context: in total about 24k input tokens per question, most of them cached, and more than twice the output tokens. At list price the difference is small, $0.039 against $0.032 per question. The build cost is the larger one: at that difference, the extra $4.68 never pays back in money. It pays back, if at all, in answer quality.
Runtime. Median times are not too far apart (6.6 against 9.7 seconds), but OKF is less predictable: the slowest tenth of the questions took up to 14.9 seconds for OKF and 8.6 for RAG, because the model sometimes walks further through the bundle.
All in all the exact numbers for quality don’t tell too much, because they are based on a very small dataset and I didn’t take the time to dig deeper into the content to verify the exactness of the judgment.
Good to know
The model without any context already knew parts of the best-known site. Even post-cutoff documents are not fully safe from prior knowledge, which is to be expected for an academic field. But the no-context approach abstained on 8 of 20 questions rather than guessing, which is a reassuring behavior of modern LLMs (better than hallucinations).
The full-context approach won on both quality and time, and it was cheap only because of prompt caching. It stops being an option when the dataset is ten times larger, or when documents have different readers.
The OKF bundle is not smaller than the papers’ text in tokens: about 118,000 tokens, against 100,000 for the papers without their reference lists (159,000 with them). It has fewer words, but its markup costs more tokens per word.
About access control
The experiment gave me one practical insight about the second argument for OKF. If the bundle’s directories are the permission boundary, then the boundary of a concept has to match the boundary of its sources. The merge step of multiple bundle files into a single concept files carries facts and footnotes from all five, which creates a risk of information spill to unauthorized users. So the bundle has to be organized, and merged, along the lines of the permissions, and the agent that writes the bundle must not cross them. With a vector database the same problem exists as metadata on chunks, and I would not claim it is easier there. What I do think is different is that it is visible: in a file tree I can see which file sits where, check who can read it with tools I already use, and review changes before they are merged.
I did not verify this in practice, and it needs a proper test: two agents with different permissions on the same bundle, and questions that try to reach across the boundary.
What this experiment cannot tell
Five papers and 20 questions are a very small sample size.
The ground truth and the judge are assisted by language models, and the judge may be strict. Honestly I don’t understand anything about the topic covered in the papers and I did not do the hard work to evaluate the proposed ground truth answers from Claude Opus nor the grading.
My RAG is decent but not tuned: no re-ranker, no query rewriting. A better RAG would possibly perform better, but so could an optimized OKF bundle builder.
Conclusion
An OKF bundle can do the job of a RAG pipeline in principle, and in my experiment it did it better on questions that need facts from several documents: 17 against 14.5 points out of 20 at roughly comparable tokens count and a comparable median time. The price is a more expensive build, more turns per question and a less predictable runtime, which will probably scale with the size of the knowledge base.
Reference: everything needed to reproduce this experiment
All material was produced on 2026-11-10. The questions, the answers, the scripts and the logs are available as downloads below. The RAG index with its embeddings and text chunks, and the OKF bundle, contain text and derivatives of the papers and are not published here; you can request them from me through the contact page.
The papers (five open access articles, CC-BY, published 2026-06-01 or later). I link the publishers’ PDFs instead of redistributing them. Publishers may change or move files; the DOI is the stable reference.
Title |
Journal, date |
DOI |
Full text |
|---|---|---|---|
Settlement and temporal dynamics during the LBK and post-LBK transition |
Archeologické rozhledy, 2026-11-03 |
||
Neolithic Bodies in Vráble – 7000 year-old Headless Human Skeletons in an Enclosed LBK Settlement in South–West Slovakia |
Proceedings of the Prehistoric Society, 2026-06-02 |
||
The issue of the ‘resurgence’ of hunter-gatherer lifestyles in the region between the middle courses of the Southern Buh and Dniester rivers during the post-LBK period |
Archeologické rozhledy, 2026-08-22 |
||
Early farming and wild plant exploitation in the Linear Pottery Culture of southeastern Poland: new evidence from the sites Rzeszów-Zwięczyca and Łańcut |
Vegetation History and Archaeobotany, 2026-06-26 |
||
Interpreting funerary variability in the Alsace Plain between 5300 and 4400 BC |
Proceedings of the Prehistoric Society, 2026-08-11 |
papers.json has the same list with
the OpenAlex identifiers. Papers were selected through the OpenAlex API (open access,
license CC-BY, published since 2026-06-01, topic queries on Neolithic and LBK settlements)
and converted with pdftotext (poppler, reading-order mode, no -layout). Images were
extracted with pdfimages -png -p; they are not used in this experiment.
Questions and answers
questions.json: the 20 questions with ground truth, key facts used for grading, and the papers each draws on. The ground truth was written from the paper texts, not from the bundle or the index.results.json: for every question and approach the exact answer, the judge’s verdict and note, tokens (fresh, cache write, cache read, output), turns, time, list-price cost and, for RAG, the ids of the retrieved chunks.
The two knowledge bases
RAG index (on request):
embeddings.npy(222 chunks, 384 dimensions, float32, normalised),chunks.jsonl(chunk id, paper, text) andbuild.json(build metrics). Embedding modelBAAI/bge-small-en-v1.5throughfastembed0.7.3, BM25 throughrank-bm250.2.2, Python 3.12. Chunks: paragraphs packed to 300 words with 50 words overlap. Retrieval: top 30 of dense and BM25, reciprocal rank fusion (k=60), top 8 into the prompt. The chunk ids inresults.jsonrefer to this index.OKF bundle (on request): 160 concepts plus indexes and log, valid OKF v0.2. Built with five parallel extraction calls (one per paper) and one merge call. The exact prompts are in the scripts.
Models and settings
Answering:
claude-sonnet-5-5. Judging:claude-opus-5-5, blind to the approach.Calls through Claude Code 2.1.322 in headless mode (
claude -p --output-format json), with a short custom system prompt, no MCP servers, no skills, one run per question and approach. Tools: none for no-context, RAG and full context; Read, Grep and Glob for OKF. The working directory was outside any repository, so no project instructions were loaded.Constant overhead per call: about 600 tokens without tools, 3,200 with file tools (measured in
build-logs.zip).Token sizes of the data: each text was put into the system prompt of one call and the reported input tokens were read, minus the overhead of about 600 tokens. Papers with references 159,000, papers without references 100,000, OKF bundle (169 files with headers) 118,000 (raw numbers in
build-logs.zip).Costs are list prices as reported by the tool, not what was actually billed.
Code and raw data
scripts.zip:fetch_papers.py(download and extraction),rag.py(index and retrieval),build_okf.py(extraction and merge prompts),run_queries.py(the four approaches and their system prompts),judge.py(the grading prompt),summarize.py(the tables above),export_blog_assets.py(this package).build-logs.zip: the raw result JSON of the six OKF build calls and the overhead measurements.
Reproducing it
Fetch the papers:
fetch_papers.py <dataset-dir>withpapers.jsonsaved asmanifest.jsonthere.Build the index:
rag.py build <dataset-dir> <index-dir>.Build the bundle:
build_okf.py mapandbuild_okf.py merge(needs Claude Code).Ask the questions:
run_queries.py run, grade withjudge.py, tabulate withsummarize.py.
Expect different numbers in detail: model answers are not deterministic, I ran each cell once, and a rebuilt bundle will differ in wording and structure. The conclusions that would have to hold are the direction of the results, not the second decimal.