I Built a Local RAG Search. Having the Right File Wasn't Enough
I Built a Local RAG Search. Having the Right File Wasn't Enough
Nabil Samari
July 29, 20269 min read
Share
For a long time I had one really frustrating problem. The information was all there — I just couldn't reliably find it.
My notes lived in Obsidian. Some files came through Google Drive and Nextcloud. Client and artist data sat in their own structures. When I wanted to check an old decision, a price, or an event detail, I usually knew I had written about it at some point. I just didn't know where.
I built a local RAG search to solve this. The idea sounded straightforward: index my own data, search semantically, and let a local language model answer based on sources.
The project was inspired by a Cerebras Systems blog post. Cerebras is a US-based AI compute company founded in 2015 that builds wafer-scale processor systems for model training and inference. It's not a note-taking app or RAG startup — it's fundamentally an AI hardware and cloud company.
Cerebras's How We Built Our Knowledge Base describes their internal knowledge bank, which they say handles over 15,000 queries a day. Data is pulled from Slack, wikis, code, and incident reports. Search combines exact keyword matching, embeddings, term rarity, and recency. Each approach produces its own ranked list, the lists are merged with RRF, and the best candidates are further re-ranked by a language model before a sourced answer is returned.
Nick Saraev walks through the same architecture in his video Cerebras Killed Notion, Obsidian, and Your "Second Brain". He takes the Cerebras article as a starting point and uses a coding agent to build ingest pipelines for Slack, email, GitHub, and YouTube. In his own demo with the knowledge base, the system got 17 out of 20 questions right — versus zero without it. That's one demo, not an independent benchmark, but the idea was compelling: data can stay where it's created, as long as you build solid ingest, search, and sourcing around it.
My implementation is much smaller and entirely local. I wanted to see whether the same principle would work on top of Obsidian, Drive, and Nextcloud — without sending any material to an external model service.
The first working version looked good. Then I started measuring whether it was actually finding the answer.
That's when everything went back to the drawing board.
The Right File Isn't the Right Answer
In RAG evaluation it's easy to measure the wrong thing. If search returns the correct file, the result looks successful. But a long document may be split into dozens of chunks. The right file shows up, but the chunk that carries the actual answer falls outside the top results.
This gap turned out to be large in my tests.
I built a locked test set of 21 real work scenarios. Each question had a defined acceptable source path and specific evidence markers that had to appear in the returned chunk. I measured two things separately:
Whether the correct file appeared in the top 3
Whether the chunk that actually contained the answer appeared in the top 3
Pure vector search achieved Path Recall@3 of 0.76. Evidence Recall@3 was only 0.71. The right file was found more often than the right passage inside it.
That's a pretty big gap when an assistant needs to answer about a price, a date, or a contract term. A model can't safely infer a missing sentence just because the filename looks right.
Vector Search and Keyword Search Don't Automatically Help Each Other
I combined semantic vector search with SQLite FTS5 keyword search. Vector search found semantically similar expressions. FTS5 found names, numbers, and exact terms. The lists were merged with Reciprocal Rank Fusion, which uses rank position rather than trying to compare raw scores from two different retrieval methods.
Path Recall@3 rose from 0.76 to 0.81. At the same time, Evidence Recall@3 dropped from 0.71 to 0.67.
This wasn't what I expected, but it was useful. Hybrid search brought more correct documents into the candidate pool, but the ordering still didn't favour the evidence-carrying chunks. More retrieval technique did not automatically mean a better answer.
I added a local Qwen model to re-rank the top 12 candidates against the question. I didn't give it the entire knowledge store or free rein. It got short request-specific candidate IDs, bounded excerpts, and a strict JSON schema. Unknown IDs, duplicates, or malformed responses were rejected.
After this step, Path Recall@3 reached 1.00 and Evidence Recall@3 hit 0.952. Out of 21 questions, 20 got the evidence-carrying chunk into the top 3. One genuinely missed: the right CRM file was found, but the right passage didn't rank high enough.
I didn't tweak the test to get a perfect score. It stayed as a documented miss.
Scores from Different Collections Can't Just Be Thrown Together
Search had another deceptive trap. Data came from multiple Chroma collections. Each collection returned its own distances, but the numbers weren't directly comparable.
If collection A's best hit has a distance of 0.42 and collection B's best has 0.61, that doesn't mean A's hit is globally better. Collections can have different distributions, content, and index structures.
I fixed the merge to use per-collection rank. The best hit from each collection gets a chance before the deeper tail of another collection. At the same time, chunk IDs got a collection name prefix. Otherwise, identically named records from two collections could collide and vanish during deduplication.
Deduplication was also too aggressive at first. I kept only one hit per file path. That sounds clean, but it also throws away a second chunk from the same document that might carry the answer. The new logic distinguishes different chunks from the same path and can bring adjacent neighbour chunks into re-ranking.
Small technical details. Except those are exactly where RAG either works or lies convincingly.
The Privacy Test Found a Real Leak
Locality was the premise of the whole project. Content from Obsidian, Drive, and Nextcloud never leaves for an external model service. Embeddings and re-ranking run within my own network.
Local still doesn't automatically mean safe.
I built a filter that rejects diary and archive paths, family-related material, secret-type files, and high-confidence credentials. Finnish IBANs are redacted at the point data is read for vector search.
Then a held-out behaviour test found a 2FA recovery code embedded in an ordinary project document. The file didn't look like a secrets file, so the original path filter didn't catch it.
I wrote a regression test first. Then I expanded the content filter to recognize Finnish- and English-language 2FA and MFA recovery code headings. A rebuild removed three overlapping chunks from the index. In the test, the assistant was asked directly for the code. It didn't appear in the tool output or the answer.
This was one of the project's most important outcomes. A security rule that hasn't been challenged in a test is basically a wish.
Source Text Can Try to Command the Model
In RAG, a retrieved document is not a trusted instruction. It's data.
A note can accidentally or deliberately contain text that looks like an agent command. That's why source markers and terminal control characters are neutralized before local re-ranking, and again before the package is handed to the assistant. The model may only choose from candidate IDs the search system already knows about.
The final evidence package is capped at three sources and 4,000 characters. Each source has a machine-verifiable start and end boundary. If the package is broken, the assistant doesn't get it.
This may sound overly cautious. My data includes client history, prices, contract terms, and personal notes. I'd rather the search fails visibly than returns a convincing answer from the wrong passage.
An Index That Doesn't Update Is a Silent Time Bomb
The first real-world query uncovered one more problem. The vector store and the keyword index were out of sync. One ingest pipeline was still pointing to an old embedding service, and the FTS5 index had to be built manually.
I fixed the upstream ingests to use the current local embedding service. Then I wired FTS5 rebuild to successful Obsidian and Drive/Nextcloud ingests. The pilot has no separate scheduler. The existing ingest remains the source of truth, and a standalone oneshot service updates the keyword index only after a successful run.
A process lock prevents two updates from writing to the SQLite index simultaneously. Service files live in the repo, and a rollback check compares hashes of running files against version-controlled copies.
The final index contains 5,523 approved chunks. A dry run rejected 301 chunks hitting private paths, nine secret-referencing paths, and three secret contents. The built index had zero rejectable rows.
Why This Didn't Become an Automatic Everywhere AI Memory
The final 21-question test achieved Path Recall@3 of 1.00 and Evidence Recall@3 of 0.952. A separate 12-question held-out set scored 1.00 on both. No re-rank fallbacks were triggered in either run.
Quality was good enough for manual use. Speed isn't yet there for an invisible background feature. Local re-ranking took a median of 8.47 seconds on the main test and 7.79 seconds on the held-out set.
That's why search only triggers when I explicitly ask the assistant to query local knowledge. It doesn't read Gmail, doesn't inject context into every conversation, and doesn't replace regular web search.
I now have a working local knowledge search. I still don't call it a finished universal memory. The test covered 33 retrieval questions and three behaviour cases. The next important data will come from real use: what it doesn't find, where it's slow, and when a source is ambiguous.
The Most Important Part of RAG Wasn't the Language Model
The project uses BGE-M3 embeddings, ChromaDB, SQLite FTS5, RRF, and a local Qwen re-ranker. Every one of them affected the result.
But the most important part was the test that separated the right file from the right evidence.
Without it, I could have looked at a clean search result, recognized a familiar filename, and declared the system working. In reality, some answers would have been based on the wrong chunk, one secret would have leaked through the index, and the stale keyword index would have broken silently.
Building the local RAG wasn't the hardest part. The hardest part was building a way to see when it was wrong.
