The Vault RAG and MCP System

A single growing record of my work, filled and searched by the systems I use to deliver it: a search index and an MCP server over my Obsidian vault. It is live, in use and monitored; recovery is manual, and self-healing is the next step, not built. This page covers what flows in, the moving parts, and what running it has taught me.

I keep a single, growing record of everything I have done. Most of it fills itself: the systems I use to deliver work write their notes, decisions and handoffs into it as they go. The same systems then retrieve from it, whether the question is what was decided, what was said in a meeting, or how I handled something last time. Nothing has to be written up twice, and nothing gets lost between one piece of work and the next. The setup is AI-led, but what I value is the record.

It started with wanting Claude to reach my notes from a phone as easily as from the desk, and to put things back as well as read them. Keyword search over the vault was the first cut: fine for exact terms, useless for "what did I decide about that?". So I added a semantic layer, and an MCP server in front of it.

Running it costs about £5 a month on Cloudflare. The effort went on keeping the index in step with the vault, and on knowing when it had quietly stopped.

148,364vectors across 8,220 notes

5 daysthe index stopped while health read green

£5a month on Cloudflare in normal running

~3.75 hto rebuild the whole index from scratch, unattended

01What flows in

Project notes

Notes from every project I deliver with these systems, which is now most of my work: decisions, specs, handoffs, working logs.

Meeting transcripts

Every transcript I've been able to get hold of.

Documents

A Markdown version of every document I produce, so a report or a deck is searchable as text.

It is all Markdown, so there is one pipeline for the lot. The first bootstrap indexed 72,128 chunks from 5,099 files. Today 8,220 of 8,893 indexable files are indexed.

02The moving parts

Obsidian Markdown

The vault is plain Markdown files in Obsidian. No database sits between me and my notes.

GitHub as the source of truth

The vault lives in a private GitHub repository. A launchd job commits, fetches, merges and pushes every 60 seconds, whether or not Obsidian is open. Everything downstream can be rebuilt from the repository.

A scheduled Cloudflare Worker

On a timer, with no webhooks, it asks GitHub what changed and processes the diff. Notes split at every heading from H1 to H4, long sections again at paragraph breaks, and each chunk keeps its heading trail.

Vectorize for semantic search

Chunks are embedded with bge-base-en-v1.5 (768 dimensions) on Workers AI and stored as vectors in Vectorize. Metadata sits in D1 and the raw chunks in R2.

An MCP server

This is how those systems reach it, behind bearer auth. They can read: semantic search, search by path, list recent, read a note. They can also write: create a note, append to one, edit one. Writes go to the same GitHub repository, so they ride the same loop as anything I type, down to my desk and into the index. Each edit carries the file's version and a stale one is refused, so nothing can overwrite a change I made a moment before. New notes written this way land in an inbox folder.

I have no figures for query quality or latency. Retrieval was never where the effort went.

03What running it taught me

The bill rose only when something went wrong: in August, when a loop re-embedded the same files, and in October, when the index had to be rebuilt. Both were small beside what they taught me.

Sync tied to an app (August). The Obsidian Git plugin only ran while Obsidian was open, so closing the app or letting the Mac sleep stopped everything, silently. The launchd job replaced it.
The loop (August). A large diff hit the Worker's subrequest ceiling, the stored position never advanced, and the same files were re-embedded every 15 minutes for days. I found it on the bill, not from an alert. Large diffs now go through a resumable queue.
The Vectorize stall (15 to 20 October). The index applied no change after 15:31 UTC on 15 October while still accepting every write. For five days, search served old versions of edited notes and none of the new ones, and the health check stayed green because it measured the sync, not the index. The cause is unknown. I haven't raised it with the vendor; I'm waiting to see whether it happens again. Rebuilding took about 3.75 hours, unattended.

A reachability check on the Worker would have called the August incident healthy every 15 minutes.

What watches it now proves the system is doing its job, not that a URL answers:

Heartbeat. A run stamps success only after the sync resolves. /health/sync returns 503 at 35 minutes of sync age and a monitoring dashboard polls it. The alert path was proved on 17 August.
Index lag. The same endpoint returns 503 when a write stays unapplied for 120 minutes. On 20 October it returned 503 against the frozen index and 200 against the rebuilt one.
Spend guard. A daily job reads yesterday's usage and trips on a repeat of the August pattern.
Runbooks. Diagnose, fix, confirm, by hand, including the rebuild.

Detection is built. Recovery is not. Self-healing is the next step, and it is design only: restart a stalled bootstrap, rebuild an index when lag stays red, resolve known conflicts without me. Nothing does that today.