Inspect chunks
See exactly where the document was cut, what each piece contains, and which sentence ended up split across two of them.
Retrieval, made visible
Don’t just build RAG. See why it works.
A hands-on lab for chunking, embeddings, BM25, vector search, hybrid ranking and evaluation — on a real corpus, with real measurements, and without an LLM bill.
01 · input
“why does my container keep restarting”
384 dimensions
02 · retrieve
3 chunkers × 3 modes
9 searches, 1 embedding
03 · rank
top 5 chunks, per configuration
compared by rank, not score
- Retrieval modes
- 3
- vector, BM25, hybrid
- Hidden steps
- 0
- every chunk is readable
- Labelled questions
- 38
- 28 tuning, 10 held out
- Conclusions reversed
- 1
- by the held-out set
01 / The retrieval lab
One query. Three ways to find it.
Pick a question, then change the chunker and the search mode. Watch the evidence reorder. Every result below was retrieved from a live index and recorded — nothing here is illustrative.
Ranked evidence
No relevance score is shown. A cosine distance and a BM25 score are not on the same scale, so a single column of numbers across three modes would be meaningless. Rank is comparable, and rank is what changes.
02 / From black box to glass box
Every step leaves evidence.
Compare retrieval
Keyword, vector and hybrid on the same question, side by side, with the section each one put first.
Measure quality
Recall@K over labelled questions, and a held-out set that decides whether an improvement was real.
01 — The problem
A question is words. The answer is buried in 117,000 characters.
24 sections of Kubernetes documentation. Someone types a question. Something has to decide which part to hand back. “why does my container keep restarting”. Something has to decide which part of those 24 sections to hand back.
Two independent decisions control the rest: where you cut the document, and how you search the pieces. Three of each, nine combinations, all measured. where you cut the document, and how you search the pieces. Three ways to cut, three ways to search, nine combinations. Mimiron runs all nine and scores them against the same questions.
- 01 The question is embedded once — 384 numbers
- 02 Three collections, one per way of cutting the document
- 03 Three searches each: vector, BM25, hybrid
- 04 Nine ranked lists, compared by position
02 — Cutting
The same page, cut three ways.
One document, cut three ways. Where a chunk ends decides which answers are findable.
Fixed size
chunk_text(text, 1000, 200)
Cut every 1000 characters, advancing by 800, so each chunk repeats the last
200 of the one before. A sentence on a boundary survives whole in
one of them. Cost: 25% more vectors.
chunk_size − overlap = 800, so each chunk repeats the last 200
characters of the one before it. A sentence straddling a boundary survives
whole in one of them. Cost: 25% more vectors to store and compare.
repeated text
Recursive
split_recursively(text)
Cut at the strongest separator the author typed — paragraph, then line, then sentence. Uneven, because documents are. Respects structure, never reads meaning.
Semantic
chunk_semantically(text, embedder)
Embed every sentence; cut where similarity between neighbours drops below a percentile. The only strategy that reads the text. Costs one embedding pass, thrown away: 0.78s over 117k characters.
similarity between neighbouring sentences — a dip is a subject change
03 — Searching
Keyword search never looks at the vector.
The question becomes a vector and stays text. Which one the search uses depends entirely on the mode. and kept as text. Which of those two the search uses depends entirely on the mode.
Vector
uses: the vector only
Finds chunks whose meaning sits near the question’s. Works when asker and document use different words. Fails when the answer hinges on an exact token.
BM25
uses: the text only
Scores chunks by which query terms they contain, how rare each is, how long the chunk is. The vector is computed, then ignored. Fails on synonyms.
Hybrid
uses: both
Runs both and merges the lists. Weaviate fuses by Relative Score Fusion since v1.24. Hybrid was never last on either question set.
A worked failure
recursive/bm25 is the only one of the nine that misses “What is the
default value of initialDelaySeconds?” Four of its five results come from Service type,
the fifth from Pod phase.
BM25 adds a score for every matching term. Count them:
| chunk | what | default | value | initialDelaySeconds | terms matched |
|---|---|---|---|---|---|
| the one it returned | 1 | 1 | 1 | 0 | 3 |
| the correct one | 0 | 0 | 0 | 1 | 1 |
Three moderate terms beat one rare term.
is and of are stopwords in Weaviate’s en preset. what is not. Adding
what and value fixes this one question and lifts recursive/bm25 by 0.045 at
k=5.
It was tested and reverted. The setting was chosen after seeing which question failed, so the number measured the tuning, not the retrieval.
04 — Measuring
A similarity score is not a measure of correctness.
Every search returns a score, and treating a high one as a right answer is how a retrieval system quietly stops working while its dashboard stays green.
The measure is Recall@K: of the sections a human marked correct, how many appeared in the top K. 38 questions carry labels.
Three things deliberately not measured
- Similarity scores. A short chunk scores higher on cosine than a long one covering the same material, and BM25 scores are on another scale entirely. Rank is comparable; score is not.
- Duplicate sections. Five chunks from one correct section count once. Otherwise recall passes 1.0 and chopping finely looks like quality.
-
Unanswerable questions. Six of the 38 have no answer in the corpus. Recall
is undefined there, so the code raises rather than returning
0.0— a zero averages in as a failure that never happened.
Labels name sections, not chunks, because chunk ids change with every chunker. That choice has a cost, stated below.
05 — The reversal
The second question set reversed the conclusion.
22 labelled questions said keyword search wins: every BM25 and hybrid configuration beat every vector-only one. That was written down as a property of the system.
Ten fresh questions, against sections the first file never covered, committed before being run, scored once. Same collections, same code, same metric. committed to git before being run, and scored once. Same collections, same code, same metric.
BM25 went from first on every chunker to last on every chunker.
Seven of the ten held-out questions are solved by all nine configurations. The
reversal rests on three, and only one is a pure vocabulary gap: “expose
a service on an IP address I already own outside the cluster” never
says externalIPs, so BM25 returns Service type three times and never reaches the
right section.
The other two are subtler. Asked “how many kinds of probe does the
kubelet run”, BM25 matches probe fine — and ranks Define a TCP liveness
probe above Container probes, because a page about one probe uses the word
more densely than the page listing them all. Not blind; confidently precise about
the wrong page.
What both runs support, and neither supports alone:
Whether BM25 or vector wins is decided by whether the asker already knows the right word. Hybrid was never last on either set.
What this is not
Ten questions is small, and three carry the reversal. This is not proof vector search is better. It is proof the first conclusion outran its evidence.
A limit of this metric
Labels name sections, not chunks. Two configurations score full marks on the
initialDelaySeconds question while returning chunks that never
contain that string. Right neighbourhood, wrong chunk — and the metric
cannot tell the difference.
A measurement trap found here
Weaviate’s BM25 returns a lower-scoring top result at limit 1 than at limit 5. Scoring recall@1 that way inflated two BM25 configurations by up to 0.091. These figures retrieve five and truncate.
06 — The path
Everything an AI engineer has to understand, one explainer at a time.
Ten explainers, in the order the material has to be learned. Retrieval is finished; the rest land over the coming months.
Each is built the same way: implement it, measure it, publish what the measurement said. Nothing gets taught here that has not been built first.
-
live now
01Retrieval
Chunking, embeddings, BM25, hybrid search, Recall@K — and a held-out set that reversed the conclusion.
9 configurations · 38 labelled questions
-
coming soon
02Fusion
Two ranked lists, one answer. RRF ranks by position; Relative Score Fusion normalises and adds. Weaviate has used the second since v1.24. The first has never been measured here.
A tenth configuration, same 38 questions
-
coming soon
03Why models make things up
A model optimises for plausible, not true. There is no fact-checking step anywhere in it, so a confident right answer and a confident wrong one look identical from the inside.
The reason retrieval exists at all
-
coming soon
04Temperature and sampling
Every logit is divided by the temperature before softmax, stretching or squashing the gaps. Move the slider, watch the distribution sharpen.
Low temperature on a wrong answer is reliably wrong
-
coming soon
05Generation and grounding
The other half of RAG, and four separate measures. Context relevance grades retrieval; grounding grades generation; correctness checks the world.
An answer can be right and ungrounded — right for the wrong reason
-
coming soon
06Transformers, from 200 lines
Attention is where a token looks at the tokens before it. Query is what it wants, key what it holds, value what it offers. The MLP is where it thinks.
Built up one component at a time, bigram table to multi-head
-
coming soon
07Evaluation harnesses
How to score a system whose output is a sentence. Held-out sets, LLM judges, and why tuning on your test set measures the tuning.
The skill this whole site is an argument for
-
coming soon
08Agents and multi-agent systems
Tool calling and agent loops. When several agents share a task: who decides, who holds state, how a loop is stopped. LangGraph, LangChain and MCP.
Last for a reason — hardest to measure, easiest to fake
-
coming soon
09Production
Latency budgets, what to cache and what never to, cost per question. Embedding, search and generation each fail in a different shape.
The second-round question: it works, now it costs money
-
coming soon
10When not to use RAG
Prompting, fine-tuning, a SQL query and a lookup table each beat retrieval somewhere. Saying a problem does not need a vector database is the judgement the rest is for.
Recognising the problem is not the one you have a tool for
Check back — this list is where each new explainer appears as it is finished.