"What's in my biography" failed to find my biography file. "Show me my biography" worked. Building Ragbot, my open source personal retrieval-augmented generation (RAG) system, with Claude Code turned that frustrating discrepancy into a practical question: at which stage did the request stop meaning what I intended?
A retrieval system can rewrite a query, classify it, embed it, rank candidate passages and trim them to a context budget before a model sees any evidence. A disappointing answer gives me a reason to inspect that chain. It does not tell me which step went wrong.
September 2026 update: The source comparisons below include Ragbot’s later bounded document scan. The downloadable workbook was added with this update.
Keep the query constant while testing a cause
My original diagnosis blamed the apostrophe. But the successful query also changed “what's in” to “show me.” That comparison changed the wording and the kind of request together. It could not isolate punctuation as the cause.
I directed Claude to add normalization:
query = query.replace("what's", "what is")
Expanding common contractions, including “can't,” “won't” and “it's,” eliminated a class of failures I was seeing. The one-line example is the idea behind the change; the current implementation lowercases the query and uses a dictionary with word boundaries.
The string "what's" contains the substring "what". And a contraction need not be one tokenizer token. A local check of the pinned all-MiniLM-L6-v2 tokenizer, with padding and special tokens excluded, gives these results alongside the selected Ragbot preprocessing functions:
| Input | Token pieces | Normalized query | Document-lookup hint |
|---|---|---|---|
what's in my biography | what, ', s, in, my, biography | what is in my biography | biography |
what is in my biography | what, is, in, my, biography | what is in my biography | biography |
what’s in my biography | what, ’, s, in, my, biography | unchanged | none |
The last row uses a curly apostrophe. In the September implementation, contraction expansion handles the straight form, and document classification examines the original query. Those details explain the different local routing results in this table.
Token pieces also do not tell us whether two embeddings will retrieve the same candidates. That would require running the relevant embedding model against a controlled corpus and inspecting its scores. The exercise here stops before that step.
Try an explainable route before adding another model call
I initially assumed that classifying a query as “document lookup” or “information synthesis” would require an LLM. I asked Claude what alternatives might work. It suggested pattern matching. These were the three expressions I described in the original article:
DOCUMENT_LOOKUP_PATTERNS = [
r"^show\s+(?:me\s+)?(?:my\s+|the\s+)?(.+)$",
r"^what(?:'s| is)\s+in\s+(?:my\s+|the\s+)?(.+)$",
r"^(?:get|fetch|retrieve)\s+(?:my\s+|the\s+)?(.+)$",
]
These expressions provide a deterministic local rule without another classification API call. The September implementation adds patterns and optional model-based query planning.
A match is only a proposed route. “Show me how authentication works” matches the first expression and produces the hint how authentication works. In the exercise, the declared reader intent is to obtain an explanation. Treating that hint as a confirmed document identity would send the request down the wrong path.
I would check the hint against the available documents before returning a file. An exact, unique match can proceed. Multiple matches can ask for clarification. A missing match needs an explicit fallback or an honest not-found result. A regex non-match is not the only occasion on which the system needs another plan.
Start with an explainable rule for the requests you can define; use observed failures to decide where another method earns its cost.
Give whole-document requests their own path
“How do I configure authentication?” may need passages from several documents. “Show me my biography” asks for a particular object. Returning three relevant fragments can be reasonable for the first request and deeply irritating for the second.
I asked Claude for options, and it proposed two paths:
- Information synthesis: retrieve relevant chunks and use them to answer a question across sources.
- Document lookup: identify the requested document and retrieve its full content when it fits the selected budget.
I learned the need for that second path through the frustration of using my own application. I wanted to see my biography, not persuade the system to assemble an approximation from fragments.
The September Ragbot implementation reconstructs a candidate document from indexed chunks, ordered by their positions; it does not simply open the original file. Its chunk scan is bounded, and it checks an approximate context allowance before returning the reconstructed text. “Full document” therefore needs a check on both identity and completeness. A plausible filename and a long answer are insufficient.
The downloadable retrieval workbook uses two fictional documents, each exactly 54 characters long. One is a biography; the other contains authentication advice. The expected biography includes its document ID and both lines of text. Substituting the other document fails even though the character count stays correct. Substituting only its content under the correct ID also fails.
The workbook separately models ambiguity, a missing document and an over-budget document. Its budget is explicitly in characters so that the example cannot be mistaken for a model's token accounting. It reports the over-budget condition instead of silently presenting a shortened result as the complete file.
Increase the allowance when useful evidence is being cut
Claude initially implemented a 2K-token allowance for retrieved context. I asked it to increase the allowance to 16K. In my use, the results improved immediately:
- More complete document coverage.
- Less information loss from truncation.
- Answers that felt more comprehensive.
The cost increase was negligible for my personal workload. An eightfold allowance is not necessarily eight times the actual input, and the retrieval allowance is only one part of a request that can also include instructions, conversation history and output capacity.
In a quality-first application, give the model room for the evidence it needs. A small default can discard useful material long before the model reaches its capacity. Compare the evidence retained, actual token use, latency and answer quality for your model and workload. Extra room helps when relevant material was being cut; filling it with irrelevant passages leaves the retrieval problem unsolved.
Make ranking changes visible in a candidate table
Semantic similarity can find a related passage in the wrong document. When I asked for my biography, I wanted the biography file to receive credit for matching that explicit name. I asked Claude to add ranking boosts:
- Filename match: +0.5 per matching term.
- Title match: +0.3 per matching term.
Claude’s initial values, 0.2 for filename matches and 0.15 for title matches, had left semantically similar but wrong documents ahead of exact name matches in my use.
The December 15 change records increases from 0.3 and 0.2, respectively. The motivation was to move the intended file above semantically related but wrong results. The size of a useful boost depends on the underlying score distribution and the queries being served.
The workbook's base scores are supplied numbers, not measured embeddings. The authentication document's title mentions “Biography,” giving it a title match for this query.
| Candidate | Base score | Filename matches | Title matches | New score |
|---|---|---|---|---|
| Biography | 0.35 | 1 | 0 | 0.85 |
| Authentication | 0.50 | 0 | 1 | 0.80 |
The biography score is 0.35 + 0.5 = 0.85; the other score is 0.50 + 0.3 = 0.80. With the committed predecessor weights, they would be 0.65 and 0.70, reversing the order. Exchanging the new filename and title boosts produces 0.65 and 1.00, which the fixed expectations reject.
Keep queries where a name match should win and queries where it should not, then inspect which documents move. The same reasoning can guide experiments with section headings or other metadata, without assuming every exact word match deserves the largest boost.
Follow a default through all five locations
After Claude changed the core retrieval allowance to 16K, the web UI still showed 2K and returned truncated results. I found that the default existed in five places. Two had been updated and three had been missed:
| Location | Initial change |
|---|---|
| Core retrieval function | Updated |
| Chat function | Updated |
| Streaming chat kwargs | Missed |
| Pydantic request model | Missed |
| React state initialization | Missed |
The December 15 commit records all five defaults changing from 2000 to 16000.
A useful read-only locator was:
grep -r "2000" --include="*.py" --include="*.tsx" | grep -i "rag\|token\|context"
A search result identifies places to inspect. It does not establish which value reaches a request. Follow the path from the React setting into the API payload, through the request model and the chat or streaming function, and finally into the retrieval function's max_tokens argument. Check both an omitted setting and an explicit override. A default in one layer may never take effect because an earlier layer always supplies a value.
The workbook reads five exact source windows and checks that each records 16000. A separate, constructed propagation trace checks the default and an explicit value of 4096. Changing any one of its five defaults back to 2000 fails a named check; losing the override in the streaming branch also fails. These local checks do not execute the web UI or API. In the application, exercise those actual boundaries too, because a unit test that starts inside the retrieval function cannot show what the browser sent.
Run the controlled example
The download includes the selected Ragbot preprocessing functions, the pinned tokenizer, the two fictional documents, the literal expected results and the fault controls. It requires Python 3.12 and tokenizers==0.22.2. From the extracted folder:
python3 -B check.py
python3 -B controls.py ../retrieval-control-results
Choose a new output directory for the second command. It keeps the deliberately wrong copies and their results for inspection. The supplied package passes 28 checks, and all nine wrong copies fail the unchanged expectations with their intended errors. Both commands were also run from a clean extraction of the ZIP.
The example runs locally without model weights, an embedding service or a retrieval database. Its query inspection uses selected source functions; its document resolver, scores and configuration propagation are teaching fixtures. Use them to develop checks for your own retrieval system.
What I would build earlier
I would give explicit document requests their own path from the beginning, add query preprocessing earlier and bring real user queries into testing sooner. The biography problem emerged from using the system. Controlled variants then make it possible to ask a narrower question about what failed.
I had inherited an information-synthesis-only assumption from tutorials. Using the application made its limitation obvious: when I ask for a document, I want that document. I would also start with more room for useful evidence instead of carrying forward an unnecessarily tight context budget.
For the next retrieval change, I would keep three artifacts beside the code: a set of real query variants, expected document identities and content, and a trace of the effective configuration. Contractions, typos and synonyms belong in that query set. An explainable rule belongs where its behavior is useful and its fallback is clear. A larger context allowance belongs where it retains evidence the answer needs.
When my biography disappears from the answer, those records let me ask whether the system misread the request, chose the wrong file or cut away the right one.
This article is part of the synthesis coding series. For related content on AI pipelines, see Data Format Contracts for AI Pipelines.