Retrieval · Documents
Finding the right part of Kenyan tax law before the model answers
An assistant that answers Kenyan tax questions from KRA guides and the tax Acts, with a citation for every answer. So far the work is on search: from 70% to 88% of questions finding the right section in the top five.
- Python
- PostgreSQL
- pgvector
- BM25
- nomic-embed-text
- Reranker
- Gemini
Retrieval · DocumentsKenya tax
- keyword search70%
- embeddings68%
- both together78%
- + section lookup, reranker88%
- right section in the top 5 (35 of 40)
- 88%right section in the top 5 (35 of 40)
- test questions, written before any code
- 50test questions, written before any code
- of them have no answer in the documents
- 10of them have no answer in the documents
- per question on a CPU, everything on
- ~7 sper question on a CPU, everything on
Starting with the questions
Before building anything I wrote 50 questions by hand, each with the document and section that answers it. If you write them after, you end up writing questions your system already handles.
I mixed them on purpose: different words from the document ("salary" against "employment income"), an exact section number, an answer inside a table, an answer split across two sections, a rule that changed between two Finance Acts, and 10 questions the documents don't answer at all.
The project has one rule: every step has one number, and the next step doesn't start until that number is written down.
Results so far
For the 40 answerable questions: is the right section in the top five results?
- Keyword (BM25)70%
- Embeddings68%
- Both, merged78%
- + section lookup82%
- + reranker88%
The reranker on its own added nothing (31 of 40). Together with section lookup it took the last step up.
Each step sends the model about 1,000 words, so the gain doesn't come from just sending more text.
What I noticed
- Keyword search and embeddings miss different questions. That's why merging them (reciprocal rank fusion) helps when neither is better alone.
- Questions with an exact section number and answers inside tables were the weakest for keyword search: 4 of 8 each.
- Two of my own answer keys were wrong. Fixing them moved the baseline from 26 to 28. Check the ruler before the thing you're measuring.
- Five questions are still missed, and they're listed by number, ready for the next round.
What's next
| Step | The number |
|---|---|
| Answers with citations, refusals when the sources are weak | correct refusals out of 10, against wrong ones |
| Memory across sessions, with corrections | at turn 30, does it remember the old fact or the new one? |
| Memory and search sharing one token budget | top 5 with memory on and off |
| Its own MCP server | it works from Claude Code |
The code and the full notes are on GitHub ↗
MSODOKI