Intent routing · Banking
Understanding what a bank customer wants, in English, Kiswahili or Sheng
An engine that reads a customer's message and decides what to do with it: answer, ask them to confirm, say no, or pass it to an LLM when it isn't sure. It runs on a laptop CPU, and money never moves without the customer saying yes.
- Python
- PyTorch
- EmbeddingGemma
- scikit-learn
- FastAPI
- PostgreSQL
- React
- Docker
Intent routing · BankingNia
- salio langu ni ngapianswer
- tuma 2k kwa KCB yanguconfirm
- nimeibiwa pesa na mtu wa simudecline
- ni swali, hii process ikojeask the LLM
- first guess right, on 298 unseen messages
- 86.6%first guess right, on 298 unseen messages
- critical errors on money or fraud
- 0critical errors on money or fraud
- right when it answered on its own
- 100%right when it answered on its own
- per message, on a CPU
- ~104 msper message, on a CPU
Why I built it
People in Kenya don't write to their bank the way chatbots expect. They mix English, Kiswahili and Sheng in one sentence ("nicheki balance ya savings", "doh imebaki aje"), shorten everything ("bal?", "lst 5 txns") and mention other banks and M-Pesa all the time.
I wanted to see how far a small open model on a normal CPU could get before an LLM is needed. Sending every message to an LLM is slow and costs money each time. For a bank it also means customer messages leave the building. Nia (Swahili for intention) is that experiment.
How the engine works
Every message goes through the same five steps, and each one has a single job.
| Step | What it does |
|---|---|
| 1. Embed | EmbeddingGemma, running locally, turns the message into a 768-number vector. |
| 2. Score | A logistic regression classifier gives a probability for each of the 10 intents. |
| 3. Rules | Facts a model can't know. Naming another bank or M-Pesa means an external transfer; a statement for someone else is refused. |
| 4. Check | The top intent must clear a threshold and be clearly ahead of the runner-up. A money intent phrased as a question isn't acted on. |
| 5. Route | Answer directly, ask to confirm, decline, or hand over to the LLM. |
The kind of intent decides the route. Looking at a balance or a statement is answered directly. Anything that moves money (a transfer, Lipa na M-Pesa) always goes to confirm, however sure the router is.
The router never does anything itself. It returns a decision with its top guesses, the closest training examples and its reasons, and the app shows them under "Why this answer?".
Results
The test set is 298 messages, about 100 per language, written by a different model than the training data and never used for tuning. A critical error is a message the router handled on its own with the wrong intent, where money or fraud was involved.
| EmbeddingGemma + | First guess | Answered alone | Right when alone | Critical |
|---|---|---|---|---|
| nearest examples | 76.5% | 13.4% | 97.5% | 1 |
| logistic regression | 83.6% | 24.5% | 98.6% | 1 |
| logistic regression + rules | 86.6% | 27.9% | 100% | 0 |
| small neural net + rules | 79.5% | 52.7% | 93.6% | 7 |
- English91%
- Sheng92%
- Kiswahili77%
Kiswahili is the weakest of the three, and the next thing to work on.
What the experiments showed
- The task prefix matters. Without "task: classification | query:" in front, "what's my balance" came out closest to "send 5000 to my KCB account".
- Embeddings measure topic, not intent. "How do I send money to KCB" and "send 5000 to KCB" scored 0.87 on both EmbeddingGemma and Gemini embeddings. One is a question and one is an instruction to move money, which is why the question check exists.
- Some distinctions are facts, not language. Whether "my Co-op account" is internal depends on which bank we are. A config file and one rule took internal transfers from 16 of 30 to 28 of 30.
- A trained classifier beats nearest-example scoring, and the comparison is one-sided: in paired counts, the rules fixed 9 messages and broke none.
- Most messages sent to the LLM already had the right top intent. What's left is confidence, not knowledge.
Speed and fine-tuning
It takes about 104 ms per message on a CPU, and 135 ms for the slowest 1%. Almost all of that is the embedding model, so the rules and checks are effectively free.
I also fine-tuned EmbeddingGemma to raise its confidence. A full fine-tune scored higher, but made 2 critical errors. For a bank that's the wrong trade, so I kept the version that made none.
Keeping the score honest
- Training data and test sets never overlap. A check fails the run if any sentence appears in two files, ignoring case and spacing.
- Nothing is tuned on a test set. Thresholds are chosen on held-out training scores, then applied to the test set once.
- Results are reported as paired counts (A right and B wrong, against B right and A wrong), because with 300 sentences a few points can be noise.
- Every result is stamped with the hash of the training file that produced it.
The bank around it
To test the engine somewhere close to real, I built Benki Yetu, a made-up bank, in FastAPI, PostgreSQL and React, all in Docker. You log in, check balances and statements, and chat with Benki AI.
A transfer becomes a pending action that expires after five minutes. The confirm endpoint is the only place money moves, inside one database transaction, after checking ownership, expiry, required fields and balance. Every router decision goes to an audit table. Everything in it is fake: the bank, the users, the money.
What's next
The LLM fallback, with retrieved examples and structured answers, and no way for it to skip confirmation. Then an end-to-end test of accuracy, how often the LLM is called, and latency. And a test set written by native Kiswahili and Sheng speakers, which is the number I'd trust most.
The code and the full notes are on GitHub ↗
MSODOKI