Search · Evaluation
Does a bigger embedding model help a small shop's search?
I built a fake Kenyan electronics shop and asked 11 search setups the same 71 questions in English, Swahili and Sheng, to see if Google's EmbeddingGemma 2 is worth it over what Typesense already gives you.
- Python
- Typesense
- EmbeddingGemma 2
- MiniLM
- E5
- FastAPI
- React
- Docker
Search · EvaluationShop search
- words only59%
- small built-in model90%
- EmbeddingGemma92%
- speed cost25× slower
- right product first, best setup
- 92%right product first, best setup
- search setups compared
- 11search setups compared
- questions in three languages
- 71questions in three languages
- slower for one extra right answer
- 25×slower for one extra right answer
The question
Google released EmbeddingGemma 2, and I wanted to know if it's worth using for product search when Typesense already has small models built in. So I made a shop with 200 products, priced in KES, and wrote 71 questions.
The questions are the kind people actually type: exact product names, look-alikes (iPhone 15 against 15 Pro against an iPhone 15 case), vague needs ("wifi goes off whenever KPLC cuts power"), typos, Swahili and Sheng, and things the shop doesn't sell.
Results
- Typesense, words only59%
- Typesense + MiniLM90%
- Typesense + multilingual E582%
- Typesense + EmbeddingGemma92%
- EmbeddingGemma alone89%
AI setups shown with Typesense trusting the AI 70%. Laptop CPU, no graphics card.
Speed, memory and Swahili
| Setup | Swahili / Sheng | Time per search | Memory |
|---|---|---|---|
| Typesense, words only | 1 of 8 | 0.001 s | 7 MB |
| Typesense + MiniLM | 3 of 8 | 0.007 s | ~190 MB |
| Typesense + multilingual E5 | 4 of 8 | 0.008 s | ~590 MB |
| Typesense + EmbeddingGemma | 5 of 8 | 0.17 s | ~970 MB |
| EmbeddingGemma alone | 5 of 8 | 0.16 s | ~970 MB |
What I found
- One Typesense setting mattered more than the model. By default it trusts word matching 70% and the AI 30%, so turning on a model barely helps: "how much is the iphone 15" returns an iPhone 15 wallet case. Flip it to trust the AI 70% and every model jumps to 82–92%.
- EmbeddingGemma was best, but only just. 92% against 90% is one question out of 71, and the small model is about 25 times faster with a fifth of the memory. For most shops, I'd pick the small one.
- Swahili is still unreliable everywhere. "Chaja ya simu ya gari" (a car phone charger) got a drone from the small model, and "simu rahisi ya M-Pesa kwa bibi yangu" (a simple M-Pesa phone for my grandma) got a MiFi from EmbeddingGemma.
- "We don't sell that" is hard. Word search always returns something: "book a hotel in Mombasa" gives you a Kindle. The AI models say no more often, but still get fooled: "men's running shoes" returns running headphones.
- EmbeddingGemma is a 1.5 GB download and took 33 seconds to prepare 200 products. At 10,000 products you'd want a GPU.
Try it
The repo has the shop and a compare page: ask one question and see what every setup returns, side by side, with the right products ticked. Every answer from every setup is saved in the results, so the numbers can be checked.
The code and the full notes are on GitHub ↗
MSODOKI