Bilingual product search and recommendation
Product search for Japanese and English shoppers on Amazon's public Shopping Queries dataset (ESCI), plus next-purchase recommendation on real e-commerce transactions, served by a small FastAPI app on a laptop CPU. Every number is measured on held-out data, with a 95% confidence interval, separately for each language.
Each language has 1,000 test queries from ESCI's official test split, searched against 50,000 products: every judged product for those queries plus look-alike distractors. NDCG@10 rewards putting exactly matching products at the top.
| Japanese | English | |
|---|---|---|
| BM25, whitespace tokens | 0.280 | 0.422 |
| BM25, MeCab / word tokens | 0.492 | 0.455 |
| Multilingual E5-small (dense) | 0.495 | 0.500 |
| Hybrid, reciprocal rank fusion | 0.534 | 0.507 |
| Hybrid, weight tuned on train queries | 0.548 | 0.532 |
| Hybrid + cross-encoder rerank (top 20) | 0.567 | 0.560 |
| Hybrid, E5 fine-tuned on ESCI | 0.582 | 0.559 |
| Hybrid + reranker, both fine-tuned on ESCI | 0.595 | 0.576 |
NDCG@10 with ESCI gains (Exact 1, Substitute 0.1, Complement 0.01). 95% bootstrap intervals are about ±0.02 for every cell.
- For Japanese, tokenization matters more than the model. Japanese has no spaces, so an English-style analyzer treats a whole query as one word. MeCab segmentation alone takes BM25 from 0.280 to 0.492, a bigger jump than any model change after it.
- Sparse and dense search fail on different queries. On Japanese, BM25 and E5 tie (0.492 vs 0.495), yet combining them reaches 0.548 (p < 0.001). The weight was tuned on separate training queries, never on the test set.
- 7% of Japanese-site queries contain no Japanese. Model numbers and English words are where exact matching helps: on those 72 queries E5 scores 0.40 against BM25's 0.47, and the hybrid covers both.
- An off-the-shelf reranker is not a free upgrade. It adds +0.02 to +0.03 but takes 1.2 s per query on CPU, and on ESCI's own ranking task it is no better than E5 alone, because it was trained on web passages, not products. Fine-tuned on ESCI, it does beat E5 (next point).
- Fine-tuning on ESCI's own labels is the biggest gain after the hybrid. Both models trained on a free Colab GPU in about 40 minutes: hybrid + rerank goes from 0.567 to 0.595 (Japanese) and 0.560 to 0.576 (English), all p ≤ 0.002, with test queries kept out of training.
- An open LLM is not a free replacement for human labels. Qwen2.5-7B on a free Colab GPU agrees with ESCI's human labels at Cohen's κ 0.16–0.21, no better than always guessing the most common label. A reranker trained on its labels is worse than one trained on the human labels of the same 40k pairs (English 0.545 vs 0.557, p < 0.001). In about half the disagreements I checked, the human label looked wrong.
Online Retail II: 803k purchases from a UK online retailer. Models train on data before September 2011 and predict what each of 2,336 customers buys from September to December. The strongest baseline is "buy it again": 38% of next-quarter purchases are repeats, and simply re-recommending a customer's own items beats every learned model. The learned models earn their place on new items, where EASE is 1.7× better than popularity.
| All purchases | New items only | |
|---|---|---|
| Popularity | 0.107 | 0.044 |
| Popularity, last 30 days | 0.109 | 0.055 |
| Item kNN (customers also bought) | 0.209 | 0.083 |
| ALS (matrix factorisation) | 0.247 | 0.082 |
| EASE | 0.260 | 0.093 |
| Re-recommend own past purchases | 0.336 | — |
NDCG@20. Hyperparameters tuned on June–August 2011, never on the test months.
One FastAPI app serves both catalogues and the recommender. The language is detected from the query, so a Japanese query searches the Japanese catalogue. Almost all hybrid time is encoding the query; BM25 and the 50k-vector dot product take under 2 ms. Each language's index is about 50 MB.
| Median | p95 | |
|---|---|---|
| BM25 | 3.5 ms | 4.7 ms |
| Dense (E5) | 32 ms | 43 ms |
| Hybrid | 32 ms | 40 ms |
| Hybrid + rerank | 1,171 ms | 1,243 ms |
| Recommendation (EASE) | 3.3 ms | 4.5 ms |
200 held-out queries per language, one at a time over HTTP, on a 2-core laptop CPU.
- ESCI labels about 28 products per query; relevant products nobody labelled count as wrong, so absolute scores are lower bounds.
- The corpus is 50k products per language, not a full catalogue. Each fine-tuning and LLM result is a single run with fixed settings.
- One UK retailer with many wholesale buyers makes "buy it again" unusually strong; a marketplace would balance re-buys and discovery differently.