Every shopping assistant demo looks impressive. The fluent paragraph, the confident recommendation, the natural follow-up question. After two years of building these systems, I watch a different part of the demo, because the fluency is the cheap part. Whether the right products sit behind the fluency is decided in a layer the demo never shows: retrieval.
An assistant is a voice attached to a shelf-fetching mechanism. If the mechanism brings back the wrong products, the language model will describe the wrong products beautifully. Most of the real quality differences between assistants, and most of the real failures of store search, live in that mechanism.
The mechanism of the failure
Run one query through the two layers and the whole problem is visible:
Shopper types: cream for redness after retinol
Keyword search: 0 results
The catalog, meanwhile: “Barrier repair moisturizer. Ceramides, panthenol. Fragrance free.”
The store stocks exactly the right cream. Not one word is shared between the question and the product, so the shopper who described their need perfectly is told, in effect, that the store cannot help.
Keyword search checks whether the shopper’s words occur in your product text. When they do, it is excellent: exact names, part numbers, instant. The failure begins where the words diverge, and the divergence is a measured property of language, not an edge case. Furnas, Landauer, Gomez, and Dumais showed in the vocabulary problem that two people name the same thing with the same word less than 20 percent of the time, and that single-vocabulary interfaces fail 80 to 90 percent of first attempts.
Shopping makes the problem worse, because customers don’t even describe products. They describe situations: “for muddy forest trails,” “gift for a dad who cycles.” No product title will ever contain those words, and the shoppers typing them are the ones who most needed help.
This failure has been benchmarked at scale. Baymard’s audits of e-commerce search break shopper queries into types and find sites failing exactly on the situational ones: 43 percent of sites mishandle use-case queries (“wedding gift”), 44 percent mishandle compatibility queries, 54 percent abbreviations, with 56 percent failing to adequately support search overall. And Amazon’s own research group published the ESCI Shopping Queries dataset, 130,000 hard queries with 2.6 million human relevance judgments, because query-to-product matching “still has a large room for improvement” at the largest retailer on earth. Retrieval is the open problem, not a solved substrate the AI sits on.
The existing answers
The industry has three generations of response.
Manual patching: synonym lists, typo tolerance, curated redirects. This works, one gap at a time, forever. Teams that maintain great synonym lists are doing real, valuable work with no end state.
Semantic (vector) search: embed product text and queries into a space where distance means relatedness, so “cream for redness” lands near “barrier repair moisturizer” with zero shared words. This is the technique behind most of what gets marketed as AI search, and it dissolves the vocabulary problem rather than patching it.
And now, LLM-based assistants layered on either. The model interprets the conversation, extracts what matters (“under 30, fragrance free, for sensitive skin”), queries the retrieval layer, and writes the recommendation.
Why each only partially works
Manual patching cannot reach situational language. You can alias “vitamin E” to “tocopherol.” You cannot enumerate every “gift for a dad who cycles.”
Vector search has a marketing problem: it is sold as strictly better, and it is not. Ask a vector index for an “ACME X200 laptop” and it happily returns the X220 and the X180, because they are semantically close; the shopper wanted the letters, not the vibe. A hard constraint like “under 200 euros” is a filter, not a similarity. And the research agrees the superiority story is oversold: the BEIR benchmark, evaluating retrieval systems zero-shot across heterogeneous tasks, found the boring keyword baseline BM25 “a robust baseline” that dense retrieval models often failed to beat out of domain. Embeddings also inherit your text: a product described as “Bestselling serum!” occupies a meaningless point in the space. Semantic search on thin data is a precise map of nothing.
LLMs on top fix none of this and can mask all of it. The model cannot recommend what retrieval did not return, but it will fluently rationalize whatever was returned. This is the demo’s trick: fluency reads as competence, and the two are uncorrelated. A weak retriever behind a strong model produces confident, wrong, persuasive answers, which is strictly worse than a search box returning nothing.
The better model: a pipeline with a ceiling
The frame I use: discovery is a pipeline, and retrieval sets its ceiling. The language model decides how close to the ceiling you get; only the retrieval layer decides where the ceiling is.
In practice that means hybrid systems, each mechanism where it wins: exact matching for names and part numbers, filters for hard constraints like price and stock, vector similarity for situational language, and the LLM as interpreter and presenter rather than as the knowledge. This is how we built Probedots: embedding search runs inside Postgres next to the catalog rows, filters stay filters, and a merchant who has spent years tuning an Algolia index can plug that in as the retrieval layer instead, because tuned retrieval is the asset worth keeping.
The same frame tells you where the improvement budget goes. Not to a bigger model: to the shelf.
Practical implications
The consequence that matters most for a merchant is a mundane one: your product text is now your retrieval surface. Embeddings are built from your descriptions, so the words you wrote decide which questions your products can answer.
Describe use, not just labels: “compresses to bottle size, warm enough for summit breaks” places a jacket in the path of a dozen situational queries a spec table never touches. Say who each product is for. Include your customers’ words, unglamorous ones included, next to your official vocabulary. Start with your ten best sellers rather than the whole catalog.
And evaluate assistants at the shelf, not at the voice. Ask a candidate assistant something vague and check which products came back before admiring how the answer reads. Ask what happens when nothing relevant exists; the correct behavior is an honest empty result, and an assistant that invents around it is spending trust it cannot repay.
The shift
The industry talks about AI shopping assistants as a language breakthrough, and the language is real. But the language was never the bottleneck. Stores have been fluent for a century; catalogs, copywriters, and salespeople all speak. What stores could not do is reliably connect a customer’s situation to the right three products out of thousands. That is a retrieval problem, it was a retrieval problem before LLMs existed, and the assistants that feel intelligent are the ones that solved it underneath the conversation. Judge the shelf, not the voice.