Improving an assistant often feels like putting a finger in the dike. You patch one issue with a prompt adjustment, eyeball the fix for a given conversation, and without you knowing two more leaks are appearing somewhere else.
My goal with Probedots was to create a system allowing shop owners to ship new instructions to their shopping assistant without harming existing conversations.
Why eyeballing fails
Reading a few conversations cannot tell you whether the dike is holding, for three reasons.
Memory keeps the wrong sample. You remember the standout conversation and the disaster; the five hundred ordinary conversations decide revenue, and nobody reviews five hundred by eye.
Every version sounds good. Change the instructions and the new wording reads better, partly because you wrote it, and because fluent text reads as competence either way. Opposite changes both pass this test: “be more concise” sounds obviously better, and so does “explain your reasoning more.” They cannot both be right for your shoppers, and intuition will argue either side with equal conviction.
And the intuition problem is general to product changes, not specific to AI. At Bing, an employee once proposed a small change to how ad headlines displayed. Experienced product people judged it low priority and shelved it for months, until an engineer ran it as an A/B test and it turned out to be worth over 100 million dollars a year, one of the best revenue ideas in Bing’s history. Kohavi and Thomke tell the story in their Harvard Business Review account of online experimentation. If experts misjudge ordinary UI ideas that badly, nobody can rank prompt edits by feel. An assistant’s instructions are among the most consequential text in a store; one edited paragraph changes every conversation. That reach deserves at least checkout-level discipline.
The existing approaches
Teams evaluate assistants in three ways, in ascending order of cost and validity.
Golden questions: a fixed list of hard inputs run against every new version in a playground. Cheap, fast, catches regressions on known failure modes.
Offline judgment: sample real transcripts, have humans or an LLM rate them against criteria.
Online experiments: split live traffic between versions and compare behavior.
All three are worth running. Each one lies to you in a specific way if it runs alone.
Where each one breaks
Golden questions test the first turn of a conversation against the questions you already thought of. Assistants fail in the middle of conversations, on turn three, when context has accumulated, and they fail on the questions you did not think of. A version that aces your list can still lose the plot mid-dialogue.
Offline judgment inherits the judge’s biases, and with LLM judges the biases are documented rather than hypothetical: Zheng et al.’s work on LLM-as-a-judge measured position bias, verbosity bias, and self-enhancement bias, alongside roughly 80 percent agreement with human raters. Eighty percent agreement is useful, and it is not ground truth. A machine judge will happily prefer the longer, more confident answer; shoppers often prefer the shorter one that ends in a purchase.
Online A/B tests are the closest to truth and the most expensive to reach it: real shoppers, real weeks, and a subtle trap in the unit of randomization. Split by conversation and a returning shopper meets the brief version on Monday and the talkative one on Wednesday; their sessions contaminate both arms, and you are now measuring confusion. The split has to be by shopper, deterministically, so the same visitor always lands in the same arm. We build this constraint into the platform (each store’s test locks every shopper to the control or the challenger agent for the whole run) because it is invisible when violated: the test still produces numbers, just meaningless ones.
The model that organizes it: replay, then split
The mental model I now use: offline evaluation exists to protect shoppers from bad candidates; online evaluation exists to measure good ones. The bridge between them is replaying real conversations, which is the piece that does the most work for us.
The idea: take audited past conversations, replay each one against the changed assistant, and compare outcomes. One replay, side by side:
Three weeks ago, real conversation:
Shopper: something for muddy trails, under 2,000
Assistant: listed four bikes with no reasons. Shopper clicked nothing and left after a second message.
Tonight, replayed against the challenger:
Shopper’s opening message: replayed verbatim, word for word.
Challenger: Is maintenance a concern, or mostly grip?
Simulated shopper, allowed to know only what the real one revealed: “grip mostly”
Challenger: recommends two bikes, a reason each.
Replay ends here. The real shopper gave two turns, so the replay gets two turns.
Judge: challenger handled the intent better. Or worse. Or: too little evidence, inconclusive.
The opening message is replayed verbatim because it defines the intent. Every turn after that cannot be pasted from the transcript, because the new assistant answers differently and the old shopper turns stop making sense against new responses; that is what the simulated shopper is for.
The cutoff in that transcript is the fidelity rule: the replay never uses more shopper turns than the real shopper gave. A shopper who left after one message is replayed for one message. Extending the dialogue past the evidence means inventing engagement the real shopper never showed, at which point the eval measures fiction. For the same reason, thin conversations get marked inconclusive rather than scored; a judge forced to rule on one turn of evidence is just adding noise with confidence. Roughly speaking: replay against your real past demand, never against the demand you wish you had.
Replay finds regressions before any live shopper meets them. What it cannot do is tell you a challenger sells more; simulated shoppers do not have wallets. That remains the A/B test’s job: one change per test, one primary metric chosen in writing before launch (we use clicks on recommended products, with answer ratings as a guardrail so a pushier variant cannot win on clicks while quality slides), run through at least one full week including the weekend, ended deliberately with the winner recorded.
Practical implications
If you run an assistant, in any tooling: keep a golden list and grow it from production failures. Get replay-based regression checks if you can, and respect the fidelity limit if you build them. Randomize online tests by shopper, never by conversation. Decide the metric before the test, or you will shop through the results for whichever number flatters the change. And treat a draw that simplifies your instructions as a win; shorter briefs are easier to keep true.
And distrust any vendor, us included, whose improvement story is adjectives. A claim you can use is a mechanism plus a measurement.
The shift
Teams ask “how good is our assistant?” as if goodness were a property of the current version. After two years of this, I think the better question is “how quickly do we find out when it gets worse?” A static assistant with excellent evaluation will beat a brilliant assistant with none within a quarter, because catalogs shift, shoppers shift, models shift, and only one of those teams notices. Evaluation is not the report card. It is the steering.