If you attended any of the major retail conferences in the first half of 2024 — NRF in January, Shoptalk in March, the various tech-vendor summits in between — you saw a version of the same demo. A shopper asks a conversational interface for "a navy blazer for a fall wedding under $300." The assistant returns three considered options, explains the tradeoffs, asks about fit preferences, suggests a tie. The crowd nods. Investors take notes.

Nine months in, the demos have mostly survived. The implementations have not — or at least, not in the form they were originally pitched.

The gap between demo and deployment

We spoke to product and e-commerce leads at eight retailers — a mix of mass, specialty, and mid-market — who launched or piloted an AI shopping assistant in 2024. The pattern across conversations is striking. Almost everyone reports that the version of the assistant currently live looks materially different from the version they launched.

The most common cuts:

• Open-ended natural language input got narrowed. Several retailers found that the long-tail of customer queries — vague, ambiguous, off-topic, or testing the system — was either expensive (token costs) or embarrassing (hallucinations). The current generation of live assistants tends to nudge users toward structured prompts or button-based flows much faster than the launch version did.
• The "personality" got toned down. A head of e-commerce at a mid-market specialty retailer told us their first version had a chatty, recommendation-forward voice. "Customers found it weird. We dialed it back to something closer to a smart search bar."
• Recommendations got more conservative. Early versions optimized for "interesting" suggestions. Several retailers found that interesting often meant low conversion and high returns. Live versions skew toward bestseller-adjacent recommendations with more guardrails.

What's actually moving numbers

The retailers reporting genuine commercial lift from AI assistants in 2024 are, almost without exception, talking about narrower use cases than the launch announcements implied.

The categories where operators report real impact:

Search relevance. Replacing or augmenting the legacy site-search stack with an LLM-powered retrieval layer is the use case most retailers are willing to put numbers behind — usually framed as a single-digit percent lift in search-led conversion. Not transformational, but real, and the ROI math works.

Customer service deflection. Pre- and post-purchase Q&A — sizing, returns policy, order status, fit guidance — is where the savings show up. The retailers furthest along are reporting deflection rates that materially change contact-center economics.

Product-detail-page enrichment. Generating consistent, on-brand attribute descriptions and "why this product" copy at scale. Less glamorous than a conversational concierge, but several retailers cited it as the AI investment with the clearest unit economics.

What is not moving numbers in any way operators are willing to share: the conversational discovery concierge, in its original form. The use case the demos were built around.

The cost conversation has changed

When we asked retailers in late 2023 about AI assistant economics, the answer was usually some version of "we'll figure that out." In September 2024, the answer is much more specific. Inference costs per session have come down meaningfully through the year — both because the underlying model prices have dropped and because retailers have gotten better at routing simple queries to cheaper models. But the operators we spoke to are still cautious about volume.

A digital VP at a mass retailer described their current posture: "We can afford this for the customers who are likely to convert. We cannot afford it for everyone who lands on the homepage." That has led to a quieter trend through 2024 — AI assistants gated behind sign-in, behind certain product categories, or behind specific points in the funnel rather than offered as a universal layer over the site.

The platforms-versus-build question

The other thing that has shifted is the make-versus-buy calculus. Twelve months ago, many of the retailers we spoke to were leaning toward building in-house, partly because the off-the-shelf options felt immature and partly because the strategic narrative favored "AI as differentiator." Today, most of those same retailers are using some combination of platform-provided capabilities — from their commerce vendor, their search vendor, or a horizontal AI provider — with thinner custom layers on top.

The honest summary from one operator we've been tracking since the start of the year: "We thought we were building a flagship feature. We ended up rebuilding our search bar. The search bar is better. I'm not sure that's the story we'd have told in January."