RankDude

How I test AI shopping assistants

Nine tests, the same for every assistant, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, with real products, real prices and real orders, and the timings, screenshots and transcripts behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

Each assistant gets the same twenty shopping tasks from a written list: ten named items with a size, colour or specification (a specific pair of running shoes in a size, a particular 65-inch television), five needs-based briefs (a gift under a budget, a laptop for video editing, a stroller that fits a small car boot) and five comparisons between two or three named products. Consumer assistants are tested as a shopper in their own app, extension or chat. Assistants sold to stores are installed on my own test shop, which carries the same 200-product catalogue for every tool, and are tested from the shopper's side with the same tasks adapted to that catalogue. Nothing is graded from a feature list, a case study or a demo video.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 9 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (nine when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. A research-only assistant that cannot add to a cart or track an order gets no grade on those tests rather than a 1, and the review says so.

The 9 tests

1.Finding the Right Product

Each task is typed once, in plain words, with its constraints (budget, size, delivery deadline). A task counts as solved when the assistant's first recommendation, or one it offers after a single clarifying question, meets every requirement and is in stock at the quoted store when I open the listing. A recommendation that breaks a constraint, is out of stock or does not exist counts as failed. The tally is the grade; the number of clarifying questions needed is reported alongside.

  • 5: 18 to 20 solved
  • 4: 15 to 17
  • 3: 11 to 14
  • 2: 6 to 10
  • 1: 5 or fewer

2.Product Facts & Prices

Across the twenty tasks I record the first three facts the assistant states about a recommended product: usually the price, a specification and stock or delivery. Each is checked on the retailer's page within five minutes, with a screenshot of both. A price is right when it matches the listing to the cent before shipping; a quoted sale price that has ended counts as wrong. Invented products, prices or reviews are counted separately and reported.

  • 5: 58 to 60 right, nothing invented
  • 4: 54 to 57 right, at most one invented item
  • 3: 45 to 53 right, or two or three invented items
  • 2: 30 to 44 right, or an invented product
  • 1: under 30 right

3.Comparisons

Each of the five comparison tasks is marked on five points: the specifications it quotes are correct, the current prices are correct, it names a trade-off rather than declaring a winner with no reason, it says where its information came from, and it answers a follow-up on one specific difference without contradicting itself. One point each, 25 in total.

  • 5: 23 to 25 points
  • 4: 19 to 22
  • 3: 14 to 18
  • 2: 8 to 13
  • 1: 7 or fewer

4.Cart & Checkout Actions

For five of the named-item tasks I ask the assistant to add the item to a cart and buy it. I check that the right size, colour and quantity landed in the cart, that the assistant showed the total including shipping and asked before paying, and that the order went through and appears in the store's order history. One order is then cancelled or returned through the assistant to see what happens. Orders are placed for real at the lowest-value item that fits the task and refunded or kept.

  • 5: all five correct, each confirmed before payment, order placed and visible
  • 4: four correct, all confirmed
  • 3: three correct, or one order placed without a confirmation step
  • 2: one or two correct
  • 1: none correct, or no cart actions at all

5.Speed

A screen recording runs through every task. The clock starts when I send the message and stops when the first recommendation with a name and a price is on screen, not when the assistant starts typing. The median of the twenty timings is the grade; the slowest answer is reported.

  • 5: median under 10 seconds
  • 4: 10 to 19 seconds
  • 3: 20 to 39 seconds
  • 2: 40 to 89 seconds
  • 1: 90 seconds or longer

6.Orders, Returns & Handoff

After the test orders I ask the assistant for the status of a specific order, for the return steps and deadline for that item, and to reach a person. Each answer is checked against the order page and the written returns policy. For the handoff I record whether a human replied and how long it took. An assistant that cannot see orders is told so in the review and is graded on the return policy and the handoff only.

  • 5: all three correct, human reached within 1 hour
  • 4: all three correct, human within 24 hours
  • 3: one answer wrong, or no human channel but correct answers
  • 2: two answers wrong, or an invented return deadline
  • 1: all wrong, or it refuses every question

7.Privacy & Permissions

Read in the privacy policy and tried in the product: what shopping history is stored, whether it is used for ads or sold, whether chats can be deleted, whether an account can be deleted from the app, whether paid or sponsored results are labelled, whether a saved card can be used without a confirmation, whether spending limits can be set, whether a browser extension reads pages outside shopping sites, whether a data export exists, and whether the policy names the AI model provider that receives the chats. One point each.

  • 5: 9 or 10 points
  • 4: 7 or 8
  • 3: 5 or 6
  • 2: 3 or 4
  • 1: 2 or fewer, or a card charged without confirmation

8.Setup Time

For a consumer assistant the clock runs from installing the app or extension to the first correct answer on task one. For an assistant sold to stores it runs from starting the install on my test shop to the first correct answer about a product in its catalogue, including any catalogue sync, with the number of settings I had to change counted. Time waiting for a sales call or a manual onboarding is included.

  • 5: under 5 minutes
  • 4: 5 to 29 minutes
  • 3: 30 minutes to 4 hours
  • 2: 4 hours to 3 days
  • 1: over 3 days, or a sales call is required before anything works

9.Pricing & Value

Every plan is read from the checkout or billing page, not the marketing page. For a consumer assistant I note whether it is free, what the paid tier adds, and how it earns money: affiliate commissions, a subscription, or paid placements, as stated in its terms. For a store assistant the figure graded is the yearly cost of the cheapest plan that handles 3,000 shopper conversations a month, with per-conversation, per-resolution and per-order fees worked into it. Renewal prices and usage caps are listed with the payment methods seen at checkout.

  • 5: free for shoppers, or under $300 a year for a store, with no per-conversation fee
  • 4: under $1,200 a year, or a paid consumer tier with a free tier that does the tasks
  • 3: $1,200 to $3,599 a year, or usage caps that stop the tasks on the free tier
  • 2: $3,600 or more, or pricing only on a sales call
  • 1: poor value or costs not disclosed before signing

Documented but not graded

Visual search from a photo, price alerts and price history, personalisation over time, proactive messages to shoppers, languages, voice, and the channels a store assistant can run on (website, app, WhatsApp, phone) are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real use of the product on a real account, with real products, live prices and real orders, never promotional descriptions, case studies or demo stores run by the vendor.
  • Identical tasks, catalogue, tests, thresholds and sample sizes for every assistant in the category.
  • Authorised accounts only; no password sharing, no bypassing of plan limits or bot checks, and test orders placed for real at the lowest value that fits the task and refunded or kept.
  • Every grade has evidence: a timing, a screenshot, a transcript or a table, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI shopping assistant for testing. No AI shopping assistant has been through these tests yet; every one I follow is listed on the AI shopping assistants page, and reviews appear there as tests finish.