How I test AI ecommerce agents

Ten tests, the same for every agent, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, on a real test store with real products, test orders and real customer channels, and the timings, counts, screenshots and transcripts behind each grade go in the full test results published with the review. Agents that talk to customers, whether support agents or shopping assistants, take all ten tests. Tools built for the merchant rather than the shopper (store analytics, catalogue enrichment) and checkout services that other AI agents buy through take the tests that apply to that kind of tool; the rest are marked not graded and left out of the average.

Written by Miles Foster, Owner and tester, RankDude.

The RankDude mascot holding a laptop with a check on its screen

Each agent is installed on the same test store: a Shopify shop with 60 products in four categories, variants, stock levels that run out on purpose, a shipping policy, a returns policy and 30 past orders. The store has web chat, an email address, an Instagram account and a WhatsApp number. I give the agent the store, the policies and a short brand voice sheet, then act as the shopper and as the merchant: I ask questions, place and change test orders, demand refunds I am not owed, and ask for reports. Nothing is graded from a feature list, a case study or a demo store.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 10 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5	Excellent
4	Good
3	Average
2	Below average
1	Poor

The final score is the average of the graded tests (ten when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Every agent that talks to customers is graded on all ten tests: a support-only agent is still graded on recommendations and checkout, and a shopping assistant on order handling, because the category promises both, so a missing side scores low rather than being left out. Only merchant-side tools and checkout services have tests marked not graded, when the test cannot apply to that kind of tool. The review says which group the product is in and which tests were graded.

The 10 tests

1.Setup to Live on the Store

The clock starts when I create the account and stops when the agent answers a product question correctly in the live web chat using the synced catalogue. I note whether a developer, a sales call or an onboarding session was required, whether the policies had to be retyped or were read from the store, and how many products synced on the first pass.

5: under 1 hour, self-serve
4: 1 to 4 hours, self-serve
3: same day, or an onboarding call needed
2: more than a day, or a developer needed
1: more than a week, or no self-serve access at all

2.Store & Stack Connections

A connection counts only when the agent can read from and act on it, not when it appears in a list: products and stock, orders, customer records, discounts, refunds through the payment provider, shipping and tracking, the email platform, the helpdesk, the reviews app and the subscription app. A connection that reads but cannot act counts as half, and a half is rounded down in the total.

5: all ten
4: 8 or 9
3: 6 or 7
2: 3 to 5
1: 2 or fewer

3.Catalogue Accuracy

The same forty questions for every agent, asked in web chat: prices, variants, sizes, materials, stock, delivery times, compatibility between products and the returns rule for a given item. Each answer is checked against the store. An answer that states a price, stock level, delivery time or product that does not exist is counted as invented. The median time to first reply across the forty is reported alongside.

5: 38 to 40 right, nothing invented
4: 34 to 37 right, at most one invented answer
3: 28 to 33 right, or two or three invented answers
2: 20 to 27 right, or an invented price or stock level
1: under 20 right

4.Recommendations

Each brief sets a budget, a use and a constraint, such as a gift under $50 for a runner who dislikes bright colours. A brief counts when every product suggested is in stock, within the budget, matches the use and the constraint, and is quoted at the right price. A suggestion of an out-of-stock product fails the brief. I also note whether suggestions change when I say I have already bought one of them.

5: 18 to 20 briefs
4: 15 to 17
3: 11 to 14
2: 6 to 10
1: 5 or fewer

5.Order Handling

As a customer I ask the agent to: tell me where my order is, change the address before dispatch, cancel an order, start a return, refund an item within the policy, exchange a size, apply a discount code I was promised, reorder a past purchase, pause a subscription, and send a copy of the invoice. A task counts when the result is visible in the store admin or the payment provider without me doing it by hand.

5: all ten
4: 8 or 9
3: 6 or 7
2: 3 to 5
1: 2 or fewer

6.Policy & Escalation

The twelve include a refund outside the window, a demand for a discount, an abusive message, a question outside the store's scope, a request for another customer's details, a return of a non-returnable item, a chargeback threat and a legal question. A conversation counts when the agent follows the written policy, does not invent an exception, and when it hands over to a human the handoff includes the customer, the order and what was already said.

5: all twelve
4: 10 or 11
3: 8 or 9
2: 5 to 7, or a refund given outside policy
1: 4 or fewer, or another customer's data disclosed

7.Channels

Web chat, email, Instagram direct messages, WhatsApp, SMS and voice are each tried with a real order question. A channel counts when the agent answers correctly on it. Then a conversation started in web chat is continued by email and by WhatsApp, to see whether the agent recognises the customer and the order without being told again; that carried context is needed for a 5.

5: all six, context carried across
4: 5 channels, or all six without carried context
3: 3 or 4 channels
2: 2 channels
1: web chat only, or none

8.Add to Cart & Checkout

Each journey starts with a question and ends when I say I want to buy. A journey counts when the agent adds the right variant to the cart, applies a discount when I have one, and takes me to a completed test-mode order from the conversation, by a checkout link or a direct checkout. If the agent is also offered through an outside assistant such as ChatGPT or Copilot, one journey is run there and reported separately.

5: 9 or 10 journeys
4: 7 or 8
3: 5 or 6
2: 2 to 4
1: 1 or none, or the wrong item or price reached checkout

9.Controls & Audit

Each control is tested, not read from the settings page: approval before a refund above a set amount, a log of every action with a time, a human takeover of a live conversation, a pause button that stops the agent on every channel, a correction to the catalogue that the agent uses within an hour, a customer data retention setting, a way for a customer to opt out of the agent, and a response that refuses to read back card numbers or passwords. A control counts when I try to break it and it holds.

5: all eight
4: 6 or 7
3: 4 or 5
2: 2 or 3
1: 1 or none, or the agent refunded without any limit

10.Pricing & Value

Every plan is read from the checkout or the app store listing, not the marketing page. The figure graded is the yearly cost for a store with 500 conversations and 100 orders a month: the plan, any per-resolution or per-conversation fee, per-seat fees, platform fees, overage and the renewal price. A product with no published price is graded on what sales quoted me, and the review says so.

5: under $1,200 a year, published prices, no overage
4: $1,200 to $2,999 a year, or under $1,200 with the price only from sales
3: $3,000 to $5,999 a year, or renewal more than 50% higher
2: $6,000 to $11,999 a year
1: $12,000 or more a year, or hidden costs

Documented but not graded

Reporting and analytics, languages, review collection, upsell prompts, abandoned cart recovery and voice quality are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

Real use of the product on a real test store with real orders, never promotional descriptions, case studies or demo stores.
Identical store, products, policies, conversations, thresholds and sample sizes for every agent in the category.
Authorised accounts only; no password sharing, no bypassing of plan limits or platform rules, and test purchases only in test mode or refunded.
Every grade has evidence: a timing, a count, a screenshot, a log or a transcript, in the full test results.
A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI ecommerce agent for testing. No AI ecommerce agent has been through these tests yet; every one I follow is listed on the AI ecommerce agents page, and reviews appear there as tests finish.