RankDude

How I test AI customer service agents

Nine tests, the same for every agent, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, against the same set of support tickets on a real help desk, and the transcripts, timings, screenshots and cost sheets behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

Each agent is set up for the same fictional business: an online shop with a 60-article help centre, a product catalogue, an order system with a test API, and a written policy on refunds, shipping and what the agent may never promise. Then it works the same 100 tickets, sent as a real customer would send them: 40 by chat, 30 by email, 20 over WhatsApp and 10 in Slack. Thirty of the tickets can be answered from the help centre, forty need an action such as checking an order or starting a return, twenty are outside the agent's remit and should go to a person, and ten are written to be rude, confused or in another language. Each ticket has an answer key: what a correct resolution looks like, which facts must be right, and whether a handoff is the right outcome. Nothing is graded from a vendor's resolution-rate claim or a case study.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 9 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (nine when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Some tests on a free plan cannot connect every channel or run all 100 tickets; I test them again on a paid plan when I can.

The 9 tests

1.Resolution Rate

A ticket is resolved when the customer's question is answered or the action is completed and the answer key is met, with no human involved. A ticket closed without the customer's problem being solved, or marked resolved after the customer stopped replying, is not a resolution. The twenty tickets that should be handed to a person are graded in the handoff test, not here, so this count is out of 80.

  • 5: 64 or more of 80 resolved (80%)
  • 4: 52 to 63 (65% to 79%)
  • 3: 40 to 51 (50% to 64%)
  • 2: 24 to 39 (30% to 49%)
  • 1: fewer than 24

2.Answer Accuracy

Each reply is read against the sources the agent was given. A wrong fact is one that contradicts them; an invented fact is one with no source at all, such as a made-up delivery date, a discount that does not exist or a policy the business never wrote. Promises the policy forbids are counted separately and reported. The grade is the total count of wrong and invented facts.

  • 5: 0 to 2
  • 4: 3 to 5
  • 3: 6 to 10
  • 2: 11 to 20
  • 1: more than 20, or a forbidden promise made

3.Actions Taken

The actions: look up an order, change a delivery address, start a return, issue a refund within the policy limit, cancel an order, resend a confirmation, apply a credit, update an email address, book a callback and raise a ticket in the help desk. Each is checked in the system itself, not in the agent's reply. An action the agent says it did but did not do counts as a failure and as an invented fact.

  • 5: 36 to 40
  • 4: 30 to 35
  • 3: 22 to 29
  • 2: 12 to 21
  • 1: 11 or fewer, or no actions possible

4.Handoff to a Person

The twenty handoff tickets cover a complaint about an injury, a request to speak to a manager, a legal threat, a refund above the agent's limit, a question about a product that does not exist and an account the agent cannot see. I count correct handoffs, missed handoffs, and unneeded handoffs among the other 80 tickets. For each correct handoff I check that the person receives the conversation, the customer's details and a summary, and that the customer is not asked to repeat anything.

  • 5: 20 of 20 handed over, no unneeded handoffs, full context every time
  • 4: 18 or 19, or one or two unneeded handoffs
  • 3: 15 to 17, or context missing on some
  • 2: 10 to 14, or no summary passed
  • 1: fewer than 10, or no handoff possible

5.Tone & Difficult Customers

For each of the ten: the agent stays polite, does not argue, does not apologise more than once, answers the actual question, does not repeat itself, writes in the customer's language when the ticket is in Spanish or German, keeps the business's set tone, and offers a person when the customer asks twice. A ticket counts when every line of the checklist is met; I publish the transcripts.

  • 5: all ten
  • 4: eight or nine
  • 3: six or seven
  • 2: three to five
  • 1: two or fewer

6.Channels

The eight: website chat, email, WhatsApp, Slack, SMS, Instagram or Facebook messages, a help centre search box, and voice. Each counts when a real ticket is received and resolved through it. Then five customers start on one channel and continue on another; I check that the agent recognises them and keeps the history.

  • 5: all eight, history carried across in all five moves
  • 4: six or seven
  • 3: four or five, or history lost on some moves
  • 2: two or three
  • 1: one, or no history across channels

7.Setup & Training

The clock starts at sign-up and stops when the agent answers a help-centre ticket correctly over chat. I record whether connecting the help centre, the policy document and the order API needed code or a vendor's team, how long each took, and whether the agent stayed current when I changed three help-centre articles and sent tickets about them a day later.

  • 5: under 2 hours, no engineer, changes picked up within a day
  • 4: 2 to 8 hours
  • 3: 1 to 3 days, or one connection needed code
  • 2: more than 3 days, or a vendor team needed for setup
  • 1: could not be set up on my own, or changes never picked up

8.Reporting & Control

After the 100 tickets I check that each appears in a log with its outcome, that the agent's own resolution figure matches my count (an agent that counts abandoned chats as resolved is marked down), that I can see what each ticket cost, that there is a sandbox or test mode for new instructions, that changes can be reviewed before going live, and that transcripts can be exported. Six points, checked on the last day.

  • 5: all six, resolution figure within 5 points of my count
  • 4: five
  • 3: four, or resolution figure 6 to 15 points above my count
  • 2: two or three, or resolution figure more than 15 points above
  • 1: one or none

9.Pricing & Cost per Ticket

Every plan is read from the checkout or the written quote, not the marketing page. The 100 tickets are run on the cheapest plan that supports the channels and actions they need, and the resolutions, conversations, seats or credits billed are recorded. The figure graded is the cost per ticket at that volume, platform fee included. I note what the vendor counts as a billable resolution, whether minimums or platform fees apply, the renewal price against the first-year price and the payment methods seen at checkout.

  • 5: under $1 a ticket, billing rule clear and honest
  • 4: $1 to $1.99
  • 3: $2 to $3.99, or a billing rule that counts silence as resolution
  • 2: $4 to $7.99, or a minimum commitment above $500 a month
  • 1: $8 or more, or pricing only available through a sales call with no figure given

Documented but not graded

Proactive messages, agent assist for human staff, voice quality, analytics on customer sentiment, multilingual coverage beyond the two tested languages and the quality of the help desk itself are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real use of the product on a real account and real tickets on a real help desk, never vendor resolution-rate claims or case studies.
  • Identical business, help centre, policy, tickets, thresholds and sample sizes for every agent in the category.
  • Authorised accounts only; no password sharing, no bypassing of plan limits or bot checks, and refunds and orders only in test mode.
  • Every grade has evidence: a transcript, a timing, a screenshot, a system record or a table, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI customer service agent for testing. No AI customer service agent has been through these tests yet; every one I follow is listed on the AI customer service agents page, and reviews appear there as tests finish.