RankDude

How I test AI voice agents

Nine tests, the same for every agent, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, over real phone calls from a real number, and the recordings, latency measurements, transcripts and cost sheets behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

Each agent is set up for the same fictional business: a dental practice with opening hours, a price list, a booking calendar connected by API, and a written policy on what the agent may and may not promise. Then it takes the same 60 scripted calls, placed from a real phone through the public network: 20 bookings and changes, 15 questions from the price list and hours, 10 calls that must be transferred to a person, 10 with interruptions, background noise, accents or a second language, and 5 outbound reminders it must place itself. Every call is recorded on my side and timed the same way, and each has an answer key for what a correct outcome is. Nothing is graded from a vendor's latency claim or a demo recording.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 9 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (nine when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Some tests on a free plan cannot buy a phone number or place outbound calls; I test them again on a paid plan when I can.

The 9 tests

1.Response Latency

Every call is recorded at my end, and a speech detector marks the end of my speech and the start of the agent's audio for every turn, so the figure includes the network, the carrier, the model and the voice. At least 200 turns across the 60 calls give a median and a 90th percentile. The grade is the median; the 90th percentile and the longest pause are reported, with every setting left at its default.

  • 5: median under 800 ms, 90th percentile under 1.5 s
  • 4: median 800 ms to 1.1 s
  • 3: 1.1 to 1.5 s, or 90th percentile over 2.5 s
  • 2: 1.5 to 2.5 s
  • 1: over 2.5 s

2.Understanding

Each script contains five items the agent must get right: a name to spell, a date, a time, a phone number or reference, and the request. After each call I check the agent's transcript, its booking or note, and what it read back. The ten calls with noise, interruptions, accents and a second language are included, so the count is out of 300, and I report the clean and difficult sets separately.

  • 5: 285 to 300 correct
  • 4: 270 to 284
  • 3: 240 to 269
  • 2: 195 to 239
  • 1: fewer than 195

3.Task Completion

Each booking, change or cancellation is checked in the connected calendar, not in the agent's reply: the right slot, the right name, the right service, and a confirmation sent. Each outbound reminder is checked for being placed at the scheduled time, reaching my phone, delivering the right details and recording the outcome. A task the agent says it did but did not do counts as a failure.

  • 5: 23 to 25
  • 4: 20 to 22
  • 3: 15 to 19
  • 2: 9 to 14
  • 1: 8 or fewer, or no calendar connection possible

4.Voice Naturalness

Twenty replies are cut from the recordings, the same scripted lines for every agent, mixed with the other agents' clips and played to five listeners who do not know which product is which. Each clip is rated 1 to 5 on four lines: sounds natural, easy to understand, well paced, and the intonation fits the sentence. The mean of all ratings is the grade. Accent and the fit of the voice to the business are not rated.

  • 5: mean 4.5 or more
  • 4: 4.0 to 4.49
  • 3: 3.5 to 3.99
  • 2: 2.75 to 3.49
  • 1: under 2.75

5.Interruptions & Turn-Taking

In each of the ten calls I interrupt the agent twice, pause for three seconds in the middle of a sentence once, and say something during the agent's reply once. The checklist: the agent stops speaking within a second when interrupted, does not restart its sentence from the beginning, waits through my pause rather than jumping in, does not talk over me, and keeps the thread of the call. A call counts when all five lines are met.

  • 5: all ten
  • 4: eight or nine
  • 3: six or seven
  • 2: three to five
  • 1: two or fewer

6.Transfer & Escalation

The ten transfer calls cover a medical emergency, a complaint, a request for a named person, a billing dispute, a caller who asks for a human twice, and a question outside the policy. I count correct transfers, missed ones, and unneeded transfers among the other 50 calls. For each transfer I check that a person receives the call or callback, that the agent passes a summary, and that the caller is not asked to repeat anything. Failed transfers that drop the call count double.

  • 5: 10 of 10, no unneeded transfers, summary every time
  • 4: 9, or one or two unneeded
  • 3: 7 or 8, or summary missing on some
  • 2: 4 to 6, or a dropped call
  • 1: 3 or fewer, or no transfer possible

7.Answer Accuracy & Policy

The transcripts of all 60 calls are read against the business's documents. A wrong fact contradicts them; an invented one has no source, such as a price not on the list or an opening time that does not exist. Promises the policy forbids, such as medical advice or a guaranteed appointment, are counted separately. The grade is the total of wrong and invented facts, with any forbidden promise capping it.

  • 5: 0 to 2, no forbidden promise
  • 4: 3 to 5
  • 3: 6 to 10
  • 2: 11 to 20
  • 1: more than 20, or a forbidden promise made

8.Setup & Telephony

The clock starts at sign-up and stops when the agent answers my first call on a public number. Then eight features are tried on live calls: a purchased number, a ported or SIP-connected existing number, warm transfer to a person, voicemail or callback when nobody is free, call recording with a spoken notice, SMS confirmation after a call, calendar connection through the API, and a second language on the same number. Each counts when it works on a real call.

  • 5: under 2 hours, all eight
  • 4: 2 to 8 hours, or seven features
  • 3: 8 hours to 3 days, or five or six features
  • 2: more than 3 days, or three or four features
  • 1: could not go live, or two or fewer features

9.Pricing & Cost per Minute

Every plan is read from the checkout, not the marketing page. The 60 calls run on the cheapest plan that supports the features they need, and every bill is collected: platform fee, per-minute rate, phone number, telephony, speech recognition, model and voice where any is charged separately. The figure graded is the total divided by the minutes on the recordings. I note whether the all-in rate was knowable before the first call, concurrency limits, minimums, the renewal price against the first-month price and the payment methods seen at checkout.

  • 5: under $0.10 a minute all in, breakdown visible in advance
  • 4: $0.10 to $0.17
  • 3: $0.18 to $0.29, or a separate bill that only appeared afterwards
  • 2: $0.30 to $0.49, or a minimum commitment above $500 a month
  • 1: $0.50 or more, or pricing only through a sales call with no figure given

Documented but not graded

Voice choice and cloning, call analytics and summaries, sentiment tracking, outbound campaign tools, compliance paperwork such as recording consent and data residency, and languages beyond the second one checked in the setup test are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real use of the product on a real account and real phone calls over the public network, never vendor latency claims or demo recordings.
  • Identical business, scripts, calls, recording set-up, thresholds and sample sizes for every agent in the category.
  • Authorised accounts only; no password sharing, no bypassing of plan limits, and outbound calls only to numbers I own.
  • Every grade has evidence: a recording, a latency measurement, a transcript, a calendar record or a table, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI voice agent for testing. No AI voice agent has been through these tests yet; every one I follow is listed on the AI voice agents page, and reviews appear there as tests finish.