RankDude

How I test AI phone assistants

Nine tests, the same for every assistant, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, with real calls to a real number from handsets the assistant has never seen, and the recordings, transcripts, timings and screenshots behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

The category covers business phone apps built around an AI assistant and self-serve AI phone agents that a solo operator or small team puts on its own number; both kinds take the same calls. Each assistant is put in charge of the same phone line: the business number of an imaginary one-person accountancy practice with 20 published facts, a calendar with gaps in it, a short list of known contacts and clear rules about who gets through, who gets a message and who gets nothing. Then I phone it: 70 scripted calls over five working days from different handsets, including clients, strangers, sales calls, robocalls, a withheld number, bookings, emergencies and awkward callers. Every call is recorded with the caller's knowledge, and nothing is graded from a feature list or a demo video.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 9 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (nine when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Assistants built into a phone or a phone system are tested on that phone or system, and the review says what else you have to buy to get them.

The 9 tests

1.Pick-up & Response Speed

Every call that should be answered is timed from the first ring to the greeting and checked for missed or dropped pick-ups. On 20 calls I measure the gap between the end of my sentence and the start of the assistant's reply at five points, from the recording. The grade combines the pick-up rate and the median gap; the longest gap is reported alongside.

  • 5: every call answered within two rings, median reply gap under 1 second
  • 4: every call answered, median gap 1 to 1.5 seconds
  • 3: one or two calls missed, or median gap 1.5 to 2.5 seconds
  • 2: three to five calls missed, or median gap 2.5 to 4 seconds
  • 1: six or more calls missed, or median gap over 4 seconds

2.Voice & Conversation

Three fixed 30-second samples from the recordings are rated on naturalness, clarity, pacing and intonation, each on a 1 to 5 line, by me and two other listeners who do not know which assistant they are hearing. On ten calls I interrupt mid-sentence and count whether the assistant stops, listens and answers the new question. Ten further calls go off script, among them a wrong number, a caller who switches to Spanish, a rude caller, a bad line, a long silence, a request outside the business, a caller who changes their mind twice. Each is marked handled, muddled or failed.

  • 5: listener rating 4.5 or more, all ten interruptions handled, all ten awkward calls handled
  • 4: rating 4 to 4.4, at least eight of each
  • 3: rating 3 to 3.9, or five to seven of either
  • 2: rating 2 to 2.9, or fewer than five of either
  • 1: rating under 2, or the assistant hangs up on a real caller

3.Screening & Spam

The rules say four named contacts ring through, strangers are asked their name and reason and then put through or taken as a message, sales calls and robocalls are declined, and withheld numbers are screened. Twenty calls test them: four from the named contacts, six from strangers with legitimate reasons, six robocall-style pitches played from a recording, two live sales pitches and two from a withheld number. For each I record what happened, what the caller heard and what the owner saw. A known contact made to explain themselves, or a robocall passed through as important, counts as a fail.

  • 5: all twenty sorted correctly
  • 4: eighteen or nineteen
  • 3: fifteen to seventeen
  • 2: ten to fourteen, or a known contact screened like a stranger
  • 1: nine or fewer, or spam passed through as important, or no screening offered

4.Answer Accuracy

The profile holds 20 facts: fees, hours, address, what the practice does and does not do, filing deadlines it handles, how to send documents, payment methods. Each fact is asked for on at least one call, some in two wordings. An answer counts as right only if it matches the profile. Invented answers, including any tax advice, are counted separately and weigh more than a plain 'I don't know'.

  • 5: 19 or 20 right, nothing invented
  • 4: 17 or 18 right, at most one invented answer
  • 3: 14 to 16 right, or two invented answers
  • 2: 10 to 13 right, or three or four invented answers
  • 1: under 10 right, five or more invented answers, or professional advice given

5.Messages & Summaries

Each of the twelve callers gives a name, a callback number, what they need and how urgent it is; three spell their details out, three mumble them, and two correct a mistake mid-call. I check the message, summary or notification for every field and the urgency, and time how long it takes to arrive by text, email, app or in the dashboard. A wrong phone number counts as a failed message, because the call-back never happens.

  • 5: all twelve complete and correct, every one delivered within 2 minutes with a transcript
  • 4: ten or eleven correct, or delivery within 10 minutes
  • 3: eight or nine correct, or delivery within an hour
  • 2: five to seven correct, or a wrong phone number in any message
  • 1: four or fewer correct, or messages only visible by logging in

6.Booking & Transfers

The calendar has fixed gaps and one planted clash. Eight calls book an appointment; after each I check the date, time, name, number and reason on the connected calendar. Eight calls need the owner now: an urgent client, a caller who insists, a named contact. I count whether each reaches the owner's phone, whether the owner hears or sees a summary before being connected, and what the caller hears if the owner does not pick up. Taking a message instead of transferring, or logging a booking request for a person to confirm, counts as a partial.

  • 5: all sixteen correct, failed transfers fall back to a message
  • 4: fourteen or fifteen correct
  • 3: eleven to thirteen, or bookings are requests a person must confirm
  • 2: seven to ten, or a double booking, or the caller is cut off when nobody answers
  • 1: six or fewer, or neither booking nor transfer is offered

7.Texting & Follow-up

Ten callers ask for something that should arrive by text: the address, a booking confirmation, a link to upload documents, a call-back time. For each I check that the caller receives the right text and the owner receives a notification, and I time both. Then I reply to three of the caller texts and check whether the reply reaches the owner and whether the assistant keeps the thread going. Assistants with no texting score 1.

  • 5: all twenty texts correct within 2 minutes, replies reach the owner
  • 4: seventeen to nineteen correct
  • 3: thirteen to sixteen correct, or replies go nowhere
  • 2: eight to twelve correct, or texts to the caller only
  • 1: seven or fewer, or no texting

8.Setup & Integrations

The clock starts at sign-up and stops when a call to the business number is answered correctly, my own fixes included. Then each point is tried, not read from a feature list: conditional forwarding (only when unanswered), a new number, keeping an existing number, Google Calendar, Outlook calendar, a CRM, Zapier or a webhook, call recordings, transcripts, and different rules for after hours. Assistants that only work on one make of phone are noted and the phone is named.

  • 5: live within 30 minutes, 9 or 10 points working
  • 4: live within 2 hours, 7 or 8 points
  • 3: live the same day, 5 or 6 points
  • 2: live within a week, 3 or 4 points, or setup needed a sales call
  • 1: not live within a week, or 2 or fewer points

9.Pricing & Value

Every plan is read from the checkout, order page or app store listing, not the marketing page. The figure graded is the monthly cost of handling 150 calls averaging two minutes on the cheapest plan that includes screening, messages by text and a transfer, with overage, number rental and texting fees included. If the assistant needs a particular phone, a particular carrier or a phone system subscription, that cost is added. I record the billing unit, the contract length, the trial, whether a card is needed, and what the terms say about recordings, transcripts and whether calls train the vendor's models.

  • 5: under $50 a month, no overage surprise, works with the phone and number you have
  • 4: $50 to $99 a month
  • 3: $100 to $199, or a yearly contract, or a new number is compulsory
  • 2: $200 or more, or a particular phone or phone system must be bought
  • 1: poor value or hidden costs

Documented but not graded

Outbound calls made on your behalf, waiting on hold for you, languages beyond English, voicemail transcription, call analytics, team inboxes and the mobile app are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real calls to a real number on a real account, never promotional descriptions or demo recordings.
  • Identical business profile, call scripts, thresholds and sample sizes for every assistant in the category.
  • Authorised accounts only; no password sharing, no bypassing of plan limits, and every test call made by me or by someone who knows it is recorded.
  • Every grade has evidence: a recording, a transcript, a timing, a screenshot or a table, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI phone assistant for testing. No AI phone assistant has been through these tests yet; every one I follow is listed on the AI phone assistants page, and reviews appear there as tests finish.