How I test AI receptionists

Nine tests, the same for every service, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, with real calls to a real number, and the recordings, transcripts, timings and screenshots behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

The RankDude mascot holding a laptop with a check on its screen

Each receptionist is set up for the same imaginary business: a three-person plumbing firm with a published price list, opening hours, a service area and a calendar with gaps in it. It gets the same 20 facts, the same booking rules and the same instructions about who to transfer to. Then I phone it: 60 scripted calls from different handsets over five working days, covering questions, bookings, messages, transfers, awkward callers and spam. Every call is recorded with the caller's knowledge, and nothing is graded from a feature list or a demo video.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 9 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5	Excellent
4	Good
3	Average
2	Below average
1	Poor

The final score is the average of the graded tests (nine when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Some services cannot transfer calls or connect a calendar on their cheapest plan; I test them again on a higher plan when I can.

The 9 tests

1.Pick-up & Response Speed

Every one of the 60 calls is timed from the first ring to the greeting, and every call is checked for a missed or dropped pick-up. On 20 of the calls I measure the gap between the end of my sentence and the start of the receptionist's reply at five points in the call, from the recording. The grade is the pick-up rate and the median reply gap together; the longest gap is reported alongside.

5: every call answered within two rings, median reply gap under 1 second
4: every call answered, median gap 1 to 1.5 seconds
3: one or two calls missed or dropped, or median gap 1.5 to 2.5 seconds
2: three to five calls missed, or median gap 2.5 to 4 seconds
1: six or more calls missed, or median gap over 4 seconds

2.Voice & Conversation

Three fixed 30-second samples from the recordings are rated on naturalness, clarity, pacing and intonation, each on a 1 to 5 line, by me and two other listeners who do not know which service they are hearing. On ten calls I interrupt mid-sentence and count whether the receptionist stops, listens and answers the new question. Robotic pauses, repeated phrases and talking over the caller are counted as faults.

5: average listener rating 4.5 or more, all ten interruptions handled
4: rating 4 to 4.4, at least eight interruptions handled
3: rating 3 to 3.9, or five to seven interruptions handled
2: rating 2 to 2.9, or fewer than five interruptions handled
1: rating under 2, or the receptionist talks over the caller on most calls

3.Answer Accuracy

The business profile holds 20 facts: prices, hours, service area, what the firm does and does not do, parking, payment methods. Each fact is asked for on at least one call, some in two different wordings. An answer counts as right only if it matches the profile. Invented answers (a price, a service or a promise that is not in the profile) are counted separately and weigh more than a plain 'I don't know'.

5: 19 or 20 right, nothing invented
4: 17 or 18 right, at most one invented answer
3: 14 to 16 right, or two invented answers
2: 10 to 13 right, or three or four invented answers
1: under 10 right, or five or more invented answers

4.Appointment Booking

The calendar has fixed gaps and two deliberate clashes. Eight calls book a job, two move an existing one and two cancel. After each call I open the calendar and check the date, time, duration, customer name, phone number and job description. A booking in a slot that was already taken, or at a time the business is closed, counts as wrong. If the service only takes a request for a human to confirm, that is reported and graded as a 3 at best.

5: all twelve correct, no clashes
4: ten or eleven correct
3: seven to nine correct, or bookings are requests a person has to confirm
2: four to six correct, or a double booking
1: three or fewer correct, or no booking at all

5.Messages & Lead Capture

Each of the ten callers gives a name, a callback number, an address and a reason for calling, two of them spelled out and two of them mumbled. I check the message, summary or lead record for every field, and time how long it takes to arrive by text, email or in the dashboard. A wrong phone number counts as a failed message, because the call-back never happens.

5: all ten complete and correct, every one delivered within 2 minutes
4: nine correct, or delivery within 10 minutes
3: seven or eight correct, or delivery within an hour
2: five or six correct, or a wrong phone number in any message
1: four or fewer correct, or messages only visible by logging in

6.Transfers & Human Handoff

The profile names two people and the reasons each should get a call. Ten calls ask for or need a person: an emergency, a complaint, a caller who insists. I count whether the call reaches the right phone, whether the person hears or receives a summary before the caller is connected, and what the caller hears when the transfer fails because nobody answers. Taking a message instead of transferring counts as a partial.

5: all ten reach the right person with a summary, and a failed transfer falls back to a message
4: nine reach the right person, or no summary is passed
3: seven or eight reach a person
2: four to six, or the caller is cut off when nobody answers
1: three or fewer, or the service cannot transfer at all

7.Awkward Calls & Spam

The ten calls include two robocall-style sales pitches, a wrong number, a caller who switches to Spanish, a caller who is rude, a request for a service the firm does not do, a caller who changes their mind twice, a caller on a bad line, a request to speak to a named person who does not exist, and a long silence. Each is marked handled, muddled or failed from the recording, and I note anything the receptionist agreed to that the business never offers.

5: all ten handled, nothing wrongly promised
4: eight or nine handled
3: six or seven handled, or one wrong promise
2: four or five handled, or the receptionist loops or hangs up on a real caller
1: three or fewer handled, or spam calls booked as jobs

8.Setup & Integrations

The clock starts at sign-up and stops when a forwarded call to the business number is answered correctly, my own fixes included. Then each point is tried, not read from a feature list: call forwarding from an existing number, a new number, Google Calendar, Outlook calendar, a CRM or job app (whichever the service offers), Zapier or a webhook, a text to the caller after the call, call recordings, transcripts, and after-hours rules that differ from daytime rules.

5: live within 30 minutes, 9 or 10 points working
4: live within 2 hours, 7 or 8 points
3: live the same day, 5 or 6 points
2: live within a week, 3 or 4 points, or setup needed a sales call
1: not live within a week, or 2 or fewer points

9.Pricing & Value

Every plan is read from the checkout or order page, not the marketing page. The figure graded is the monthly cost of handling 200 calls averaging two minutes, on the cheapest plan that includes booking and transfers, overage and any per-call fees included. I also record the billing unit (per minute, per call, per caller or flat), the contract length, the trial, whether a card is needed to try it, what the terms say about recordings and caller data, and whether a HIPAA agreement is offered. Renewal and introductory prices are listed separately.

5: under $100 a month, no overage surprise, monthly contract
4: $100 to $199 a month
3: $200 to $349, or a yearly contract
2: $350 or more, or the price is only available from a sales call
1: poor value or hidden costs

Documented but not graded

Languages beyond English, outbound call-backs, text and web chat handling, call analytics, a mobile app and the quality of the dashboard are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

Real calls to a real number on a real account, never promotional descriptions or demo recordings.
Identical business profile, call scripts, thresholds and sample sizes for every service in the category.
Authorised accounts only; no password sharing, no bypassing of plan limits, and every test call made by me or by someone who knows it is recorded.
Every grade has evidence: a recording, a transcript, a timing, a screenshot or a table, in the full test results.
A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI receptionist for testing. No AI receptionist has been through these tests yet; every one I follow is listed on the AI receptionists page, and reviews appear there as tests finish.