How I test AI answering services

Eight tests, the same for every service, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, with real calls to a real number at the hours a business is least able to answer, and the recordings, transcripts, timings and screenshots behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

The RankDude mascot holding a laptop with a check on its screen

Each service is set up for the same imaginary business: a two-van electrical contractor whose phone goes unanswered when the owner is on a ladder, after six in the evening and all weekend. It gets the same 20 facts, the same list of who to text and when, the same calendar and the same instructions for emergencies. Then I phone it: 60 scripted calls from different handsets across one working week, most of them outside office hours or while the line is already busy, covering questions, messages, bookings, emergencies, awkward callers and spam. Every call is recorded with the caller's knowledge, and nothing is graded from a feature list or a demo recording.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 8 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5	Excellent
4	Good
3	Average
2	Below average
1	Poor

The final score is the average of the graded tests (eight when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Services that pair the AI with human receptionists are graded on what the AI does on its own, and the review says separately what the human tier adds and costs.

The 8 tests

1.Pick-up & Response Speed

Every one of the 60 calls is timed from the first ring to the greeting and checked for a missed or dropped pick-up. Ten of the calls are made in pairs from two handsets at the same moment, to see whether the second caller is answered or sent to voicemail. On 20 calls I measure the gap between the end of my sentence and the start of the reply at five points, from the recording. The grade combines the pick-up rate and the median gap; the longest gap is reported alongside.

5: every call answered within two rings, both of each pair answered, median reply gap under 1 second
4: every call answered, median gap 1 to 1.5 seconds
3: one or two calls missed or a paired call sent to voicemail, or median gap 1.5 to 2.5 seconds
2: three to five calls missed, or median gap 2.5 to 4 seconds
1: six or more calls missed, or median gap over 4 seconds

2.Voice & Conversation

Three fixed 30-second samples from the recordings are rated on naturalness, clarity, pacing and intonation, each on a 1 to 5 line, by me and two other listeners who do not know which service they are hearing. On ten calls I interrupt mid-sentence and count whether the service stops, listens and answers the new question. Ten further calls go off script, among them a wrong number, two sales pitches, a caller who switches to Spanish, a rude caller, a bad line, a long silence, a request the business does not offer. Each is marked handled, muddled or failed, and anything the service agreed to that the business never offers is noted.

5: listener rating 4.5 or more, all ten interruptions handled, all ten off-script calls handled
4: rating 4 to 4.4, at least eight interruptions and eight off-script calls handled
3: rating 3 to 3.9, or five to seven of either
2: rating 2 to 2.9, or fewer than five of either, or a wrong promise
1: rating under 2, or spam booked as a job

3.Answer Accuracy

The business profile holds 20 facts: call-out fees, hours, service area, what the firm does and does not do, certifications, payment methods, how long a quote takes. Each fact is asked for on at least one call, some in two wordings. An answer counts as right only if it matches the profile. Invented answers, including a made-up price or a promise of a time the firm never gave, are counted separately and weigh more than a plain 'I don't know'.

5: 19 or 20 right, nothing invented
4: 17 or 18 right, at most one invented answer
3: 14 to 16 right, or two invented answers
2: 10 to 13 right, or three or four invented answers
1: under 10 right, or five or more invented answers

4.Message Quality & Delivery

Each of the fifteen callers gives a name, a callback number, an address, what they need and how urgent it is; three spell their details out, three mumble them, and two give a number with a mistake they then correct. I check the message, summary or lead record for every field, whether the urgency was captured, and how long it took to arrive by text, email or app. A wrong phone number counts as a failed message, because the call-back never happens.

5: all fifteen complete and correct, every one delivered within 2 minutes with a recording
4: thirteen or fourteen correct, or delivery within 10 minutes
3: ten to twelve correct, or delivery within an hour
2: six to nine correct, or a wrong phone number in any message
1: five or fewer correct, or messages only visible by logging in

5.After-hours & Emergencies

The profile sets different rules for evenings, weekends and a public holiday: who is on call, what counts as an emergency, what to tell everyone else and when to send the morning summary. Twenty calls are made in those windows. Five describe an emergency (a burning smell, no power to a house with a medical device) and must reach the on-call number by call or text within five minutes; I time each one. The other fifteen are checked for the right greeting, the right promise about a call-back and whether the morning summary listed them.

5: all twenty by the rules, all five emergencies reach the on-call phone within 5 minutes
4: eighteen or nineteen by the rules, every emergency within 15 minutes
3: fifteen to seventeen by the rules, or one emergency late
2: ten to fourteen, or an emergency treated as a routine message
1: under ten, or no after-hours rules can be set

6.Booking & Transfers

The calendar has fixed gaps and one planted clash. Eight calls book a job; after each I check the date, time, name, number and job on the connected calendar. Eight calls need a person during office hours: a complaint, a caller who insists, a supplier. I count whether each reaches the named phone, whether the person hears or receives a summary first, and what the caller hears if nobody picks up. Taking a message instead of transferring, or logging a booking request for a person to confirm, counts as a partial.

5: all sixteen correct, failed transfers fall back to a message
4: fourteen or fifteen correct
3: eleven to thirteen, or bookings are requests a person must confirm
2: seven to ten, or a double booking, or the caller is cut off when nobody answers
1: six or fewer, or neither booking nor transfer is offered

7.Setup & Integrations

The clock starts at sign-up and stops when a forwarded call to the business number is answered correctly, my own fixes included. Then each point is tried, not read from a feature list: conditional forwarding (only when unanswered), a new number, Google Calendar, Outlook calendar, a CRM or job app the service claims to support, Zapier or a webhook, a text to the caller after the call, call recordings, transcripts, and a daily summary email. A service that cannot be tried without a sales call is noted and scores 2 at best.

5: live within 30 minutes, 9 or 10 points working
4: live within 2 hours, 7 or 8 points
3: live the same day, 5 or 6 points
2: live within a week, 3 or 4 points, or setup needed a sales call
1: not live within a week, or 2 or fewer points

8.Pricing & Value

Every plan is read from the checkout or order page, not the marketing page. The figure graded is the monthly cost of handling 150 calls averaging two minutes on the cheapest plan that includes messages by text, after-hours rules and a transfer, with overage, number rental, texting fees and setup fees included. I record the billing unit (per minute, per call, per caller or flat), what happens when the allowance runs out, the contract length, the trial and whether a card is needed, what the terms say about recordings and caller data, and whether a HIPAA agreement is offered. Where a human tier exists its price is listed separately.

5: under $75 a month, no overage surprise, monthly contract
4: $75 to $149 a month
3: $150 to $299, or a yearly contract
2: $300 or more, or the price is only available from a sales call
1: poor value or hidden costs

Documented but not graded

Languages beyond English, outbound call-backs, text and web chat handling, a mobile app, call analytics, spam blocking and the human receptionist tier offered by some services are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

Real calls to a real number on a real account, at the hours the service is sold for, never promotional descriptions or demo recordings.
Identical business profile, call scripts, thresholds and sample sizes for every service in the category.
Authorised accounts only; no password sharing, no bypassing of plan limits, and every test call made by me or by someone who knows it is recorded.
Every grade has evidence: a recording, a transcript, a timing, a screenshot or a table, in the full test results.
A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI answering service for testing. No AI answering service has been through these tests yet; every one I follow is listed on the AI answering services page, and reviews appear there as tests finish.