Smith.ai test data

Every call was placed from Smith.ai's in-browser test line to the business number (484) 276-2543 and timed from the moment the call page opened to the first word of the greeting. On the five final calls the gap between the end of each of my sentences and the receptionist's reply was measured on the stereo recording, twice: to the first sound (the typing 'thinking sound' Smith.ai plays while it works) and to the first spoken word. The medians below are the median of the per-call medians.

Calls were answered 25 times out of 25 attempts that rang. Four early attempts of call 1 were cut short by the receptionist because the caller side had gone digitally silent; that was a fault on my side, fixed with a little line noise, and those four are not counted against Smith.ai. A 26th attempt did not ring at all because the monthly allowance was used up.

The greeting arrived 12.3 to 15.5 seconds after the call page opened on the final calls (median 12.7 s), of which the first three to four seconds are call set-up before the ring tone starts, so the caller hears roughly nine seconds of ringing, about two rings.

Reply gaps: median 2.6 s to the first sound and 4.1 s to the first spoken word. The longest wait for a voice was 12.1 s, at the start of the angry call, after a 10-second opening speech. Several replies began with 'One moment please' or 'Give me just a second' and the real answer followed a few seconds later.

No call was dropped by the receptionist once the caller side was producing sound. The ring time was the same on every call, so the test line seems to add a fixed delay rather than vary.

• Measured over Smith.ai's own test line, not a phone handset; the fixed 12 to 15 seconds to the greeting may partly be the test page.
• Five calls measured instead of ten; to be repeated on a new account and from a real phone.

The default voice (Ava) was kept, as for every brand. Naturalness was judged from the recordings; the three-listener panel has not been run yet. Interruptions were tested on two calls: talking over the receptionist mid-sentence in the water heater call and in the ceiling-leak emergency. Repeated phrases, talking over the caller and robotic pauses were counted from the transcripts.

The voice is clear, paced like a call-centre agent and reads prices and times without stumbling ('two hundred twenty-five dollar call-out fee, in addition to the hourly rate of one hundred forty dollars per hour'). It never talked over me.

Both interruptions were handled: in the water heater call it stopped within about a second, answered the new question (Saturday hours) and then went back to the water heater; in the emergency call it stopped three seconds into 'That sounds urgent, and I want', waited, and resumed with the same line.

What breaks the illusion is repetition. Every factual answer ends with 'Is there anything else I can help you with today?', including the reply to 'thanks, bye'. In the emergency call 'That sounds urgent, and I want to help get this sorted right away' was said three times, each time before another intake question. The angry caller was asked for the service address three times in three wordings after giving it.

• No blind listener ratings yet; two interruptions tested instead of five.

The 20 facts in the business profile were entered as 20 FAQs plus the description and opening hours. They were asked for across the fact calls (1 to 5 and 8, recorded during the tuning of the caller side, and call 1 again in its final form), some twice in different words, and once under a false claim ('your website says you do AC'). An answer counts as right only if it matches the profile; an invented price, service or promise counts separately and weighs more than 'I don't know'.

19 of the 20 facts were asked (the office address was not). 18 were answered right, every price among them: $225 after-hours fee on top of $140 an hour, $95 diagnostic waived on a same-visit repair, $189 and $395 for drains, $240 boiler service, $220 plus the fixture, tankless from $3,900, free estimates over $1,000, 10% off labour at 65 and over, card, Zelle or check with no cash discount and no financing, one-year labour warranty, Brooklyn south of Prospect Park including Bay Ridge, no New Jersey, no air conditioning, no septic, founded 2016 by Dan Okafor, the three names, Saturday 8 to 1, closed Sunday with 24/7 emergencies, and 'typically within two business days with two-hour arrival windows'.

One fact was never stated: the $1,850 tank water heater price. Asked plainly ('how much is a new 40-gallon tank?') the receptionist moved into its qualification questions instead of answering, twice across two calls, while it did quote the tankless price. Removal of the old unit was 'included' in one call and 'I can't confirm whether removal is included' in the next.

Nothing was invented. Under the false claim it said: 'Our information says we do not provide air conditioning or HVAC services. I don't have access to verify the website wording, but someone can follow up about the discrepancy.' Pushed to make the senior discount 20%: 'we only offer the 10% labor discount for customers aged 65 and over, with no other discounts.' Asked whether it was a real person: 'You're speaking with an automated assistant.'

Call 5, the false claim

Me: Your website says you do air conditioning. AI: Our information says we do not provide air conditioning or HVAC services. I don't have access to verify the website wording, but someone can follow up about the discrepancy.

Call 4, the real-person question

Me: Am I talking to a real person or a computer? AI: You're speaking with an automated assistant. I can help with general information and get you to the right person if needed.

• Six of the fact calls were recorded while the caller side was still being tuned, so some questions were put more than once; the answers are the receptionist's own and are kept in full.
• The address fact was not asked; the second-wording checks for four facts are still to run.

Booking needs three things on Smith.ai: the paid appointment-booking feature (pre-paid credits, $3 a call on the Free plan), a scheduling tool from its list (Calendly, Acuity, Housecall Pro or Lawmatics; a free Calendly account is refused), and the tool connected to the new-client flow. All three were set up: $15 of credits, Calendly Standard at $12 for a month on the test Google calendar with a two-hour 'Plumbing visit' event, and seven existing jobs plus an all-day 'Dan off' on Monday 26 October as the deliberate clash. The five booking calls (three new bookings, a change and a cancellation) were not run before the call allowance ran out, so the test is not graded.

Setup checked in the dashboard: billing credits $15, Calendly Standard paid and connected (the Integrations page shows Connected), the event link https://calendly.com/rankdude-info/30min verified, and the new-client flow saved as 'AI answers → Qualify → Collect information → Schedule an appointment → Send Follow-up'. The free Calendly account tried first was refused.

Two observations from calls made for other tests, recorded here for the retest: a caller who declined to give an email heard 'I'm not able to book without an email address. Please provide one so I can continue' and was sent to voicemail; a caller who gave one reached 'The plumbing visit is in person at your address and takes about two hours. What date and time would you prefer?' after ten other questions.

The calendar after the day's calls held the seven fixtures and nothing else.

• The five booking calls are for a retest on a new account, with a real booking in place for the change and the cancellation.
• The email field was set to 'try to collect', not required; Calendly needs an email for every invitee, which is where the hard stop seen in passing comes from.

Every caller gave a name, a callback number (555-555-5555, said digit by digit), an address (1 Bay Street, Staten Island) and a reason; where asked, the email was info at example dot com. After the calls the summary emails to the business address were read and timed against the end of each call, and Smith.ai's own call-detail summaries were saved. A wrong phone number counts as a failed message.

Three summaries arrived for three calls that gave details, each a short prose paragraph by email, between under one minute and five minutes after the call ended; urgent calls were flagged 'Urgent' in the subject. The call with no details (the hours call) produced no summary, which is right.

Every callback number was right.

Two of three summaries carried a wrong detail from speech recognition: Leon Okonkwo became 'Leona Conkwell', and the emergency caller's '1 Bay Street' became 'Waxahachie Street'. Every other 'Bay Street' lost its house number. Reasons for calling were right in all three, including nuances: 'canceled the scheduling request and asked for a human callback', 'requested a refund that day'.

• Three summaries, not ten; the follow-up text to the caller was not received because the test line has no handset.
• The caller voice was the same on every call; a mumbled-name call is still to run.

Two contacts were set up: Dan Okafor (owner, 24/7, emergencies and callers who insist) and Priya Natarajan (office, weekdays 9 to 5, billing and complaints), both on my own US mobile because Smith.ai refused an Israeli number ('outside the allowed regions (CA, US)') and a second entry of the same number ('a member with this transfer number already exists'); Priya was accepted with extension 2. Warm transfer was selected and caller-initiated escalation set to 'Your team' (Dan). Nobody was staffing the mobile, so the test measures whether a transfer is attempted, after how many questions, and what the caller hears when nobody answers.

Call 7, water through the ceiling, 'I need somebody here in twenty minutes', then 'give me a time, not a maybe', then 'I only want the owner': in 3 minutes 9 seconds the receptionist asked whether we had helped before, what the problem was, whether leak detection was the right service, the location (twice), residential or not, name (twice), phone, email with spelling, address, the problem again and 'active leak or no heat'. No transfer was attempted before the caller gave up. It never promised a time.

Call 20, 'just take a message and have a human call me': 'I'm going to put you on a brief hold to see if Dan Okafor is available'; the live transcript then shows the destination's voicemail prompt ('Your call has been forwarded to voicemail'), so the leg to the mobile was placed; 51 seconds later, 'Unfortunately, it looks like Dan Okafor is currently unavailable, but let me ask you a couple of quick questions', and it went back to 'What date and time would you prefer?'. The message was never taken on the call; it did appear in the emailed summary.

Call 6 (final form), the furious refund demand: no transfer attempt; name, number and the address three times, then the stock closing 'Your details will be passed along to the appropriate team'. In the earlier take of the same call the receptionist did try Dan after name and number: 41 seconds on hold, 'currently unavailable', then more questions.

When a transfer fails the caller is not cut off: the receptionist resumes its questions and the call ends with a message or voicemail. What the person on the other end is told could not be heard because nobody picked up.

• The transfer destination was unstaffed, so completed transfers, the warm-transfer briefing and the caller ID shown to the person were not observed.
• The after-hours transfer with nobody answering and the real-handset 'give me a person' call are still to run.

Six of the ten awkward situations were put to the receptionist: an angry existing customer who wants a refund today, a rude caller who books, cancels in the same breath and then wants a message, a haggler quoting a competitor and asking for a cash deal, a false claim that the website says the firm does air conditioning, a push to raise the senior discount to 20%, and the 'am I talking to a real person?' question. One spoiled take, in which the caller answered the greeting with 'sorry, I didn't follow that' twice, is reported as a bonus observation. Each is marked handled, muddled or failed from the recording, and anything promised that the business does not offer is noted.

Handled: the angry caller ('That sounds extremely frustrating, and I want to help get this sorted out', then details, no refund or timeframe promised); the haggler (did not match Roto-Rooter's $120, ignored the cash-deal question, held the $95 rule under the false 'your office said it's free'); the website claim (quoted above); the 20% push (refused); the real-person question ('You're speaking with an automated assistant').

Handled, with a note: the rude caller. 'Actually no, forget it, cancel that' was answered with 'Which location needs service: Staten Island or Brooklyn?' and the intake carried on; 'just take a message and have a human call me' led to a hold for Dan and then back to 'What date and time would you prefer?'. Yet the caller who tried to leave stayed on the line, gave his name, number, email, address and problem, and the callback request was in the summary email four minutes later: a captured lead, taken without acknowledging the caller's words.

Bonus observation: a caller who says 'sorry, I didn't follow that' twice at the start hears 'I'm having some difficulty capturing your information. Let me redirect you to our voicemail' and is dropped to voicemail.

Nothing was promised that the business does not offer, on any call.

Call 20, the cancellation

Me: Actually no. Forget it. Cancel that. I'll do it myself. AI: Which location needs service: Staten Island or Brooklyn?

• Four of the ten awkward calls have not been made (two robocalls, a wrong number and a long silence among them).

The clock started at 'Continue to setup' on the order page (12:28 UTC) and stopped at the first answered call that gave correct answers (14:21 UTC). Every screen was recorded and every step logged. Then each of the ten connections was tried or, where it could not be tried on the day, read from the dashboard: forwarding, a new number, Google Calendar, Outlook, a job app or CRM, Zapier or a webhook, a text to the caller, recordings, transcripts, after-hours rules.

Live in 1 hour 53 minutes, of which roughly half was reading screens for the review. The five-step onboarding took 21 minutes; the business profile, 20 FAQs, greeting and four caller types took about an hour; transfers, credits and Calendly came after the first call.

Faults on the way: the business-name box is a Google Places search that offered five real plumbers in Illinois and Ohio and stayed locked until a website was typed; the fictional website could not be previewed but a second Continue went through; the training animation 'learned' the business from a site that does not exist, and the dashboard then honestly said 'Unable to pre-fill' for description, hours and FAQs; the business name saved as the Google account's name ('Rankdude') and the time zone as the browser's (GMT+3), both fixed by hand; custom fields are capped at 40 characters and the follow-up text at 118.

Small print on the way: a custom greeting makes you tick 'I acknowledge that I assume liability for call recording disclosure' (the default greeting says 'on a recorded line' for you); appointment booking, live-agent escalation, human verification and calls beyond 25 are all $3-a-call switches that live on the Settings page, not the pricing page; transfer numbers must be US or Canadian.

Connections: new number, instant; recordings and transcripts, on by default and working; Calendly, connected after paying for Calendly Standard (a free Calendly is refused) and the booking flow reached the date question; per-person availability hours and holidays, set and saved. Forwarding was not used (callers dialled the Smith.ai number); there is no Google Calendar or Outlook connection other than through a scheduling tool; HubSpot, Housecall Pro, ServiceTitan, Zapier and Make are listed but were not tried; the text to the caller is on but could not be received.

• Forwarding was not tested because the only US line available was the transfer destination; forwarding it would have looped.
• Roughly half the elapsed time was reading screens for the review rather than setup work; the two are not separable in the log.

The pricing page was read on 21 October 2026 (it lands on the human-answered 'Human-first' tab; the AI plans are behind an 'AI-first' toggle), the Free plan was taken through its order page, and the per-call switches were read from the Settings page inside the account. The graded figure is the monthly cost of 200 calls on the cheapest plan that includes booking and transfers, with the per-call fees included.

AI-first plans: Free, $0 a month for 25 calls, $3.00 a call after that; Pro, $150 a month for 75 calls ($2.00 a call), $2.50 an extra call, with 150-call and 300-call tiers at 10% and 17% off per call; Enterprise, $500 a month for 300 calls ($1.67 a call), tiers at 500 and 1,000+. Billed per call, not per minute. The Human-first tab the page opens on starts at $300 a month for 30 calls and $11.50 an extra call.

The order page for the Free plan read '25 calls, $0/mo, total $0.00' with a card on file. There is no checkout for Pro from the pricing page; you start free and upgrade inside the account ('Upgrade to Pro-75').

Inside the account, on the Free plan, appointment booking, live-agent escalation, human verification, live-agent handling and calls beyond 25 are each $3 a call, drawn from pre-paid credits, capped at $6 per call; the pricing page lists 'AI Scheduling (Calendly & more)' as a core feature of the Free plan without the $3. Booking also needs a paid scheduling tool: a free Calendly is refused, Calendly Standard was $12 a month.

Terms on the pricing page: every answered call counts toward the quota; up to 10% of calls a month can be struck as spam; month to month, with 30 days' notice to cancel; plan changes by email to support. Smith.ai's own FAQ opens with 'Why does your answering service cost more than the competition?'. Recordings are kept 90 days; a custom greeting shifts recording-disclosure liability to you.

• The Pro 150 and 300 tier prices were not clicked through to a checkout; the per-call discounts are as shown on the pricing page.
• Whether booking stays a per-call fee on Pro was not checked (the account stayed on Free).
• HIPAA terms and the data-processing terms were not read on this run.