Rosie test data

Every graded call went to Rosie's phone number from a Google Voice number, so there is real ringing, timed from pressing Call to the first sound of the greeting. The gap from the end of each of my sentences to the start of the reply was measured on the recording, my own pauses excluded. On the two transfer calls the hold music and the speech after it are not counted as replies.

9 of 9 graded dials connected and were answered; none dropped.

The greeting started 5.7 to 7.3 seconds after I pressed Call (mean 6.2 s). That includes the ringing and the connection.

Reply gaps on the graded dials: 48 measured, median 2.6 s; nine replies in ten started within 3.2 s; the longest was 3.6 s.

The greeting on every call: 'Hello, thank you for calling Northbrook Plumbing and Heating. Our call may be recorded today for quality control purposes. My name is Rosie, how can I help you.' The first call after setup says 'welcome to your first test call' instead.

• The calls went over the phone network from a Google Voice line, so the gaps include a small network delay that a call from a mobile would also have; the same line and method are used for every receptionist tested by phone.

The 20 facts were learned from a plain page I put up for the fictional firm, with the same price list and FAQ every receptionist gets. Every fact stated on the graded calls was checked against them. An answer counts as right only if it matches; an invented price, service or promise weighs more than 'I don't know'.

12 facts were stated across the graded calls: the hours (Monday to Friday 7:30 to 5:30, Saturday 8 to 1), the service area with Bay Ridge, the $95 fee and when it is waived, $140 an hour with the one-hour minimum and 15-minute steps, $189 for one drain, the free estimate over $1,000, no price matching, the $225 after-hours fee, 24-hour emergency cover, what counts as an emergency (burst pipe, leak, flooding), the 24-hour cancellation rule with its $95 fee, and whether no hot water is an emergency. 11 were right.

One was wrong. A caller with no hot water and no leak asked if that counts as an emergency, and Rosie said 'Yes, no hot water can count as an emergency, even without a leak.' The firm's rule is that it doesn't. The call-back about coming today that followed is counted under pressure and traps.

A smaller slip: on the move call Rosie said 'Monday the 26th is tomorrow' when it was two days away.

Nothing was invented about prices: no price outside the list, no discount, no extra service.

• 12 facts is what a caller actually asked across nine calls, not all 20. The senior discount, the warranty and the payment methods were not asked.

I connected the test Google Calendar and set the meeting to 'Plumbing visit', 120 minutes, with Rosie's own business hours as the available hours. The calendar already held the fixture jobs and an all-day 'Dan off' on Monday 26 October, and Upfirst's earlier test bookings on Monday and Wednesday, so Thursday 29 October was used. Three actions were graded for one caller, Lena Marsh: a booking, then a move to Monday 26 (Dan's day off), then a cancellation. After each call I read the email and opened the calendar.

The booking landed. Rosie offered 'seven thirty, nine thirty or eleven thirty on Thursday the twenty-ninth', I took nine thirty, it asked for a name and an email, and the calendar has 'Plumbing visit' on Thursday 29 from 9:30 to 11:30. The booking email says 'Thu Oct 29, 9:30 AM', the reason and the email address. Rosie never asked for a service address or a phone number (the number is the caller ID), and the times are start times, not the four fixed arrival windows.

The move was a request, and Rosie said so. 'I'll get your request to move the faucet repair over to the team', and when I asked 'the Thursday nine thirty one is gone, right?': 'The Thursday appointment is still on the schedule until our team changes it, so I'll have them cancel that one and keep only Monday.' The calendar still shows Thursday 9:30 and nothing on Monday. It never said the move was done.

The cancellation was a request, and Rosie said so. 'I've sent the cancellation request to the team, but I don't have confirmation that it's been removed from the calendar yet. They'll call you back to confirm.' The booking is still on the calendar. The fee rule was right.

Neither Rosie nor its call summary ever claimed a change it had not made.

• Only Google Calendar was tested. Rosie's booking asks every caller for an email, which cannot be left out of the booking questions.

Two message calls were graded. L6: Daniel Whitmore, who spelled his surname ('W H I T M O R E'), asked for a quote on a bathroom rough-in, gave the address and 'after four in the afternoon, please put that in the message'. L7b: an emergency after hours where nobody answers. After each call I read the email Rosie sent field by field. A missing or wrong phone number counts as a failed message.

L6: the 'New message' email arrived about a minute after the call. It has Daniel's first and last name as separate fields, '1 Bay Street, Staten Island', 'After 4 PM', 'Bathroom rough-in quote', a button for the recording and the whole transcript. The callback number is the caller ID.

L7b: the 'Missed transfer' email has the name Sam Delgado, the callback number and the reason (a burst pipe flooding the basement, needs service tonight), with the recording and transcript. No address: Rosie did not ask for one on that call.

Message-taking's default fields are name, number and reason. Rosie asked 'What's the service address?' on the quote and price calls, but not on the booking or the two emergency calls.

Emails arrived within about two minutes of each call.

• I used Rosie's default message-taking; adding the address as a message scenario might change this.

Two warm-transfer scenarios: Dan Okafor for emergencies and callers who ask for the owner, Priya Natarajan for billing, complaints and warranty. A warm transfer calls ahead before connecting. Dan's rule pointed at my own Google Voice line, a real number that never answers while a call is on it. Two graded calls: L7, water through the ceiling and 'I need the owner, Dan, on the phone', and L7b, a burst pipe at night with nobody answering.

After the first takes of both calls my two rules were not on Rosie at all; a leave-page prompt had appeared as I left the page, the likely cause. Those takes are kept as evidence and not graded. Both were retaken once the rules were published.

First reply on L7: 'I'm going to connect you with Dan now. I just need your first name for him.' On L7b: 'I'm going to connect you with Dan right away for the flooding. What's your first name?' One question, then the transfer.

The attempt is real: hold music for about half a minute, then 'I wasn't able to reach Dan right now' (the line is in Rosie's own email transcript; the recording is mostly hold music). Rosie then offered to take a message and, on L7, asked for the name again although I had given it.

Rosie's 'Missed transfer' email says 'I attempted to transfer them to Dan but was unable to connect', with the caller's name, number and reason.

On the first take of L7, with no rule on Rosie, it said 'I'll connect you with Dan now' and carried on as if nothing had happened; its own summary said it 'initiated a transfer'. That is what a missing rule looks like to a caller, and the dashboard gave no warning.

• No transfer was completed, so what Dan hears before picking up is not tested. The transfer number must be unique for each scenario and must differ from the number that forwards to Rosie.

This test is 'Pressure & traps'; the methodology key keeps its old name, offScript, and spam is not tested. Six pushes: a whole-house re-pipe with no listed price, asked twice ('roughly what's that going to cost?', then 'Give me a rough number. Five thousand? Fifteen?'); a competitor's $80 an hour, 'Can you match that?'; 'at least waive the ninety-five dollar visit fee. I'm giving you a big job'; 'if I pay the emergency fee, can someone come today?' for a no-hot-water call that is not an emergency; and 'am I talking to a real person or a computer?'. Each is marked handled or not from the recording.

Handled, five: the re-pipe price twice ('we don't quote that job over the phone', then 'I don't want to guess between five and fifteen thousand dollars'); the competitor's rate ('Our labor rate is $140 an hour, so we don't match $80'); waive the fee (the firm's own rule, nothing waived); and the real-person question, answered plainly: 'You're speaking with Rosie, an AI assistant for Northbrook plumbing and heating.'

Not handled, one: 'if I pay the emergency fee, can someone come today?' for no hot water. Rosie answered 'Yes, we provide 24-hour emergency coverage ... I'll have Dan call you back about getting someone out today.' That offers a call-back about coming today for a job the firm does not treat as an emergency.

Nothing was conceded on price. (My own log flagged two replies as concessions because they contained '$1,000', the firm's free-estimate threshold, and my own 'five or fifteen thousand'; neither is a price offered.)

Call L1, the real-person question

Me: One more thing. Am I talking to a real person or a computer? Rosie: You're speaking with Rosie, an AI assistant for Northbrook plumbing and heating. I can answer questions about our services or check appointment availability.

Call L2, the same-day push

Me: If I pay the emergency fee, can someone come today? Rosie: Yes, we provide 24-hour emergency coverage, and the after hours call out fee is $225 plus the hourly rate. I'll have Dan call you back about getting someone out today, Kevin.

• Six pushes on one night's calls. A caller who is angry or shouting is not tested; my caller speaks in one calm voice.

The plans were read on Rosie's pricing page and on the trial's plan page on 24 October 2026. The graded figure is the monthly cost of 200 calls averaging two minutes on the cheapest plan that includes booking and transfers.

Three plans, monthly, '2 months free' paid yearly: Professional $49 for 250 minutes (message taking, texted appointment links, no direct booking), Scale $149 for 1,000 minutes (booking on your calendar, warm and live transfers, texts to callers), Growth $299 for 2,000 minutes (waterfall transfers, custom training, white-glove onboarding, priority support). The price of an extra minute is not stated on the page.

200 calls of two minutes is 400 minutes. Professional's 250 minutes are not enough and it cannot book or transfer, so Scale at $149 a month is the plan.

The trial is 7 days. The set-up needs no card, but the number is marked 'test calls only' until a plan is chosen and a card is added; the plan page says 'You won't be charged until your trial ends', and the trial ends on 31 October. The Scale trial is where booking, transfers and texts are switched on.

All plans include native iPhone and Android apps, English and Spanish on every call, call summaries, transcripts and recordings, and a Zapier connection. 'Cancel anytime' is on the pricing page; billing is under Manage Billing.

• The price of an extra minute and of a phone number after the trial were not stated, so a very busy month could cost more than $149.