How I test AI personal assistants

Ten tests, the same for every assistant, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, through one real week of personal admin, and the timings, screenshots and transcripts behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

The RankDude mascot holding a laptop with a check on its screen

Each assistant gets the same week. I connect it to the same test email account, calendar and task list, seed the inbox with the same 100 messages, tell it the same 20 facts about me on day one, and then give it the same 30 everyday requests over seven days: reminders, a flight to find, a document to summarise, a reply to draft, a dinner to book, a trip to plan, a subscription to cancel. Ten of the requests need the calendar, ten need email, ten need the web or another app. Every request is typed once in plain words, and what the assistant did is checked in the real calendar, the real inbox or the real booking, not in its own report of what it did. Nothing is graded from a feature list or a demo.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 10 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5	Excellent
4	Good
3	Average
2	Below average
1	Poor

The final score is the average of the graded tests (ten when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. An assistant that cannot connect to email or a calendar at all gets no grade on those tests rather than a 1, and the review says so.

The 10 tests

1.Everyday Tasks Done

Each request is typed once, in plain words, with any detail it needs (a date, a budget, a name). It counts as done when the outcome is correct where it matters: the reminder fires at the right time, the booking exists with the right details, the summary contains the ten facts I planted in the document, the cancellation went through. A task that needed a second instruction to fix a mistake counts as not done; the number of clarifying questions the assistant asked first is reported alongside.

5: 27 to 30 done
4: 23 to 26
3: 17 to 22
2: 10 to 16
1: 9 or fewer

2.Calendar Handling

The ten calendar requests from the week: book a meeting from a free slot, move one, find a time that works across three time zones with a friend's shared calendar, set a recurring event, protect two hours for a task, add travel time, resolve a double booking, cancel with a note, invite three people by email, and answer what is on next Tuesday. Each is checked in the calendar itself: right day, time, zone, attendees and title. One point each.

5: all ten
4: 8 or 9
3: 6 or 7
2: 3 to 5
1: 2 or fewer

3.Email Triage & Drafts

The 100 seeded messages include 20 that need an action (an invoice, a question from a friend, a delivery problem, a school form) and 80 that do not. I ask the assistant what needs attention; a message counts as surfaced when it is named or placed in a to-do without me pointing at it, and a wrong one counts as noise. Then I ask for ten replies on fixed briefs; a draft passes when it contains the facts I gave, invents nothing, and could be sent as written. The grade combines the two counts.

5: 18 to 20 surfaced with under 5 noise, and 8 to 10 drafts pass
4: 15 to 17 surfaced, or 6 or 7 drafts pass
3: 11 to 14 surfaced, or 4 or 5 drafts pass
2: 6 to 10 surfaced, or invented facts in a draft
1: 5 or fewer surfaced

4.Memory

On day one I tell the assistant 20 facts in conversation: my dentist's name, a partner's birthday, a dietary restriction, the car's registration, a preferred airline seat, a working-hours rule. From day two to day seven the requests are written so that each of the 20 facts is needed once; the assistant is not reminded. A fact counts when the task used it correctly without being prompted. Facts it had to be told again are listed.

5: 18 to 20 used
4: 15 to 17
3: 11 to 14
2: 6 to 10
1: 5 or fewer, or no memory across sessions

5.Proactive Help

Every unprompted message, suggestion or action across the week is logged with a timestamp and sorted: useful (something I needed and would otherwise have missed, such as a flight delay, a bill due tomorrow, a conflict it spotted) or noise (a tip, a nudge to upgrade, a reminder I did not want, a wrong alert). The week contains eight planted moments where a good assistant should speak up. The grade is the useful count with the noise share as a cap.

5: 5 or more useful, noise under 20% of all unprompted messages
4: 3 or 4 useful, noise under 30%
3: 1 or 2 useful, or noise 30% to 50%
2: nothing useful but little noise
1: nothing useful and noise over half, or marketing nudges only

6.Connected Apps

Each connection is set up and then used for a task: Gmail, Outlook, Google Calendar, Apple or Outlook calendar, a task app (Todoist or Reminders), Notion or Drive files, Slack, WhatsApp or iMessage as a channel, a browser for a web task, a phone call made on my behalf, a payment with a saved card, and a smart-home or music action. A point counts when the task completed through that connection.

5: all twelve
4: 10 or 11
3: 7 to 9
2: 4 to 6
1: 3 or fewer

7.Speed

A screen recording runs through the week. For each of the 30 requests the clock starts when I send it and stops when the result is complete and correct on screen or in the real app, not when the assistant starts replying. Tasks that need an outside party (a restaurant to confirm) are timed to the assistant's own hand-off. The median is the grade; the slowest task is reported.

5: median under 30 seconds
4: 30 seconds to 2 minutes
3: 2 to 10 minutes
2: 10 to 60 minutes
1: over an hour, or tasks left unfinished

8.Permissions & Privacy

Tried in the product and read in the policy: asks before sending an email in my name, asks before paying, asks before deleting or cancelling anything, lets me limit which inboxes, folders or calendars it reads, states whether my data trains its models, lets me delete my data and account from the app, offers a data export, keeps a log of actions it took, supports two-factor sign-in, and names the model providers that receive my messages. One point each.

5: 9 or 10 points
4: 7 or 8
3: 5 or 6
2: 3 or 4
1: 2 or fewer, or it sent, paid or deleted without asking

9.Setup to First Task

The clock runs from the sign-up screen to the first request done, including every app connection the first request needs, any waitlist or invite, and any style or preference questionnaire. The number of permission prompts and settings changed is counted. A waitlist is timed for real; a vendor-provided account skips it and the review says so.

5: under 10 minutes
4: 10 to 29 minutes
3: 30 minutes to 4 hours
2: 4 hours to 3 days
1: over 3 days, or no access without an invite

10.Pricing & Value

Every plan is read from the checkout or billing page, not the marketing page. The figure graded is the yearly cost of the cheapest plan on which all 30 requests and the required connections work, with credit or message caps worked out against the week's real usage. Renewal prices, overage charges, and what the free tier can and cannot do are listed with the payment methods seen at checkout.

5: under $120 a year or free, no cap hit during the week
4: $120 to $239 a year
3: $240 to $479 a year, or a cap hit during the week
2: $480 or more, or credits that cannot be worked out in advance
1: poor value or costs not disclosed before signing

Documented but not graded

Voice quality, languages, family or shared accounts, desktop and mobile app quality, offline or on-device options and open-source self-hosting are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

Real use of the product on a real account, with a real inbox, calendar and bookings, never promotional descriptions or demo videos.
Identical week, requests, seeded inbox, tests, thresholds and sample sizes for every assistant in the category.
Authorised accounts only; no password sharing, no bypassing of plan limits or bot checks, and bookings and payments made for real at the lowest value that fits the task and cancelled or refunded.
Every grade has evidence: a timing, a screenshot, a transcript or a table, in the full test results.
A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI personal assistant for testing. No AI personal assistant has been through these tests yet; every one I follow is listed on the AI personal assistants page, and reviews appear there as tests finish.