RankDude

How I test AI executive assistants

Nine tests, the same for every assistant, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, through one real week of executive admin, and the timings, screenshots and transcripts behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

Each assistant gets the same week in the same job. I connect it to a test executive's email, calendar and contacts, seed the inbox with the same 150 messages (25 of which need something done), and run the same five days: ten meetings to schedule with outside people across three time zones, five recorded meetings to take notes on, five morning briefings, 15 replies to draft from a written style sample, 20 commitments buried in mail and calls that must not slip, and a trip to arrange. What the assistant did is checked in the real calendar, the real inbox and the real notes, not in its own summary of what it did. Nothing is graded from a feature list or a demo.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 9 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (nine when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. A tool that only does one part of the job, such as scheduling, gets no grade on the tests it does not attempt rather than a 1, and the review says so.

The 9 tests

1.Inbox Triage

The 150 messages include 25 that need the executive to act (a board request, a contract to sign, an investor asking for a slot, a complaint from a customer) and 125 that do not (newsletters, receipts, cc threads, internal chatter). I ask what needs attention. A message counts as surfaced when it is named, flagged or placed on a to-do without me pointing at it; a wrong one counts as noise. New messages arrive each day and the count is repeated on day five.

  • 5: 23 to 25 surfaced, under 5 noise
  • 4: 20 to 22 surfaced, under 10 noise
  • 3: 15 to 19 surfaced, or 10 to 20 noise
  • 2: 9 to 14 surfaced, or more noise than hits
  • 1: 8 or fewer

2.Replies in My Voice

On day one the assistant gets a 2,000-word sample of the executive's past emails. Each of the 15 briefs names the recipient, the decision and two facts to include. A draft passes when it contains those facts, invents nothing, matches the sample's length and form of address, and could be sent as written. Drafts that needed a factual fix, a rewrite for tone, or that committed the executive to something not in the brief are counted separately.

  • 5: 13 to 15 pass
  • 4: 10 to 12
  • 3: 7 to 9
  • 2: 4 to 6, or an invented commitment
  • 1: 3 or fewer

3.Scheduling Across Time Zones

Ten scheduling jobs, each with a real counterpart I control in London, New York or Singapore: propose times, handle a counter-proposal, reschedule when the other side cancels, book a room or a video link, add travel time between two in-person meetings, protect a daily focus block, and resolve a double booking without me. A job counts when the event sits in both calendars at the right local time with the right people and link. The number of email rounds each took is reported.

  • 5: all ten
  • 4: 8 or 9
  • 3: 6 or 7
  • 2: 3 to 5, or a wrong time zone
  • 1: 2 or fewer

4.Meeting Notes & Actions

The five meetings are read from scripts by real people on a video call so every tool hears the same thing. Each script holds 20 checkable facts (figures, names, dates, decisions) and eight action items with an owner and a deadline. The notes and the action list are marked against the script. Facts stated wrongly and actions assigned to the wrong person are counted as errors, not misses.

  • 5: 92 or more facts and 36 or more actions, no wrong owners
  • 4: 85 to 91 facts, or 32 to 35 actions
  • 3: 70 to 84 facts, or 25 to 31 actions
  • 2: 50 to 69 facts, or a wrong figure in the summary
  • 1: under 50 facts

5.Daily Briefing & Meeting Prep

Each morning the assistant is asked for the day's briefing (or sends one unprompted, which is noted). It is marked on ten points: every meeting listed with the right time, travel time between in-person meetings, who each outside attendee is, the last contact with them, open items from the inbox, commitments due today, a conflict flagged, weather or flight status where relevant, nothing invented, and nothing missing that was in the calendar or inbox. Five meeting-prep notes for named meetings are marked on the same scale.

  • 5: 90 to 100 points
  • 4: 80 to 89
  • 3: 65 to 79
  • 2: 45 to 64, or an invented fact about a person
  • 1: under 45

6.Catching Commitments

Across the week 20 commitments are planted: a promise to send a document by Thursday made in a meeting, a reply the executive owes by Friday, an invoice due, a follow-up asked for in a cc thread. Each has a deadline. A commitment counts as caught when the assistant raises it, unprompted or in a briefing, before the deadline with the right owner and date. Ones it raised after the deadline or never are listed.

  • 5: 18 to 20 caught
  • 4: 15 to 17
  • 3: 11 to 14
  • 2: 6 to 10
  • 1: 5 or fewer

7.Approvals & Security

Tried in the product and read in the policy: asks before sending any email in the executive's name, asks before accepting or moving a meeting with an outside party, can be limited to named folders, labels or calendars, keeps a log of every action it took, supports single sign-on or two-factor sign-in, states whether customer data trains its models, states a retention period and a way to delete, publishes an independent security attestation (SOC 2 or ISO 27001), lets a human assistant be delegated the same access, and names the model providers that receive the mail. One point each.

  • 5: 9 or 10 points
  • 4: 7 or 8
  • 3: 5 or 6
  • 2: 3 or 4
  • 1: 2 or fewer, or it sent or accepted something without asking

8.Setup Time

The clock runs from the sign-up screen to the first correct briefing and the first meeting landed in the calendar, including every connection, permission prompt, style questionnaire, onboarding call or waitlist. Time waiting for a sales call or a manual onboarding is included. The number of settings I had to change to stop wrong behaviour in the first two days is counted.

  • 5: under 30 minutes
  • 4: 30 minutes to 2 hours
  • 3: 2 hours to 1 day
  • 2: 1 to 5 days
  • 1: over 5 days, or a sales call is required before anything works

9.Pricing & Value

Every plan is read from the checkout or billing page, not the marketing page. The figure graded is the yearly cost for one executive of the cheapest plan on which every test could be run, with credits, message caps and per-action charges worked out against the week's real usage. Renewal prices, minimum seats, onboarding fees and what the free tier can and cannot do are listed with the payment methods seen at checkout.

  • 5: under $600 a year, no cap hit
  • 4: $600 to $1,199 a year
  • 3: $1,200 to $2,999 a year, or a cap hit during the week
  • 2: $3,000 or more, or credits that cannot be worked out in advance
  • 1: poor value, or pricing only on a sales call

Documented but not graded

Travel booking, expense capture, phone calls made on the executive's behalf, voice, languages and whether the assistant works inside Slack, Teams, WhatsApp or iMessage are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real use of the product on a real account, with a real inbox, calendar, meetings and outside counterparts, never promotional descriptions or demo videos.
  • Identical week, seeded inbox, scripts, tests, thresholds and sample sizes for every assistant in the category.
  • Authorised accounts only; no password sharing, no bypassing of plan limits or bot checks, and no real third party contacted without their agreement.
  • Every grade has evidence: a timing, a screenshot, a transcript or a table, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI executive assistant for testing. No AI executive assistant has been through these tests yet; every one I follow is listed on the AI executive assistants page, and reviews appear there as tests finish.