Assistant test pack
To compare assistants fairly, each one is asked the same things about the same invented person, and each answer is checked against an answer key I wrote before the test.
Checker · AI personal assistant
- When it runs
- At the start of every review
- Used in
- 1 review
- Runs on
- My PC
What the assistant test pack is
The person, the facts, the requests and the answer keys are one file in the methodology folder. The two supplied files are made by a script, so they are the same every time: a month of spending with a known total, and a building notice with ten planted facts.
How it works
- The facts are told once, in one chat; every later request is typed in a new chat without repeating them.
- Each answer is checked in the real app where there is one, such as the Automations list, and otherwise against the answer key.
- A request counts as done only when it needed no correction from me.
What it records
- Per request
- The request, the answer, what was checked and whether it was done.
What it cannot catch
- The names and numbers are invented, so an assistant cannot know them from anywhere else.
- A free plan with a daily message cap runs fewer than the full thirty requests; the review says how many.
Where it was used
Every published review that ran this checker, with the figure it produced there, read from the review’s own data. Each figure opens the test results behind it.
| Review | Figure | Tested |
|---|---|---|
| 7 of 7 requests done | Invalid Date |
Review
- Figure
- 7 of 7 requests done
- Tested
- Invalid Date
The other checkers for ai personal assistant
Written by Miles Foster, Owner and tester, RankDude. How RankDude tests and grades is on the methodology page.