Chatbot test pack
Every chatbot is asked the same fixed set of prompts in the same order, so a result can be counted rather than admired.
- When it runs
- At the start of every chatbot review
- Used in
- 10 reviews
- Runs on
- Files in this site's repository (evaluation/kits/ai-chatbot), used on my PC
What the chatbot test pack is
The pack holds 30 fact questions, 15 reasoning problems, 10 coding tasks, 15 writing briefs with 50 countable constraints, a document of about 14,800 words with ten planted facts, a 25-turn memory conversation, 12 live-web questions, a multimodal set and 30 safety prompts. Each prompt has its answer or check written down in advance.
The first answer to each prompt is the one that counts. No prompt is edited, retried or picked from several tries.
How it works
- Open a fresh chat for each prompt and paste it exactly as written.
- Copy the first answer without tidying it.
- Mark it against the key and list every invented fact.
What it records
- Per chatbot
- A scoring sheet with one row per item, the answers, and a note of which model answered.
What it cannot catch
- A free or work account may cap a model part way through, so a review says which model ran each block.
- Voice and live-web items depend on what the account offers.
Where it was used
Every published review that ran this checker, with the figure it produced there, read from the review’s own data. Each figure opens the test results behind it.
| Review | Figure | Tested |
|---|---|---|
| 30 of 30 facts right | Invalid Date | |
| 30 of 30 facts right | Invalid Date | |
| 30 of 30 facts right | Invalid Date | |
| 30 of 30 facts right | Invalid Date | |
| 30 of 30 facts right | Invalid Date | |
| 10 of 10 facts right | Invalid Date | |
| 30 of 30 facts right | Invalid Date | |
| 30 of 30 facts right | Invalid Date | |
| 29 of 29 answered facts right | Invalid Date | |
| 29 of 29 facts right | Invalid Date |
Review
- Figure
- 30 of 30 facts right
- Tested
- Invalid Date
- Figure
- 30 of 30 facts right
- Tested
- Invalid Date
- Figure
- 30 of 30 facts right
- Tested
- Invalid Date
- Figure
- 30 of 30 facts right
- Tested
- Invalid Date
- Figure
- 30 of 30 facts right
- Tested
- Invalid Date
- Figure
- 10 of 10 facts right
- Tested
- Invalid Date
- Figure
- 30 of 30 facts right
- Tested
- Invalid Date
- Figure
- 30 of 30 facts right
- Tested
- Invalid Date
- Tested
- Invalid Date
- Figure
- 29 of 29 facts right
- Tested
- Invalid Date
The other checkers for ai chatbot
Written by Miles Foster, Owner and tester, RankDude. How RankDude tests and grades is on the methodology page.