RankDude

Chatbot test pack

Every chatbot is asked the same fixed set of prompts in the same order, so a result can be counted rather than admired.

Checker · AI chatbot
When it runs
At the start of every chatbot review
Used in
10 reviews
Runs on
Files in this site's repository (evaluation/kits/ai-chatbot), used on my PC

What the chatbot test pack is

The pack holds 30 fact questions, 15 reasoning problems, 10 coding tasks, 15 writing briefs with 50 countable constraints, a document of about 14,800 words with ten planted facts, a 25-turn memory conversation, 12 live-web questions, a multimodal set and 30 safety prompts. Each prompt has its answer or check written down in advance.

The first answer to each prompt is the one that counts. No prompt is edited, retried or picked from several tries.

How it works

  1. Open a fresh chat for each prompt and paste it exactly as written.
  2. Copy the first answer without tidying it.
  3. Mark it against the key and list every invented fact.

What it records

Per chatbot
A scoring sheet with one row per item, the answers, and a note of which model answered.

What it cannot catch

  • A free or work account may cap a model part way through, so a review says which model ran each block.
  • Voice and live-web items depend on what the account offers.

Where it was used

Every published review that ran this checker, with the figure it produced there, read from the review’s own data. Each figure opens the test results behind it.

Review

The other checkers for ai chatbot

Written by Miles Foster, Owner and tester, RankDude. How RankDude tests and grades is on the methodology page.