RankDude
Checker · AI agent

Output checker

Every task in the AI agent test pack has a checklist and an answer key written before any agent sees it. The output checker opens what the agent produced and marks each point passed, failed, or left for my eye.

When it runs
After every task
Used in
no published review yet
Runs on
My PC, reading the files the agent made

What the output checker is

An agent's own report of a task is not evidence. It says what it did; the checker says what is actually in the file. For each task the test world carries an answer key with the facts the output must contain, and the checker reads the agent's document, spreadsheet, slide deck, picture or reply and scores every point on the checklist.

Points with a fixed answer, such as a figure in a cell or a name in a paragraph, are marked passed or failed against the key. Points that need judgement, such as whether a picture suits a page, I look at myself, and they are never counted as passed on the key alone.

How it works

  1. It cuts the agent's final reply out of the saved conversation, from its last progress line to the point where the product declares the task complete.
  2. It groups the files the agent delivered by type and hands them to the task's own checking routine. Word, Excel and PowerPoint files are opened by reading their internal XML; PDF text is extracted; images are read pixel by pixel.
  3. Every checklist point gets one of three marks: passed, failed, or left for my eye when it needs judgement.
  4. The task's verdict follows a fixed rule: not checkable when nothing could be graded; finished when nothing failed; failed when nothing passed or more than half failed; partly done otherwise.
  5. It also scans the reply and the files for phone numbers, email addresses, web addresses and figures that exist nowhere in the test world, the prompts or the answer key, and lists them for me as possible inventions.
  6. It notes whether the product itself marked the task complete, which the false-done counter uses next.
  7. The results for every task are written into one report for the review, from which the task table, the task grid chart and the grades are built.

What it records

Per task
Task id and title, the files found and missing, the verdict, how many points passed, failed or were left for me, and whether the product marked the task complete.
Per point
The point's id, its mark and a line on what the checker found.
Scan
Contact details and figures in the output that exist nowhere in the test world, and how many characters the reply ran to.

What it cannot catch

  • A live web page or app the agent published is judged by eye: every point on those tasks is left for me.
  • Pictures, links that must be opened and page layout are eye points too.
  • The checker knows where each product saves its files through a map written for that product; a new agent needs its own map before the checker can find its output.
  • Figures the checker lists as unknown are candidates for a person to check, not verdicts.

Where it was used

Built and checked; the first review to use it will appear here when it is published.

The other checkers for ai agent

Written by Miles Foster, Owner and tester, RankDude. How RankDude tests and grades is on the methodology page.