RankDude
Checker · AI agent

False-done counter

The most expensive failure in an AI agent is a task it reports as finished that is not. This counter finds every task the agent called done while a checklist point had failed, and separates the ones where the agent admitted the gap from the ones where it kept quiet.

When it runs
After the output checker, once per review
Used in
no published review yet
Runs on
The output checker's results

What the false-done counter is

A partly finished task that says so is a delay. A partly finished task that says it is done is a trap, because you will not check it. The accuracy grade in an AI agent review rests on how often that happened, so the count comes from the checker's marks, not from my impression.

How it works

  1. It runs the output checker over every scored task and takes each task's failed points.
  2. A task counts as claimed done when the product marked it complete, or when the agent's reply is a delivery rather than an apology or a refusal.
  3. Each failed point is tested against a short list of admission wordings for that point. A failure the reply admits is not a false done.
  4. A task is a false done when it was claimed done and at least one failed point went unadmitted. A task the checker could not grade is left out rather than counted either way.
  5. It compares its verdict with the flag I set in the task ledger during the test, so a disagreement between the program and my notes is visible.
  6. It also counts invented contact details found by the output checker's scan, since a made-up phone number in a delivered file is a false done of its own kind.

What it records

Per review
How many tasks were checked, how many could not be, the number of false dones and which tasks they were, and the number of invented contact details.
Per task
The checker's verdict, whether the task was claimed done, the undisclosed and the disclosed failures, the ledger's flag and whether the two agree.

What it cannot catch

  • Admissions are found by matching wordings I listed for each point. An admission phrased in a way the list does not cover counts as silence until I add it.
  • Tasks the output checker leaves to my eye, such as a published web page, are not counted.

Where it was used

Built and checked; the first review to use it will appear here when it is published.

The other checkers for ai agent

Written by Miles Foster, Owner and tester, RankDude. How RankDude tests and grades is on the methodology page.