RankDude

How I test vertical AI agents

Ten tests, the same for every agent, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, on a fixed pack of real tasks from the agent's own field, checked against an answer key written with a practitioner in that field, and the timings, counts, screenshots and transcripts behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

A vertical agent is built for one job, so it is tested on that job. For each field I follow (law, HR and recruiting, accounting and finance, healthcare, property, research, data and spreadsheets, hospitality, logistics, manufacturing, customer contact centres and software teams) I keep one task pack: 40 graded tasks, 30 knowledge questions, 20 end-to-end cases and 10 cases that must go to a person, written once with a practitioner who does that work for a living, with an answer key and a baseline of how long the practitioner takes. Every agent in a field gets the same pack. The packs are published with the first review in each field, so a reader can see what was asked. Nothing is graded from a feature list, a case study or a demo.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 10 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (ten when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Agents are compared within their field; a legal agent's grade and an accounting agent's grade are earned on different packs, and the review says which pack was used.

The 10 tests

1.Setup to First Real Task

The clock starts when I create the account and stops when the first task in the pack is completed with the field's documents or records loaded. I note whether a developer, a sales call, an onboarding session or an implementation partner was required, and whether the agent could read the field's standard files, contracts, CVs, ledgers, listings or papers, without conversion.

  • 5: under 1 hour, self-serve
  • 4: 1 to 4 hours, self-serve
  • 3: 1 to 3 days, or an onboarding session needed
  • 2: 1 to 4 weeks, or a developer needed
  • 1: more than a month, or no self-serve access at all

2.Systems of Record

Each field's pack names its eight systems: for law a document store, practice management, e-signature, email, calendar, billing, a court or registry lookup and Word; for recruiting the applicant tracking system, job boards, calendar, email, an assessment tool, HRIS, a messaging channel and a background-check service; and so on for each field. A connection counts when the agent reads from it and makes a change in it that I can see. Read-only counts as half, and a half is rounded down in the total.

  • 5: all eight
  • 4: 6 or 7
  • 3: 4 or 5
  • 2: 2 or 3
  • 1: 1 or none

3.Accuracy on the Field's Tasks

The forty tasks are the field's bread and butter: clauses to find and flag, CVs to screen against a job spec, transactions to reconcile, listings to price, papers to summarise, formulas to write, shipments to re-route. Each is marked right or wrong against the key by me and checked by the practitioner. Any citation, case, figure, candidate, transaction or paper that does not exist is counted as invented, and one invented item caps the grade at 3.

  • 5: 38 to 40 right, nothing invented
  • 4: 34 to 37 right, nothing invented
  • 3: 28 to 33 right, or any invented item
  • 2: 20 to 27 right
  • 1: under 20 right

4.Domain Knowledge

The questions are written by the practitioner: definitions, thresholds, deadlines, which form, which standard, which jurisdiction. Five ask about a rule that changed in the last two years, where the old answer is wrong. An answer counts when it is right for the jurisdiction and date I give. Whether the agent cites a source for each answer is recorded.

  • 5: 28 to 30 right, all five changed rules right
  • 4: 24 to 27 right
  • 3: 18 to 23 right, or two or more changed rules wrong
  • 2: 12 to 17 right
  • 1: under 12 right

5.Finished Without a Person

Each case is a whole job from the field: a candidate screened, scheduled and messaged; an invoice matched, coded and posted; a contract reviewed, redlined and sent for signature; a lead qualified and booked. A case counts when every step was done, the result matches the key, and the record in the connected system is right. A case I had to finish or fix does not count.

  • 5: 18 to 20 cases
  • 4: 14 to 17
  • 3: 10 to 13
  • 2: 5 to 9
  • 1: 4 or fewer

6.Escalation & Handoff

The ten include an ambiguous instruction, a conflict of interest, a request outside the agent's remit, a figure over an approval threshold, a candidate who discloses a protected characteristic, a client who mentions a complaint, and a document the agent cannot read. A case counts when the agent stops, hands over to a named person or queue, and the handoff carries the file, the history and what the agent had already done. A case the agent completed anyway fails.

  • 5: all ten
  • 4: 8 or 9
  • 3: 6 or 7
  • 2: 3 to 5
  • 1: 2 or fewer, or the agent acted on a case it should have escalated

7.Compliance Controls

Each field's pack names its eight controls; most packs share a core: role-based access, a data retention setting, a written opt-out from model training, redaction of personal or patient data in logs and outputs, consent capture where the field needs it, an export of the records the agent created, a data region choice, and the field-specific agreement offered in writing, such as a business associate agreement in healthcare or a data processing agreement in Europe. A control counts when I try it and it works.

  • 5: all eight
  • 4: 6 or 7
  • 3: 4 or 5
  • 2: 2 or 3
  • 1: 1 or none, or personal data shown to a user without rights to it

8.Speed Against a Practitioner

The practitioner timed themselves on twenty of the graded tasks when the pack was written. I time the agent on the same twenty, including the time I spend checking and correcting its output, because an answer that needs re-doing saved nothing. The grade is the agent's total time as a share of the practitioner's total. Only tasks the agent got right are counted in both totals.

  • 5: under 25% of the practitioner's time
  • 4: 25% to 49%
  • 3: 50% to 74%
  • 2: 75% to 99%
  • 1: as long as the practitioner or longer

9.Audit & Explainability

For each graded task and case I check whether I can see: the source the answer came from, the steps the agent took, a log of each action with a time, who approved any step, an export of the log, and a way to undo an action the agent took in a connected system. Each item counts when it is there for every task; an item present for some tasks counts as half, and a half is rounded down in the total.

  • 5: all six
  • 4: 5
  • 3: 3 or 4
  • 2: 2
  • 1: 1 or none

10.Pricing & Value

The figure graded is the yearly cost for five practitioners doing the pack's volume of work each month: seats, platform fee, per-document, per-case or per-hire charges, usage credits and the renewal price. Prices come from the checkout where there is one; where there is not, from a written quote, and the review says so. The cost of the test month, as actually billed, is reported alongside.

  • 5: under $3,000 a year, published prices
  • 4: $3,000 to $7,499 a year, or under $3,000 with the price only from a written quote
  • 3: $7,500 to $19,999 a year, or renewal more than 50% higher
  • 2: $20,000 or more a year
  • 1: poor value or hidden costs

Documented but not graded

Languages and jurisdictions covered, templates, mobile apps, voice, how the agent improves after corrections and the quality of its own suggestions are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real use of the product on a real account with the field's real documents and records, never promotional descriptions, case studies or analyst quotes.
  • Identical task pack, answer key, baseline, thresholds and sample sizes for every agent in the same field.
  • Authorised accounts only; no password sharing, no bypassing of plan limits, and no real client, patient, candidate or customer data used.
  • Every grade has evidence: a timing, a count, a screenshot, a log or a transcript, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit a vertical AI agent for testing. No vertical AI agent has been through these tests yet; every one I follow is listed on the Vertical AI agents page, and reviews appear there as tests finish.