RankDude

How I test enterprise AI agents

Ten tests, the same for every platform, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, on the same three business workflows with the same documents, apps and people, and the timings, counts, screenshots and logs behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

Each platform is given the same small company to work for: a ten-person business with a Google Workspace, a Slack workspace, a HubSpot CRM, a Zendesk helpdesk, a QuickBooks file, a Jira board and a shared drive of 200 documents, including contracts, invoices and policies. It is asked to run the same three workflows: invoice intake and approval, new-hire onboarding, and a weekly sales report built from the CRM and a spreadsheet. Each workflow is run ten times with varied inputs, and I count what finished correctly without me stepping in. Nothing is graded from a feature list, an analyst report or a demo.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 10 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (ten when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. A platform sold only through a sales process is tested on whatever account the vendor provides, and the review says so.

The 10 tests

1.Setup to First Workflow

The clock starts when I create the account and stops when the invoice workflow completes its first real run end to end. I note whether a developer, a sales call, an onboarding session or an implementation partner was required, and how many of the three workflows could be built from templates rather than from scratch.

  • 5: under 2 hours, self-serve
  • 4: 2 to 8 hours, self-serve
  • 3: 1 to 3 days, or an onboarding session needed
  • 2: 1 to 4 weeks, or a developer needed
  • 1: more than a month, or no self-serve access at all

2.Business Connections

Email, calendar, the shared drive, Slack, the CRM, the helpdesk, the accounting file, a spreadsheet, the Jira board, a web browser, a custom API by webhook and an MCP server are each connected. A connection counts when the agent reads from it and makes a change in it that I can see in the app. A connection that reads but cannot act counts as half, and a half is rounded down in the total.

  • 5: all twelve
  • 4: 10 or 11
  • 3: 7 to 9
  • 2: 4 to 6
  • 1: 3 or fewer

3.Workflow Completion

Each workflow runs ten times with different inputs: ten invoices of varied layouts and currencies, ten new hires with different roles and start dates, ten weeks of CRM data. A run counts when every step was done and the output is right: the invoice approved by the right person and recorded, the hire's accounts and tasks created, the report's figures matching the source. A run that needed me to fix or finish it does not count.

  • 5: 28 to 30 runs
  • 4: 24 to 27
  • 3: 18 to 23
  • 2: 10 to 17
  • 1: 9 or fewer

4.Accuracy on Company Documents

The same fifty questions for every platform: contract dates and parties, invoice totals, policy rules, figures from reports and the answer to five questions the documents do not cover. An answer counts when it is right and cites the document. An answer that states a figure, clause or document that does not exist is counted as invented; the right answer to an uncovered question is to say so.

  • 5: 48 to 50 right, nothing invented
  • 4: 44 to 47 right, at most one invented answer
  • 3: 38 to 43 right, or two or three invented answers
  • 2: 30 to 37 right, or an invented figure or clause
  • 1: under 30 right

5.Permissions & Data Controls

Each control is tested, not read from the settings page: roles with different rights, connector access limited per agent, an agent unable to show a user a document that user cannot open, single sign-on, an approval step before any action that leaves the company's systems, a data retention setting, a written opt-out from model training, redaction of personal data in logs, revocation of a connector's token from the source app, and a kill switch that stops every agent at once. A control counts when I try to break it and it holds.

  • 5: all ten
  • 4: 8 or 9
  • 3: 6 or 7
  • 2: 3 to 5
  • 1: 2 or fewer, or the agent showed a document to a user without rights to it

6.Audit Trail

For each run I check whether I can see: a log entry with the time, the inputs and outputs, every tool call and what it changed, who approved any step, the model used, the cost of the run, an alert for the runs that failed, and an export of the whole log. Each item counts when it is there for all thirty runs; an item present for some runs counts as half, and a half is rounded down in the total.

  • 5: all eight
  • 4: 6 or 7
  • 3: 4 or 5
  • 2: 2 or 3
  • 1: 1 or none

7.Failure Handling

The ten faults: an expired connector token, an invoice with no total, a duplicate invoice, an app that is down, an instruction with two readings, a document in the wrong language, a start date in the past, a missing approver, a spreadsheet with a renamed column, and a request outside the agent's remit. A fault counts as handled when the agent stops, asks a person or escalates with a clear message. A silent wrong result fails the fault.

  • 5: all ten
  • 4: 8 or 9
  • 3: 6 or 7
  • 2: 3 to 5
  • 1: 2 or fewer, or a silently wrong payment or record

8.Administration

Add a user, give them a role, share an agent with a team, publish a new version of an agent and roll it back, change the model behind an agent, set a spending cap for a team, duplicate an agent from a template, and remove a leaver and everything they could reach. Each task counts when done without a support ticket; the total time is reported.

  • 5: all eight
  • 4: 6 or 7
  • 3: 4 or 5
  • 2: 2 or 3
  • 1: 1 or none

9.Deployment & Model Choice

Each point is confirmed in the product or in the contract offered, not on a sales slide: a cloud plan, a private cloud or VPC option, an on-premises or self-hosted option, a choice of at least three model providers, bringing my own model key, a choice of data region including the EU, and a way to run a workflow with no data leaving the company's own environment.

  • 5: all seven
  • 4: 5 or 6
  • 3: 3 or 4
  • 2: 2
  • 1: 1 or none

10.Pricing & Value

The figure graded is the yearly cost for ten users running the three workflows about 1,000 times a month: seats, platform fee, per-run or credit charges, model usage billed through the platform, premium connectors and the renewal price. Prices come from the checkout where there is one; where there is not, from a written quote, and the review says so. The cost of the test month, as actually billed, is reported alongside.

  • 5: under $6,000 a year, published prices
  • 4: $6,000 to $14,999 a year, or under $6,000 with the price only from a written quote
  • 3: $15,000 to $39,999 a year, or renewal more than 50% higher
  • 2: $40,000 or more a year
  • 1: poor value or hidden costs

Documented but not graded

Pre-built agent templates, voice channels, multi-agent orchestration, languages, documentation and community, and the quality of the platform's own suggestions are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real use of the product on a real account and real connected business apps, never promotional descriptions, analyst reports or reference customers.
  • Identical company, documents, workflows, faults, thresholds and sample sizes for every platform in the category.
  • Authorised accounts only; no password sharing, no bypassing of plan limits or security settings, and no real money moved.
  • Every grade has evidence: a timing, a count, a screenshot, a log or an export, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an enterprise AI agent for testing. No enterprise AI agent has been through these tests yet; every one I follow is listed on the Enterprise AI agents page, and reviews appear there as tests finish.