How I test AI test automation tools

Nine tests, the same for every tool, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, against two applications I control, and the counts, run logs, screenshots and reports behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

The RankDude mascot holding a laptop with a check on its screen

Each tool is pointed at the same two applications: a small web shop (sign-up, search, basket, checkout, account pages) and a mobile app with the same features built for iOS and Android. Both applications have the same 20 written user journeys, the same ten bugs I planted on purpose and a second build with the same 15 deliberate changes to the screens (moved buttons, renamed labels, new layouts), so that self-healing can be measured rather than claimed. The counts are per application: the 20 journeys, ten bugs, 15 changes and 400 flakiness runs in the tests below are the web shop's, which set the grade, and the same work on the mobile app is reported alongside. I ask the tool to generate tests for the 20 journeys, run them in a real CI pipeline, point them at the broken build and the changed build, and count what happens. Nothing is graded from a feature list, a demo or a vendor-reported accuracy figure.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 9 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5	Excellent
4	Good
3	Average
2	Below average
1	Poor

The final score is the average of the graded tests (nine when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. A tool that only covers mobile is graded on the mobile app, with the same counts, and a tool that only covers the web is graded on the web shop alone; the review says so.

The 9 tests

1.Setup to First Passing Test

The clock starts at sign-up and stops when a test for the first journey (sign up, log in) has been created with the tool's AI and passes against the live web shop. Installs, browser extensions, agents, environment settings and any waiting for support count. The same is then timed for the mobile app and reported alongside.

5: under 30 minutes
4: 30 to 59 minutes
3: 1 to 3 hours
2: 3 to 24 hours
1: over 24 hours, or no test could be made to pass

2.Test Generation

Each journey is given to the tool the way it supports (a plain-English prompt, a recording, a URL to explore, a spec file). A generated test counts when it covers every step in the journey, checks the right outcome, and passes against the working build without me editing it. Tests that pass by checking nothing, skip steps or need a rewrite are counted separately, with the time spent fixing them.

5: 18 to 20 correct and passing first time
4: 14 to 17
3: 10 to 13
2: 5 to 9
1: 4 or fewer, or tests that pass without checking anything

3.Bug Detection & Failure Reports

The ten planted bugs include a wrong total at checkout, a broken link, a form that accepts bad input, a missing error message, a slow page and a visual fault. Each bug counts when a test fails because of it. Each failure report is then read: does it point to the step, the element and the reason, in words a developer could act on, or just say 'failed'? Screenshots, videos and traces attached to the report are noted.

5: 9 or 10 bugs caught, every report names the real cause
4: 7 or 8 caught, or two reports vague
3: 5 or 6 caught, or most reports vague
2: 3 or 4 caught
1: 2 or fewer caught, or failures blamed on the wrong cause

4.Self-Healing

The 15 changes are known in advance: moved buttons, renamed labels, changed IDs and classes, a new step in checkout, a redesigned menu. Each affected test is run. A repair counts when the test passes again, still checks what it checked before, and the tool shows me what it changed. A test that 'heals' by clicking the wrong thing or dropping an assertion counts as a wrong repair, which is worse than a failure.

5: 13 to 15 repaired, no wrong repairs
4: 10 to 12 repaired, or one wrong repair
3: 6 to 9 repaired
2: 3 to 5 repaired, or two or more wrong repairs
1: 2 or fewer, no self-healing, or silent repairs that cannot be reviewed

5.Flakiness

The suite is run 20 times against the unchanged working build, ten runs in the tool's cloud and ten from CI, at the tool's default settings. A result is inconsistent when the same test passes in one run and fails in another with nothing changed. Automatic retries are switched off for the count and reported separately. Median suite run time is recorded.

5: 0 or 1 inconsistent result in 400 runs
4: 2 to 4
3: 5 to 12
2: 13 to 40
1: over 40, or the suite could not finish 20 runs

6.CI & Workflow Integration

The ten: run the suite from GitHub Actions, run it from a second CI (GitLab or Jenkins), report a pass or fail status on a pull request, block a merge on failure, run tests in parallel and record the speed-up, trigger a run by API, post results to Slack, run against a preview or staging URL set per run, keep test data separate per run, and schedule a nightly run. Each counts when it works without a workaround or support.

5: 9 or 10
4: 7 or 8
3: 5 or 6
2: 3 or 4
1: 2 or fewer

7.Platform Coverage

Ten points, each tried rather than read off a feature page: Chrome, Firefox and Safari for the web shop; iOS and Android for the mobile app, on real devices or emulators the tool provides; an API test against the shop's backend; a visual comparison test; an accessibility check; a test with a file upload; and a test across two browser tabs or an email step (a sign-up confirmation). Each counts when a test of that kind runs and passes.

5: 9 or 10
4: 7 or 8
3: 5 or 6
2: 3 or 4
1: 2 or fewer

8.Export & Ownership

I try to export the 20 tests as Playwright, Selenium, Appium or similar code and run them on my own machine and in plain GitHub Actions without the tool. I export run history and reports. I read the terms on who owns the tests and the recordings of my application, and what happens to them when the plan is cancelled.

5: tests export as code that runs elsewhere, results exportable, clear ownership
4: code export that needs small edits to run elsewhere
3: partial export (steps as text, or results only)
2: copy and paste only
1: no export, or terms that claim rights over my tests or recordings

9.Pricing & Value

Every plan is read from the checkout or, where there is no public price, from the quote I ask for. The figure graded is the yearly cost of the cheapest plan that lets five users run the 20-test suite on web and mobile in CI, 20 times a day, in parallel. Charges per test run, per minute, per device, per user and per AI generation are listed with it, along with the renewal price against the first-year price and whether a price was published at all.

5: under $3,000 a year, published prices, same price at renewal
4: $3,000 to $7,999 a year, or under $3,000 with the price only from a quote
3: $8,000 to $19,999, or renewal more than 50% higher
2: $20,000 or more, or run-based charges that made the cost unpredictable
1: costs that were not shown before paying, or no price given even after a sales call

Documented but not graded

Test data generation, test case management, requirement linking, load testing, record-and-playback editors, IDE plugins and coding-agent integrations (MCP and command-line agents) are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

Real use of the product on a real account and real applications, never promotional descriptions or vendor-reported accuracy figures.
Identical applications, journeys, planted bugs, screen changes, tests, thresholds and sample sizes for every tool in the category.
Authorised accounts only; no password sharing, no bypassing of plan limits or run quotas, and tests run only against applications I own.
Every grade has evidence: a count, a run log, a screenshot, a report or a table, in the full test results.
A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI test automation tool for testing. No AI test automation tool has been through these tests yet; every one I follow is listed on the AI test automation tools page, and reviews appear there as tests finish.