RankDude

How I test AI workflow automation

Eight tests, the same for every platform, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, with the same five workflows running for real on the same test data for two weeks, and the timings, counts, run logs, screenshots and exports behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

Each platform builds the same five workflows for a fictional small company: lead intake (a form submission is enriched, classified by AI, added to a CRM and announced in Slack), invoice processing (a PDF arrives by email, AI extracts the fields, a person approves, the accounts sheet is updated), support triage (an inbox message is classified, a reply is drafted by AI, a person approves, the reply is sent), a weekly report (figures from three sources are pulled, summarised by AI and emailed), and a content feed (RSS items are summarised and posted to Slack and a spreadsheet). Each workflow gets the same 100 test items, so I know the right answer for every classification and every extracted field. Nothing is graded from a template gallery, a connector count on a pricing page or a demo.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 8 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (eight when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Some tests on a free plan cannot run the volume or add a second user; I test them again on a paid plan when I can.

The 8 tests

1.Setup to First Workflow

The clock starts at sign-up and stops when a real form submission has gone through the lead-intake workflow and appeared in the CRM and in Slack. Connecting the four apps, any verification, and the first AI step count. Where the platform can build the workflow from a plain-language description, I use that first and record whether the result ran without edits.

  • 5: under 30 minutes
  • 4: 30 to 59 minutes
  • 3: 1 to 3 hours
  • 2: 3 to 24 hours, or one app never connected
  • 1: over 24 hours, or the workflow never ran end to end

2.Builder & Logic

The twelve: a branch with three paths, a loop over a list of 100 items, a filter, a delay or wait until a date, a step that pauses for a human reply and resumes, a sub-workflow called from two others, a variable set in one step and used in another, a data transformation (dates and currency), a retry on a failed step, a scheduled trigger, a webhook trigger, and a code step. Each counts when done without a workaround or support. Generating a workflow from a written description is tried for all five and the number that ran without edits is reported alongside.

  • 5: all twelve
  • 4: 10 or 11
  • 3: 7 to 9
  • 2: 4 to 6
  • 1: 3 or fewer

3.AI Steps & Agents

Lead classification (100 leads into four categories) and invoice extraction (100 PDFs, five fields each) are scored against the answer sheet. The 100 support reply drafts are read and counted as usable when I would send them after a one-minute edit, with invented facts counted. Where the platform offers an agent that decides its own steps, the same support triage is given to it as a task and its decisions are counted the same way. Model choice, cost per run and the time per AI step are recorded.

  • 5: 95% or more right on both, 90 or more usable drafts, no invented facts
  • 4: 90% to 94% right, or 80 to 89 usable drafts
  • 3: 80% to 89% right, or 65 to 79 usable drafts
  • 2: 65% to 79% right, or fewer than 65 usable drafts
  • 1: under 65% right, or drafts that invented facts in 10 or more cases

4.Integrations

The twelve: a web form, Gmail or Outlook (read and send), Slack, a CRM (HubSpot), Google Sheets, a PDF attachment, an accounting tool (Xero or QuickBooks), an RSS feed, a REST API with an API key through a generic HTTP step, a webhook in and out, a database or table the platform hosts, and an MCP connection so an outside AI assistant can trigger a workflow. Each counts when data actually moved in the test. I note which needed a paid tier and which only worked through a generic step.

  • 5: 11 or 12 working
  • 4: 9 or 10
  • 3: 7 or 8
  • 2: 5 or 6
  • 1: 4 or fewer

5.Reliability

Over two weeks the five workflows receive their test items on a schedule, each of the 100 items going through its workflow once a week: about 1,000 runs in total. Each is checked at the destination (the CRM, the sheet, Slack, the sent folder). A run counts as failed when it stopped with an error, missed when the trigger never fired, duplicated when the destination got the item twice, and late when it finished more than 15 minutes after the trigger. Platform outages during the fortnight are noted with their timing.

  • 5: under 1% of runs failed, missed, duplicated or late
  • 4: 1% to 2%
  • 3: 2% to 5%
  • 2: 5% to 10%
  • 1: over 10%, or data lost

6.Monitoring & Error Handling

The six: an expired API key, a renamed spreadsheet column, a malformed PDF, a rate limit, a missing required field, and an AI step that times out. For each I record the time to an alert (email, Slack or in-app), whether the run log shows the failing step with its input and output, whether a retry or a replay from the failed step is offered, and whether the failed items were queued or dropped. Any self-repair the platform offers is tested on the same six and judged on whether its fix was right.

  • 5: all six alerted within 15 minutes, full logs, replay from the failed step, nothing dropped
  • 4: five alerted, or replay only from the start
  • 3: three or four alerted, or logs without inputs and outputs
  • 2: one or two alerted, or failed items dropped
  • 1: no alerts, or failures that were silently marked as successes

7.Human Approvals & Controls

The ten, each tried rather than read off a security page: an approval step that holds a run until a named person clicks, an approval by email or Slack without logging in, a second user with view-only access, a role that can run but not edit, an audit log showing who changed a workflow and when, version history with a rollback, separate credentials per user rather than shared, a choice of AI model or provider, a setting that keeps data out of model training, and an export of all workflows as files. Self-hosting and region options are noted but not graded.

  • 5: 9 or 10
  • 4: 7 or 8
  • 3: 5 or 6
  • 2: 3 or 4
  • 1: 2 or fewer

8.Pricing & Value

Every plan is read from the checkout, not the marketing page. The figure graded is the monthly cost of the cheapest plan that runs the five workflows at the fortnight's volume (about 2,000 runs a month, each with one or more AI steps) with two users and the integrations above. Charges per task, per operation, per run, per AI credit and per connector are listed, with the renewal price against the first-year price and the payment methods seen at checkout. Where AI steps use my own model key, that cost is added in.

  • 5: under $100 a month at the test volume, AI included, same price at renewal
  • 4: $100 to $249
  • 3: $250 to $599, or renewal more than 50% higher
  • 2: $600 or more, or costs that only became clear after the runs
  • 1: poor value, or no price without a sales call

Documented but not graded

Template libraries, prebuilt agents, browser automation, desktop RPA, document processing beyond the invoice test, self-hosting, enterprise single sign-on and community size are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real use of the product on a real account and real running workflows, never promotional descriptions, template counts or connector counts.
  • Identical workflows, test data, answer sheets, tests, thresholds and sample sizes for every platform in the category.
  • Authorised accounts only; no password sharing, no bypassing of plan limits or rate limits, and test data that belongs to no real person.
  • Every grade has evidence: a timing, a count, a run log, a screenshot or a table, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI workflow automation platform for testing. No AI workflow automation platform has been through these tests yet; every one I follow is listed on the AI workflow automation page, and reviews appear there as tests finish.