RankDude

How I test AI document automation

Ten tests, the same for every tool, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, the same stack of documents goes through every tool, and the accuracy tables, timings, screenshots and checklists behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

Each tool gets the same 200 documents: 100 supplier invoices, 50 receipts, 25 contracts and 25 filled-in forms, in PDF, scanned and phone-photo versions, in five languages, with 20 fields marked up by hand for every document. The same tools also get the same generation job: 20 documents to produce from a template and a spreadsheet. I upload, extract, review, approve, generate, export and read the terms. Nothing is graded from a vendor's accuracy claim or an analyst chart.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 10 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (ten when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. A tool built only for extraction, only for generation or only for e-signature gets no grade on the tests it cannot take, and the review says so. Some tests on a free plan can't connect accounting software; I test them again on a paid plan when I can.

The 10 tests

1.Setup to First Extraction

The clock starts at account creation and stops when ten of the invoices have been uploaded and their 20 fields come out right, with any template or model set up on the way. I note whether the tool worked on the first upload, needed examples or rules, needed an API key and code, or needed a sales call to get access.

  • 5: under 30 minutes, no code
  • 4: under 2 hours
  • 3: under a day, or code needed
  • 2: over a day, or a sales call needed for access
  • 1: could not be done

2.Extraction Accuracy

Every document has 20 fields marked up by hand: supplier, dates, numbers, totals, tax, line items, parties, terms, signatures and so on. The tool's output is compared field by field with the mark-up: 4,000 fields in all. A field is right only when the value matches exactly after trimming spaces and formatting dates and amounts the same way. Missing fields and extra fields both count as wrong. Accuracy is reported per document type as well as overall.

  • 5: 98% or more right
  • 4: 95 to 97.9%
  • 3: 90 to 94.9%
  • 2: 80 to 89.9%
  • 1: under 80%

3.Scans, Photos & Languages

Forty of the 200 are the hard set: ten phone photos taken at an angle, ten with handwritten fields, ten faint or skewed scans, and ten in languages other than English. The same field-by-field check is run on these 800 fields and graded on its own. I note which failures were the image and which were the language.

  • 5: 95% or more right
  • 4: 90 to 94.9%
  • 3: 80 to 89.9%
  • 2: 65 to 79.9%
  • 1: under 65%

4.Touchless Rate & Confidence

A document is touchless when the tool marks every field as confident and every field is right. I count the touchless share and, separately, the fields the tool marked confident that were wrong (false confidence) and the right fields it sent for review (needless review). A tool with no confidence scores or review queue is graded on touchless share alone and capped at 3.

  • 5: 80% or more touchless, 0 false-confident fields
  • 4: 70 to 79%, or up to 5 false-confident fields
  • 3: 55 to 69%, or 6 to 20 false-confident fields
  • 2: 40 to 54%, or over 20 false-confident fields
  • 1: under 40%

5.Speed

The 200 documents are uploaded as one batch and the clock runs from the end of the upload to the last document ready for review; the total divided by 200 is the figure graded. Ten single documents are also timed one at a time from upload to result, and the median is reported alongside. Queue waits and rate limits count.

  • 5: under 10 seconds per document
  • 4: 10 to 29 seconds
  • 3: 30 to 89 seconds
  • 2: 90 seconds to 5 minutes
  • 1: over 5 minutes, or the batch failed

6.Approvals & Matching

Each point is tried for real: a review screen that shows the field beside the source image, a routing rule that sends invoices over $1,000 to a second approver, approval from a phone, a two-way match against a purchase order, a three-way match with a goods receipt, duplicate invoice detection, a bank-detail change flagged, a rejection with a reason returned to the sender, a full audit trail of who changed what, and a rule that codes an invoice to the right ledger account automatically.

  • 5: all ten
  • 4: 8 or 9
  • 3: 6 or 7
  • 2: 3 to 5
  • 1: 2 or fewer

7.Document Generation

A contract template with 15 merge fields, two conditional clauses and a table is filled from a spreadsheet of 20 customers. Each document is checked for every field, the right clauses included or left out, the table complete, the formatting intact and the file name right. I also send one for e-signature and one by email from the tool. A document counts only when it needs no fixing. A tool that cannot generate documents gets no grade on this test.

  • 5: all twenty right, signature and delivery work
  • 4: 18 or 19 right
  • 3: 14 to 17 right, or no e-signature
  • 2: 8 to 13 right
  • 1: 7 or fewer right

8.Integrations & Export

Each point is tried: documents arriving by email to a dedicated address, a watched cloud folder, an API call that uploads and returns the result, a CSV and a JSON export of all 200 results, a sync of ten invoices into QuickBooks or Xero with the right accounts, a Zapier or webhook trigger on completion, a push to an ERP or a database, and an export of the original files with the extracted data attached.

  • 5: all eight
  • 4: 6 or 7
  • 3: 4 or 5
  • 2: 2 or 3
  • 1: 1 or none

9.Security & Data Handling

I read the security page, privacy policy, terms and DPA on the plan tested, and check the settings. I record whether SOC 2 or ISO 27001 is listed with a report available on request, whether the storage region is stated and selectable, whether I can set retention and delete a document and confirm it is gone, whether my documents are used to train models and how to opt out, and whether two-factor or SSO login is available on the plan tested.

  • 5: all five points met
  • 4: four met
  • 3: three met
  • 2: two met
  • 1: one or none, or documents used for training with no opt-out

10.Pricing & Value

Every plan is read from the checkout or the published price list, not the marketing page. The figure graded is the yearly cost of the cheapest plan that processes 1,000 documents a month with the review screen and the accounting or API connection included. Per-page or per-document overages, minimum terms, implementation fees, the renewal price against the first-year price and whether a price is published at all are listed.

  • 5: under $1,200 a year, published, same price at renewal
  • 4: $1,200 to $2,999
  • 3: $3,000 to $5,999, or implementation fees
  • 2: $6,000 or more, or quote-only pricing
  • 1: poor value or hidden costs

Documented but not graded

Pre-built models for other document types, classification and splitting of mixed files, mobile capture apps, e-invoicing compliance, the user interface and support are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real use of the product on a real account and the same real documents, never vendor accuracy claims or analyst charts.
  • Identical documents, templates, tests, thresholds and sample sizes for every tool in the category.
  • Authorised accounts only; no password sharing, no bypassing of plan limits or bot checks, and only test documents that contain no real personal data.
  • Every grade has evidence: an accuracy table, a timing, a screenshot or a checklist, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI document automation tool for testing. No AI document automation tool has been through these tests yet; every one I follow is listed on the AI document automation page, and reviews appear there as tests finish.