How I test AI app builders

Ten tests, the same for every builder, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, the app is built from the same written brief and published for real, and the checklist, screenshots, request transcripts, credit ledger and invoices behind each grade go in the full test results published with the review.

Tested and written by Miles Foster, Owner and tester, RankDude.

The RankDude mascot holding a laptop with a check on its screen

Every builder gets the same brief, word for word: Fitting Room, a booking and membership app for a small Pilates studio. Three kinds of user (member, instructor, admin), a timetable with a capacity rule, bookings and cancellations, an admin view with a CSV export, a contact form that saves a record, a $95 five-class pass paid through Stripe in test mode, and two emails. Behind the brief sit 30 written checks, written before any builder is opened, and ten seed facts the finished app must show. I build the app, publish it, then run the access, data, payment, editing, publishing, export, support and cost tests on what is actually live. Nothing is graded from a demo video, a template gallery or a feature list.

An app builder is judged as an app builder, not a website builder: whether two users are kept apart, whether a form writes a record that survives a redeploy, whether a payment creates an order and an email arrives, what it cost to finish, and what you keep if you leave. Those are the things buyers ask about and the things other reviews describe without measuring.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published. Builders that are not prompt-first (a visual editor with an AI assistant) get the same ten tests, and the review says how much was done by hand.

How I rate

Every tool gets one score out of 5.

First, I run the same 10 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5	Excellent
4	Good
3	Average
2	Below average
1	Poor

The final score is the average of the graded tests (ten unless some are left out, see below), to one decimal place, plus my verdict, which can move it by up to one point. When a builder needs the owner’s own keys for payments and emails and none are supplied, the six payment and email checks (C24 to C29) are left out: the actions test is not graded, and the 30-check thresholds are applied in proportion to the 24 checks that remain. A test left out does not count in the average and carries no score.

The 10 tests

1.Prompt to Working App

The clock starts when I submit the brief and stops when the app is live and 25 of the 30 checks pass. Every prompt is logged with the time and the credits it used. The time to the first draft is reported alongside.

5: under 30 minutes and 5 prompts or fewer
4: under 90 minutes and 15 prompts or fewer
3: under 4 hours or 40 prompts or fewer
2: under 12 hours or 100 prompts or fewer
1: longer, or 25 of 30 was never reached

2.Spec Accuracy

Each check is run by hand or by script against the published app and recorded as pass or fail with a screenshot. Sample data left where real data belongs, buttons that do nothing and anything the brief never asked for count against the grade.

5: 28 to 30 pass, nothing invented
4: 24 to 27
3: 18 to 23
2: 10 to 17
1: under 10

3.Sign-in & Access Control

I sign up and log in as a member, an instructor and the admin, confirm the reset email arrives, then replay the app's own data requests with the wrong session and record whether each attack is blocked or succeeds. Whether the builder's own security scan flagged anything real is recorded.

5: all flows work and both attacks are blocked
4: flows work, one weak spot that the tool itself can fix
3: flows work, but one attack succeeds
2: both attacks succeed, or roles are only cosmetic
1: sign-in does not work

4.Database & Admin

Records are created through the app, then checked after a reload, after a republish and after a design change. The builder's own data view is used for the admin checks and the export. Record and storage limits are reported as stated.

5: six of six
4: five
3: four
2: two or three
1: one or none, or data lost on a change

5.Payments & Email Actions

Paid with Stripe's test cards, timed from the click to the email, read from a seeded inbox. A builder that needs code it cannot write itself for any of the four gets a 1.

5: four of four
4: three
3: two
2: one
1: none, or needs code the tool cannot write

6.Iteration & Credit Burn

After each change the 30 checks are run again, so anything that worked before and broke is counted. For every change I record the credits, tokens or dollars used and whether it worked first try. Whether the builder asked before a destructive action, and whether it fell into a loop of failed fixes, is recorded.

5: 11 or 12 first try, no regression
4: 9 or 10, at most one regression
3: 6 to 8
2: 3 to 5, or data lost
1: 2 or fewer

7.Publish, Speed & Mobile

PageSpeed runs three times on the log-in page and the timetable with 100 seeded classes; the median is the grade, with Largest Contentful Paint, layout shift and blocking time reported. Faults are counted at 360, 768 and 1440 pixels and on a phone. Publishing to an app store is outside the review: where a builder offers it, the review says so and nothing is uploaded.

5: live in under 15 minutes, domain works, median 90 or more, 0 to 2 faults
4: domain works, median 75 or more, 3 to 5 faults
3: vendor address only, or median 50 to 74
2: median 30 to 49, or 11 or more faults
1: would not publish, or median under 30

8.Export & Ownership

The export is unpacked in a clean folder, installed and started against an empty database of mine, and the 30 checks are run again. Every call that still needs the vendor's SDK or log-in is listed, and the licence and ownership terms are quoted.

5: runs elsewhere with my own database and accounts, nothing vendor-only
4: runs after small rework
3: front end only, backend rebuilt by hand
2: code visible but will not run alone
1: no export

9.Support & Cancellation

The question is one a stuck user would ask, sent in business hours with the time noted. The cancel is done at the end of the test on the account I paid for, and the live app is checked one day and seven days later.

5: reply under 4 hours, cancel self-serve, final invoice correct
4: reply under 24 hours, cancel self-serve
3: reply within 3 days, or the cancel needs a message
2: reply over 3 days, a scripted answer that did not help, or one wrong charge
1: no reply, or charges after cancelling

10.Pricing & Cost to Finish

Every prompt's cost comes from the credit ledger kept during the build. Plan prices are captured at the checkout with the date. Separate running charges (integration credits, workload units, hosting) are priced at the vendor's published rate.

5: finished within the entry plan's allowance, a cap exists, running cost predictable
4: within one plan up from entry, or no cap but the allowance is clear
3: needed a top-up, or the running cost is unclear
2: cost more than twice the entry plan, or unexpected charges
1: could not finish within the stated allowance, or surprise bills

The 1 briefs, word for word

Every AI app builder gets these exact briefs, pasted unchanged into its AI. The facts listed under each one are what I check on the result.

Studio booking and membership app

Fitting Room

Build a web app called Fitting Room for a small Pilates studio in Austin, Texas, run by Dana Okafor. Three kinds of user: member, instructor and admin, each with their own sign-up or log-in, log-out and password reset by email. Members can browse the class timetable, book a place in a class, cancel a booking, and see a page called My bookings with only their own bookings. Instructors can see only the classes they teach, with the list of members booked on each, and mark each member as attended. Admins can add, edit and delete classes, see every booking, export all bookings as a CSV file, and see a count of bookings per class. A class has a capacity and cannot take more bookings than its capacity. A member cannot see or change another member's bookings. Add a form called Ask the studio with name, email and message; each submission is saved as a record the admin can see in the app. Members buy a 5-class pass for $95 through Stripe (test mode); a successful payment creates an order record and adds 5 class credits to the member, and booking a class uses one credit. Send a confirmation email to the member when they book, and a notification email to the admin. Seed these classes: Reformer Basics, Monday and Wednesday 7:00 am, 50 minutes, capacity 8, instructor Lena Park; Mat Flow, Tuesday and Thursday 6:00 pm, 45 minutes, capacity 12, instructor Tomás Reyes; Power Reformer, Saturday 9:00 am, 55 minutes, capacity 6, instructor Lena Park. A single class without a pass costs $24. The studio is at 2210 Larkspur Street, Austin, TX 78704, phone (512) 555-0177, email hello@fittingroom.example. Use only these facts; do not invent other classes, prices, instructors or reviews.

The 10 facts I check ›

App and studio name Fitting Room
Run by Dana Okafor
2210 Larkspur Street, Austin, TX 78704
Phone (512) 555-0177
Email hello@fittingroom.example
Reformer Basics, Monday and Wednesday 7:00 am, 50 minutes, capacity 8, Lena Park
Mat Flow, Tuesday and Thursday 6:00 pm, 45 minutes, capacity 12, Tomás Reyes
Power Reformer, Saturday 9:00 am, 55 minutes, capacity 6, Lena Park
5-class pass $95
Single class $24

Documented but not graded

Templates, team collaboration, AI features inside the finished app, agents and automations, design import from Figma and the look of the generated design are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade. Native mobile builds and app-store publishing are outside the review: where a builder offers them, the review says so, and nothing is uploaded to a store.

Rules that apply to every test

Real use of the product on a real account and a real published app, never demo videos or template galleries.
The same brief, checks, changes, attacks, thresholds and sample sizes for every builder in the category.
Authorised accounts only; no password sharing, no bypassing of plan limits or bot checks, access-control tests only on my own app, and payments only in Stripe test mode.
Every grade has evidence: a checklist, a screenshot, a request transcript, a PageSpeed report, a credit ledger or an invoice, in the full test results.
A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI app builder for testing. The tested ranking is on the AI app builders page.