How I test AI app builders

Ten tests, the same for every builder, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, the apps are built from the same three briefs and published for real, and the checklists, screenshots, security transcripts and cost sheets behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

The RankDude mascot holding a laptop with a check on its screen

Each builder gets the same three written briefs. A booking app for a hair salon: customer sign-up, a calendar of free slots, bookings by service and stylist, a staff dashboard and an email confirmation. A team expense tracker: employee and manager roles, receipt uploads, an approval workflow and a monthly CSV export. A small marketplace for handmade goods: seller listings with photos, search and filters, a checkout in Stripe test mode and an order history for both sides. Each brief comes with a hidden checklist of 25 things that must work (75 in all), written before any builder is opened. I build all three apps, publish them, then run the security, bug-fixing, publishing and cost tests on what is actually live. Nothing is graded from a demo video or a template gallery.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 10 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5	Excellent
4	Good
3	Average
2	Below average
1	Poor

The final score is the average of the graded tests (ten when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Some tests on a free plan run out of credits or cannot publish or take payments; I test them again on a paid plan when I can.

The 10 tests

1.First Build

Each brief is pasted as a single prompt into a new project, with the same wording for every builder. When the builder stops, I run the brief's checklist by hand in a clean browser: each item passes only if it works end to end for a real user (a booking that saves and shows up on the staff dashboard, not a button that does nothing). The grade is the average count across the three briefs; the time to the first working preview is reported alongside.

5: 20 or more of 25 on average
4: 15 to 19
3: 10 to 14
2: 5 to 9
1: 4 or fewer, or no working preview

2.Finishing the App

After the first build I send follow-up prompts, one failing checklist item at a time, in plain words a non-developer would use. Each app gets at most ten. After every prompt I rerun the whole checklist, so a change that breaks an item that passed before is counted as a regression. The grade is the share of the 75 checks passing at the end, lowered one level if there were three or more regressions across the three apps.

5: 75 of 75, no regressions
4: 68 to 74
3: 56 to 67
2: 38 to 55
1: under 38

3.Logins, Roles & Data

On the expense tracker and the marketplace I create five accounts each and run sign-up, email verification, login, password reset, sign-out and role changes. I check that records are saved to a real database (not the browser), that relations hold (an order belongs to a buyer and a seller), that data is still there after a refresh, on a second device and after the app is republished, and whether the data can be viewed and edited in a table view inside the builder.

5: all flows work, roles enforced server-side, no data lost
4: one flow needed a follow-up prompt to work
3: password reset or email verification missing, or roles enforced only in the interface
2: data lost on a republish, or logins only through a workaround
1: no working logins or no persistent database

4.Security & Data Access

With two ordinary accounts on each published app I try ten things from the same list every time: open another user's record by changing an ID in the address, read or edit it through the app's own API calls, call those endpoints while logged out, act as a manager from an employee account, upload a file that is not an image, inject script into a text field, look for database, payment or AI keys in the JavaScript the browser downloads, check that row-level rules exist on the database, check the session cookie settings, and run any security scanner the builder offers. Every attempt is recorded with a request transcript; nothing is tried on anyone else's app.

5: 10 of 10 attacks fail, no exposed keys
4: 9 of 10 fail, nothing exposes other users' data
3: 7 or 8 fail
2: another user's data could be read
1: another user's data could be changed, or a secret key is exposed

5.Payments & Integrations

The marketplace takes a payment in Stripe test mode and records the order; the salon app sends a confirmation email to a real inbox; the expense tracker stores receipt images and shows them back; and each app calls one public API (an exchange-rate feed for the expense tracker). I record how each was connected (built-in connector, prompt, or code I had to paste), whether keys were kept server-side, and whether the payment webhook updated the order on its own.

5: all four work without code
4: all four work, one needed a manual setting or a pasted key
3: three of four work
2: two of four, or payments only through an external link
1: one or none

6.Fixing Bugs

On the finished apps I report five bugs the same way a user would (a booking saved for the wrong day, a total that ignores refunds, a filter that shows sold items), then cause two broken builds by asking for changes that conflict. I count how many are fixed within two prompts each, rerun the full checklist after each fix, and note whether the builder showed the error, explained it and offered to roll back to a working version.

5: 7 of 7 fixed first time, no new failures
4: 6 or 7 fixed within two prompts
3: 4 or 5
2: 2 or 3, or a fix broke another feature
1: 1 or none, or no way to roll back a broken build

7.Publishing & Performance

I publish each app, connect the same test domain and time it until a valid certificate is served. I run Lighthouse on mobile three times on the home screen and one logged-in screen of each app and take the median performance score, and I check every screen at 375 pixels wide for overflowing text, unreachable buttons and forms that cannot be submitted. If the builder makes native mobile apps, I also build one for a test device and note what it took to get a store-ready build.

5: custom domain under 15 minutes, median score 90 or above, all screens usable on a phone
4: score 75 to 89, or one screen with a layout fault
3: score 50 to 74, or the custom domain needed support
2: score under 50, or several screens unusable on a phone
1: could not publish on the plan under test

8.Code & Data Ownership

I export the expense tracker's code by every route the builder offers (GitHub sync, zip download) and try to run it on a plain host with its own database, timing the setup and noting every platform-only dependency. I export the app's data to CSV or SQL and check that every table and file came out. If the builder offers no code export, I note what a customer can take with them when they leave.

5: exported app runs elsewhere in under an hour, data exports in one step
4: runs elsewhere after one to three hours of setup
3: code exports but depends on the platform to run; data exports fully
2: data export only, no code
1: neither code nor full data can be exported

9.Editing Without Prompts

I make the same ten changes to each app without asking the AI: rename a button, change the brand colour, add a field to a form, make a field required, reorder the navigation, change a page's access rule, edit an email template, replace an image, change a price and adjust spacing on one screen. I count which can be done directly (visual editor, settings or a code editor I can read), time them, and note whether direct edits cost credits.

5: all ten by hand, no credits used
4: eight or nine
3: five to seven
2: two to four, the rest only by prompting
1: one or none

10.Pricing & Credit Burn

Every plan is read from the checkout, not the marketing page. I build the three apps on the cheapest plan that can publish them and log the credits, messages or tokens each prompt used, including prompts that produced errors or had to be undone. I work out the cost of the build month and of running the three apps for a month after it, note whether usage is shown before a prompt is sent, what happens at the limit, the renewal price against any first-month price and the payment methods seen at checkout.

5: under $30 for the build month, usage visible beforehand, no charge for the builder's errors
4: $30 to $59
3: $60 to $99, or usage only visible afterwards
2: $100 to $199, or credits ran out midway through an app
1: $200 or more, or no way to see what a build cost

Documented but not graded

Templates, team collaboration, native mobile app builds, AI features inside the finished app, agent and automation features, design import from Figma and the quality of the generated design are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

Real use of the product on a real account and real published apps, never demo videos or template galleries.
Identical briefs, hidden checklists, prompts, attacks, thresholds and sample sizes for every builder in the category.
Authorised accounts only; no password sharing, no bypassing of plan limits or bot checks, security tests only on my own apps, and payments only in Stripe test mode.
Every grade has evidence: a checklist, a screenshot, a request transcript, a Lighthouse report or a cost sheet, in the full test results.
A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI app builder for testing. No AI app builder has been through these tests yet; every one I follow is listed on the AI app builders page, and reviews appear there as tests finish.