How I test AI image generators

Ten tests, the same for every generator, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, and the prompts, the images, the counts, screenshots and timings behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

The RankDude mascot holding a laptop with a check on its screen

Each generator gets the same session: the same 40 written prompts in six groups (six photographic portraits and people, six product and food shots, six images with set text, six busy scenes with counted objects and set positions, ten style prompts and six scenes for a recurring character), the same four reference photos of my own for the editing test, the same twelve edit instructions and the same account settings: the best model the plan offers, default quality, a 1:1 frame unless the prompt asks for another. Every prompt is written down with the elements it asks for, so a result can be counted rather than admired, and no prompt names a living artist or a real person.

Generators return random results, so the first image of the first run counts for every prompt, with no rerolls and no picking the best of a grid. Every image is inspected at full size, not as a thumbnail, and nothing is graded from a sample gallery, a feature list or a press kit. Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 10 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5	Excellent
4	Good
3	Average
2	Below average
1	Poor

The final score is the average of the graded tests (ten when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Some tests on a free plan use fewer images than usual because of daily limits; I test them again on a paid plan when I can.

The 10 tests

1.Prompt Adherence

Each of the 40 prompts lists its elements in advance: objects, colours, counts, positions, actions, clothing, lighting and the frame, between four and nine per prompt and about 240 in all. The six busy scenes are written to be hard: three red apples to the left of a blue jug, a dog under the table and a cat on the chair, two people passing one object between them. An element counts only when it is there and correct; a wrong count, colour or position is a miss. Anything added that the prompt did not ask for and that changes the scene is noted as well.

5: 90% or more of elements correct
4: 80% to 89%
3: 65% to 79%
2: 50% to 64%
1: under 50%

2.Photorealism & Anatomy

The six portrait and six product and food prompts are each run twice, giving 24 images. Each is inspected at 100% for faults that give a generated image away: hands with the wrong number or shape of fingers, mismatched eyes, merged teeth, waxy or plastic skin, warped text on packaging, light and shadows that disagree, reflections that do not match and objects melting into each other. An image passes only with no fault visible at full size. The number of hand and face faults is reported separately.

5: 22 to 24 of 24 pass
4: 18 to 21 pass
3: 13 to 17 pass
2: 7 to 12 pass
1: 6 or fewer pass

3.Text in Images

The six text prompts ask for exact wording in a set place: a shop sign, a product label, a poster with a headline and a date, a book cover with title and author line, a handwritten note and a street sign in a photograph. Together they hold 60 words, including two numbers and one word of twelve letters. A word counts as correct only when it is spelled exactly, legible at full size and in the place the prompt asked for. Extra invented text is counted as a fault, and an image is fully correct only with every word right and nothing extra.

5: 95% or more of words correct, five or six images fully correct
4: 85% to 94%, or four images fully correct
3: 70% to 84%
2: 40% to 69%
1: under 40%, or no readable text

4.Style Range

The ten style prompts describe the same subject, a lighthouse on a rocky coast at dusk, in ten styles named by technique, never by an artist: oil painting, watercolour, pencil sketch, flat vector illustration, Japanese woodblock print, anime cel, pixel art, clay 3D render, 1970s film photograph and black ink line art. A result counts when the style shows in the medium itself (brush marks, paper, pixels, grain) rather than as a filter over a photographic image, and when the lighthouse, the rocks and the dusk light are all still there.

5: all ten styles hit, subject kept
4: 8 or 9 styles hit
3: 6 or 7 styles hit
2: 4 or 5 styles hit
1: 3 or fewer, or every style drifts to one look

5.Character & Subject Consistency

A woman with a written description (face shape, hair, a scar above one eyebrow, a green coat) and a product (a yellow thermos with a black lid and a printed logo of my own) each go into the same six scene prompts. Where the generator offers a character or reference feature, I use it with the first image as the reference; otherwise the description is repeated word for word. Each of the twelve images is compared with the first at full size: face, hair, the scar, clothing, the product's shape, colour and logo. An image counts only when all of these hold.

5: 11 or 12 of 12 consistent
4: 9 or 10 consistent
3: 6 to 8 consistent
2: 3 to 5 consistent
1: 2 or fewer, or no way to keep a subject

6.Editing, Inpainting & Image-to-Image

Four masked inpainting edits (replace a hat, remove a cup, add a bird to the sky, change a shirt pattern), four written edits on a generated image that ask for one change each, two image-to-image restyles of my reference photos with the composition kept, and two frame expansions of 50% of the width. The first result is kept. An edit counts when the instruction is carried out, no seam shows at full size and the rest of the image, faces included, is unchanged when compared side by side. Editing that only works by generating a new image from scratch scores as missing.

5: 11 or 12 of 12 correct
4: 9 or 10 correct
3: 6 to 8 correct
2: 3 to 5 correct, or only some kinds of edit offered
1: 2 or fewer, or no editing

7.Resolution & Upscaling

Every downloaded image is measured in pixels at the default and the highest quality setting, and the aspect ratios on offer are listed. Six images from the portrait, product and text groups are then run through the built-in upscaler at its largest setting. Each upscale is compared with the original at the same size for invented texture, changed faces, halos and whether small text stays readable. File formats offered (PNG, JPEG, WebP) and any compression are recorded.

5: 2 megapixels or more native, clean upscale to 4,096 pixels or more
4: 1 megapixel or more native, upscale with minor invented texture
3: 1 megapixel or more native, with no upscaler or a soft one
2: under 1 megapixel, or the upscale changes faces
1: under 512 pixels on the long edge, or no clean download

8.Speed per Image

All 40 prompt adherence generations are timed on the same connection, from pressing generate until the finished image is shown at full resolution; progress bars, blurred previews and queue messages do not count. When a generator returns several images at once, the time to the first full image is used. The median is the grade; the average, fastest, slowest, any failed generations and the free-tier queue time, when it differs, are reported.

5: median under 10 seconds
4: 10 to 19.9 seconds
3: 20 to 39.9 seconds
2: 40 to 89.9 seconds
1: 90 seconds or longer

9.Usage Rights, Watermarks & Privacy

Each point is checked on the terms in force on the test date and on real downloads, not on a marketing page: commercial use allowed on the paid plan, commercial use allowed on the free plan, the terms state who owns or may use the output, no visible watermark on paid downloads, no visible watermark on free downloads, generations private by default rather than posted to a public gallery, an opt-out from training on my prompts and uploads, a stated source for the training data, a provenance label such as content credentials kept in the file, and a clear rule on images of real people and brands that the generator enforces when I ask.

5: all ten
4: 8 or 9
3: 6 or 7
2: 4 or 5
1: 3 or fewer, or no commercial use on any plan

10.Pricing & Value

On the free tier I generate until the generator stops me and record how many images that was and when the allowance resets. Every plan and credit pack is then read from the checkout, not the marketing page, and the cost of one full-resolution image at the default quality is worked out in dollars on the cheapest plan that allows commercial use. Credits charged for upscales, edits and the best model are listed separately, as are renewal terms and payment methods as seen at checkout.

5: under $0.02 per image, with at least 10 free images a day
4: $0.02 to $0.049 per image
3: $0.05 to $0.099 per image
2: $0.10 or more per image, or credit costs unclear
1: poor value or hidden costs

Documented but not graded

Video and animation from an image, prompt helpers and rewriting, seeds, negative prompts and other advanced settings, batch generation, mobile apps, plugins for design tools and an API are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

Real use of the product on a real account, never sample galleries, curated showcases or promotional descriptions.
Identical prompts, reference photos, settings, thresholds and sample sizes for every generator in the category, with the first result counted every time.
Authorised accounts only; no password sharing, no bypassing of usage limits, content filters or bot checks, no prompts naming living artists or real people, and only reference photos I own or have permission to use.
Every grade has evidence: the prompt, the image, a count, a measurement, a screenshot or a table, in the full test results.
A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI image generator for testing. No AI image generator has been through these tests yet; every one I follow is listed on the AI image generators page, and reviews appear there as tests finish.