RankDude

How I test AI agents for websites

Nine tests, the same for every agent, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I use, the agent is installed on real test sites and asked real questions, and the transcripts, timings and screenshots behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

Each agent is installed on the same three test sites: a WordPress site for a bakery with 30 pages of content, a Shopify shop with ten products and a set of test orders, and a plain HTML site for a photographer. Each site has a question bank: 40 questions whose answers are on the site, and 10 whose answers are not. I ask every question as a visitor, I try to get a person, I leave my details, and I try to get the agent to do things such as look up an order or book a slot. Nothing is graded from a feature list or a demo video.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 9 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (nine when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Some tests on a free plan can't connect a shop or hand over to a person; I test them again on a paid plan when I can.

The 9 tests

1.Setup to First Answer

The clock starts at account creation and stops when the widget is live on each site and answers a known question correctly from that site's content. Each site is timed separately: the WordPress plugin or script, the Shopify app, and the plain script tag. I note whether the agent crawled the site itself or needed the content uploaded, and whether any step needed a developer or support.

  • 5: all three under 30 minutes, no help
  • 4: under 1 hour
  • 3: under 3 hours, or one site needed support
  • 2: over 3 hours, or one site could not be done
  • 1: two sites could not be done

2.Answer Accuracy

The 40 questions per site cover opening hours, prices, delivery, returns, allergens, product details and booking rules, all with answers written on the site. Each is asked once, as a visitor, and marked against the written answer: right, partly right (missing a condition or a figure) or wrong. Partly right scores half. The transcript of every question goes in the test results.

  • 5: 108 or more right
  • 4: 96 to 107
  • 3: 78 to 95
  • 2: 60 to 77
  • 1: under 60

3.Saying I Don't Know

The 10 questions per site have no answer anywhere on the site (a price not listed, a service not offered, a policy that does not exist). Each is asked once. An answer counts as invented when it states a fact that is not on the site. I also note whether the agent cites where its answers come from and whether it offers a person when it cannot answer.

  • 5: 0 invented answers
  • 4: 1 or 2
  • 3: 3 to 5
  • 2: 6 to 10
  • 1: 11 or more

4.Handover to a Person

On each site I ask for a human ten times in different ways (asking outright, complaining, asking a question outside the content, during and outside set office hours). I record whether the chat reaches a live inbox or creates a ticket, how long it takes, whether the full transcript and my contact details arrive with it, and what the visitor is told when nobody is available.

  • 5: all 30 attempts handed over with the transcript, under a minute in hours
  • 4: 27 to 29, or handover takes a few minutes
  • 3: 20 to 26, or the transcript is lost
  • 2: 10 to 19
  • 1: under 10, or no handover at all

5.Lead Capture & Booking

On each site I run conversations that should produce a lead or a booking: asking for a quote, asking to be called back, booking a shoot, pre-ordering a cake. I check that the name, email, phone and request arrive where the site owner would see them, that a booking lands in the connected calendar at the right time, and that the visitor gets a confirmation. Twenty conversations in all; a capture counts only when every detail is right.

  • 5: all twenty right
  • 4: 17 to 19
  • 3: 13 to 16
  • 2: 8 to 12
  • 1: 7 or fewer, or no capture feature

6.Actions & Shop Data

Each point is tried on the test shop with test orders: look up an order by number and email, give live stock for a product, recommend a product from the catalogue with the right price, apply a discount code, start a return, change a delivery address, cancel an unshipped order with my approval, create a support ticket, write to the CRM or email list, and show the right answer in a second language. Each counts when it works end to end.

  • 5: all ten
  • 4: 8 or 9
  • 3: 6 or 7
  • 2: 3 to 5
  • 1: 2 or fewer

7.Speed & Site Impact

For 30 questions across the three sites I time from sending the message to the first word of the answer, and to the complete answer. The median of the first-word times is graded. I run Lighthouse mobile three times on each site's home page before the widget is installed and three times after, and report the change in the median performance score and the added page weight.

  • 5: median under 2 seconds, 2 points or fewer lost
  • 4: under 4 seconds, 5 points or fewer lost
  • 3: under 8 seconds, or 6 to 10 points lost
  • 2: under 15 seconds, or more than 10 points lost
  • 1: 15 seconds or longer

8.Data Handling

I read the privacy policy, terms and DPA on the plan tested, and check the settings for the hosting region, chat retention, deletion and a training opt-out. I watch the widget's network traffic to see which domains receive the chat and whether it loads before consent. I record whether the visitor is told they are talking to an AI, and how a visitor can ask for their chats to be deleted.

  • 5: all five points met
  • 4: four met
  • 3: three met
  • 2: two met
  • 1: one or none, or chats sent to undisclosed third parties

9.Pricing & Value

Every plan is read from the checkout, not the marketing page. The figure graded is the yearly cost of the cheapest plan that includes handover to a person, the Shopify connection and 500 conversations a month. The cost at 2,000 conversations, per-conversation or per-resolution charges, seat fees for the people who take handovers, the renewal price against the first-year price and the payment methods seen at checkout are listed.

  • 5: under $600 a year at 500 conversations, handover and shop included
  • 4: $600 to $1,199
  • 3: $1,200 to $2,399, or handover costs extra
  • 2: $2,400 or more, or per-resolution charges with no cap
  • 1: poor value or hidden costs

Documented but not graded

Other channels (WhatsApp, Instagram, email), voice, analytics dashboards, the quality of the visual widget and the agent's tone are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real use of the product on a real account and real test sites, never promotional descriptions or demo videos.
  • Identical sites, question banks, tests, thresholds and sample sizes for every agent in the category.
  • Authorised accounts only; no password sharing, no bypassing of plan limits or bot checks, and test orders only in test mode or refunded.
  • Every grade has evidence: a transcript, a timing, a screenshot or a checklist, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit an AI agent for websites for testing. No AI agent for websites has been through these tests yet; every one I follow is listed on the AI agents for websites page, and reviews appear there as tests finish.