How I test deployment platforms

Ten tests, the same for every platform, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on an account I pay for or am given, the apps are deployed for real and left running for a month, and the timings, response-time tables, invoices, logs and screenshots behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

The RankDude mascot holding a laptop with a check on its screen

Each platform gets the same three apps from the same three git repositories: a static documentation site built with a generator (about 200 pages), a Next.js web app with API routes, server rendering and image uploads, and a Python API in a Dockerfile with a Postgres database, a Redis queue and a background worker. Every repository has a health endpoint, a test suite and a load script. I deploy all three from the repository, connect the same test domain, push ten commits over the month, and run the scaling, performance, rollback and cost tests on what is actually live. Nothing is graded from a feature matrix or a launch post.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 10 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5	Excellent
4	Good
3	Average
2	Below average
1	Poor

The final score is the average of the graded tests (ten when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. A platform that only hosts static sites or only containers gets no grade on the apps it cannot run, and the review says which; its score is based on the tests that could be run.

The 10 tests

1.Repo to Live URL

The clock starts when I connect the repository (or run the first CLI deploy) and stops when the health endpoint answers 200 at the platform's public address. Each of the three apps is timed on a fresh project; the median is the grade. I record how many settings had to be filled in by hand (build command, start command, port, root directory) and whether the platform detected the framework and the Dockerfile on its own.

5: median under 5 minutes, nothing to configure
4: under 5 minutes with one or two settings filled in, or 5 to 9 minutes
3: 10 to 19 minutes, or a config file had to be written
2: 20 to 44 minutes, or one app needed support to deploy
1: 45 minutes or longer, or one app could not be deployed

2.Builds & Runtimes

Each app is deployed from its preset where one exists and from its Dockerfile where it must be. I then push the same ten commits to each app (dependency changes, an environment variable change, a build-breaking commit, a monorepo path change) and record each build's duration, whether the cache was used, whether the broken commit was kept off the live URL, and how clearly the build log explained the failure. Supported languages and runtime versions are checked against the documentation and tried where they differ.

5: every build succeeds, the broken commit is held back, cache cuts rebuilds in half or more
4: one build needed a manual fix
3: no build cache, or the broken commit went live briefly
2: one of the three apps could only be built with a workaround
1: one app could not be built

3.Databases & Services

The Python app needs Postgres, Redis, a worker process and file storage. I provision each on the platform where it offers them and note the plan, the backup policy, whether a restore works, and whether traffic between services stays on a private network. Where the platform offers none, I connect an external service and record the extra steps and latency. The worker is checked for running continuously, not only on a request, and cron jobs are tried.

5: all four on the platform, backups restorable, private networking, cron and workers native
4: three of the four on the platform, the rest connect in one step
3: database only, or external services with manual networking
2: no managed services; the worker needed a workaround
1: the worker or the database could not be run at all

4.Scaling Under Load

The load script sends 50, 200 and 500 requests per second to the API app and the Next.js app for five minutes each, from a server in Virginia. I record the error rate, the p95 response time, how long autoscaling took to add an instance when it is offered, whether it scaled back down, and what the run cost. A platform without autoscaling is tested at its largest single instance and at a manual scale-out to three instances.

5: under 0.1% errors at 500 rps, autoscaling within a minute, scales down
4: under 1% errors at 500 rps, or manual scaling that works without downtime
3: 500 rps produced errors but 200 rps was clean
2: errors at 200 rps, or scaling caused a restart with downtime
1: the app fell over at 50 rps or scaling is not possible

5.Response Times Worldwide

Every hour for 30 days, a monitor requests the static home page, a server-rendered page and an API route from Virginia, Oregon, São Paulo, London, Frankfurt, Mumbai, Singapore and Sydney and records time to first byte. I report the median and p95 per region and, separately, the first request after 30 minutes idle, which shows whether instances sleep. The grade uses the worst region's median on the plan under test, so a single-region platform is graded on what distant users see.

5: worst-region median under 100 ms, cold start under 300 ms or none
4: worst-region median under 250 ms
3: under 500 ms, or cold starts of 1 to 3 seconds
2: under 1 second, or cold starts over 3 seconds
1: worst-region median over 1 second

6.Uptime & Incidents

The same hourly monitor records every failed or slow check over the month. Each failure is matched against the platform's status page to see whether it was reported, how fast, and whether the post-mortem said what happened. I also read the previous twelve months of status history and count incidents that affected deploys or serving. A deploy of each app is made during the month at a busy time to see whether live traffic was dropped.

5: no failed checks, status history clean or every incident reported with a post-mortem
4: under 0.1% failed checks, all reported
3: under 0.5% failed checks, or incidents not on the status page
2: 0.5% to 2% failed checks, or a deploy dropped live requests
1: over 2% failed checks or an outage over an hour in the month

7.Domains, TLS & Networking

I connect the apex domain, a www subdomain and a wildcard to the apps and time how long until a valid certificate is served. I check certificate renewal settings, apex handling without a CNAME trick, redirect and header rules, a DDoS or WAF option, static outbound IPs for the API's database allowlist, IPv6 and HTTP/3, each with a curl transcript in the results.

5: all of the above in under 10 minutes
4: all of it, but one feature needed a support ticket or a paid add-on
3: apex or wildcard needed a workaround, or no header rules
2: certificate took over an hour or needed manual renewal
1: a custom domain could not be connected on the plan under test

8.Previews, Rollbacks & Environments

Each of the ten commits goes through a pull request. I record whether a preview URL appeared, how long it took, whether it had its own database or shared production's, and whether it was cleaned up on merge. I roll back from the build-breaking commit and time the rollback. I create a staging environment and check how secrets are stored, scoped and rotated, and whether a secret ever appeared in a log or a build output.

5: previews with isolated data, rollback under 30 seconds, scoped secrets, no leaks
4: previews share the database, or rollback takes a rebuild
3: previews by hand only, or no staging environment
2: no previews, or a secret appeared in a build log
1: no rollback possible

9.Logs, Metrics & Alerts

Each app writes a known error line once an hour. I time how long it takes to find it from the dashboard, check whether logs can be searched, filtered and exported or drained to another tool, and how long they are kept. During the load test I read CPU, memory and request metrics on the platform. I then stop the worker and break the health check on purpose and wait for an alert by email, Slack or webhook.

5: searchable logs with retention of 7 days or more, metrics, drains, alert within 5 minutes
4: all of it but one feature is a paid add-on
3: logs tail only or 24-hour retention, metrics present, no alerting
2: no metrics, or logs not searchable
1: logs missing lines, or no logs on the plan under test

10.Pricing & Monthly Bill

Every plan is read from the checkout, not the marketing page. I run the three apps for the month on the cheapest plan that supports them, with the load tests included, and compare the invoice with the estimate I made from the pricing page on day one. I record every line item (compute, bandwidth, build minutes, databases, seats), whether a spending cap or alert exists, what happens when a free tier runs out, the renewal price against any first-month price and the payment methods seen at checkout.

5: under $25 a month, within 10% of the estimate, spend limit available
4: $25 to $59, or within 25% of the estimate
3: $60 to $119, or the bill was over the estimate by more than 25%
2: $120 to $249, or no way to cap spending
1: $250 or more for three small apps, or a surprise charge not on the pricing page

Documented but not graded

Team seats and roles, a CLI and API, infrastructure-as-code, bring-your-own-cloud, edge functions, GPU instances, compliance reports (SOC 2, HIPAA) and data residency are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

Real deployments of the same three apps on a real account, left running for a month, never a feature matrix or a launch post.
Identical repositories, commits, domains, load scripts, monitoring regions, thresholds and sample sizes for every platform in the category.
Authorised accounts only; no bypassing of plan limits, rate limits or bot checks, and load tests kept within the platform's acceptable-use policy.
Every grade has evidence: a timing, a response-time table, a curl transcript, a log, an invoice or a screenshot, in the full test results.
A grade changes only when a retest produces new evidence; the review records the date and the reason.

Questions about a specific grade go to the contact page. Makers can submit a deployment platform for testing. No deployment platform has been through these tests yet; every one I follow is listed on the Deployment platforms page, and reviews appear there as tests finish.