RankDude

How I test self-hosted AI agents

Ten tests, the same for every agent, each graded from 1 to 5 on a fixed scale. The overall grade is their average plus my verdict, which can move it by up to one point. Every test is run by hand on the same two machines I own, with the network watched, and the timings, counts, captures, screenshots and transcripts behind each grade go in the full test results published with the review.

Written by Miles Foster, Owner and tester, RankDude.

Each agent is installed from scratch on the same two machines: a desktop with 32 GB of memory and a mid-range graphics card, and a plain 8 GB Linux server with no graphics card. Each gets the same local model served through the same runtime, and the same twenty tasks: file and folder work, web research with sources, writing and running a script, summarising a folder of documents, extracting data from PDFs, drafting messages, and a few multi-step jobs that chain those together. I run the tasks with the local model first and with a cloud model second, and I record every outbound connection the agent makes while it works. Nothing is graded from a README, a star count or a demo video.

Each review states how the account was obtained: bought at the public price, used on a free tier, or provided by the vendor. Most agents in this category have no account at all; where a paid tier or hosted plan exists, the review says whether it was used and how it was paid for. A vendor-provided account never changes a grade, and a vendor never sees a review before it is published.

How I rate

Every tool gets one score out of 5.

First, I run the same 10 tests on every tool and give each one a grade from 1 to 5. The average of the graded tests is the starting point; a test that cannot be graded is left out of it, and the review says how many tests the score is based on.

Then I add my verdict: what the tests cannot measure, such as how the tool holds up after the test weeks, how billing and cancelling work in practice, and how it compares with the others in the category. My verdict can move the score up or down by one point at most.

Every review shows how much my verdict changed the score, and why, in its scores table.

What a grade means

5Excellent
4Good
3Average
2Below average
1Poor

The final score is the average of the graded tests (ten when every test can be graded), to one decimal place, plus my verdict, which can move it by up to one point. If a test couldn't be finished properly it gets no grade, and the review says how many tests the score is based on. Code frameworks (BabyAGI, MetaGPT, CrewAI and the like) are listed under AI agent builders, and model runtimes such as Ollama or LM Studio are not listed at all: a runtime only serves the model the agents here use. A chat front-end with agent features (Open WebUI or LibreChat, say) serves the same model itself and does the twenty tasks with its own chat and agent features, and the review says so.

The 10 tests

1.Install to First Task

The clock starts on a fresh operating system with only the model runtime installed and stops when the agent completes the first of the twenty tasks. I count every manual step: a command, a file edited, a key pasted, a setting changed. The timing is done on both machines; the slower one is graded. Whether the install is a single command, a container or a build from source is recorded.

  • 5: under 20 minutes, 5 steps or fewer
  • 4: 20 to 44 minutes, or 6 to 10 steps
  • 3: 45 to 89 minutes, or 11 to 20 steps
  • 2: 90 minutes to half a day, or more than 20 steps
  • 1: more than half a day, or could not be installed on one of the machines

2.Model Support

Each point is tried, not read from the docs: works with the local runtime out of the box, accepts any OpenAI-compatible endpoint, swaps model without reinstalling, finishes at least half the twenty tasks on an 8-billion-parameter local model, runs a local vision model, uses a local embedding model for documents, runs with no cloud key set at all, and lets a cloud model be added as an option rather than a requirement.

  • 5: all eight
  • 4: 6 or 7
  • 3: 4 or 5
  • 2: 2 or 3
  • 1: 1 or none, or a cloud key is required to start

3.Task Completion

The same twenty tasks for every agent, run with the same local model on the desktop. A task counts when the result is right and I did not finish it by hand: the file is where it should be, the script runs, the summary matches the documents, the extracted figures are correct, the research answer cites real pages. The run is repeated with a cloud model and that count is reported alongside, but the local count is the grade.

  • 5: 18 to 20 tasks
  • 4: 14 to 17
  • 3: 10 to 13
  • 2: 5 to 9
  • 1: 4 or fewer

4.Offline & Privacy

With the local model set and the network captured, ten tasks are run that need no internet. Every connection to anything other than my own machines is logged: update checks, telemetry, crash reports, model downloads, licence checks. I then check whether telemetry is documented, off by default or has a switch, and whether the agent's data, memory and logs are stored where I can read and delete them.

  • 5: no outbound connections
  • 4: update checks only, documented and switchable
  • 3: telemetry on by default but documented and switchable
  • 2: undocumented connections, or telemetry with no switch
  • 1: task content or files sent out without being told

5.Security Defaults

Each is checked on the fresh install: the agent runs in a sandbox or container by default, asks before running a shell command, has file access limited to a chosen folder, requires a login on its web interface or gateway, does not listen on the network by default, stores keys and secrets encrypted or outside plain config, has an allow-list for tools, separates permissions per channel or user, writes a log of every action, and ships a hardening guide. A protection counts when it is on without me turning it on.

  • 5: all ten
  • 4: 8 or 9
  • 3: 6 or 7
  • 2: 3 to 5
  • 1: 2 or fewer, or the fresh install was reachable from the network without a login

6.Memory Across Sessions

On day one I tell the agent ten facts in the course of normal tasks: names, preferences, a deadline, a file location, a rule. On day two, in a new session after a restart, I ask questions that need each fact. The count recalled is the grade. I also check whether I can read the stored memory as plain files or a page, edit an entry and delete one, and whether the deletion holds after a restart.

  • 5: 9 or 10 recalled, memory readable, editable and deletable
  • 4: 7 or 8 recalled
  • 3: 4 to 6 recalled, or memory not editable
  • 2: 1 to 3 recalled
  • 1: none recalled, or no memory across sessions

7.Resource Use

On each machine I measure the agent's own idle and peak memory, CPU during tasks and disk used after install, with the model runtime measured separately so the agent is not blamed for the model. The grade is whether the agent completes the twenty tasks on the 8 GB server and how much memory it uses of its own. Idle power draw on the desktop is reported for agents that run always-on.

  • 5: completes the task set on the 8 GB server, under 1 GB of its own memory
  • 4: completes on the server, 1 to 2 GB
  • 3: completes on the server, 2 to 4 GB, or needs a swap file
  • 2: runs only on the desktop, or needs its graphics card
  • 1: needs more than the desktop's 32 GB of memory, or hardware the desktop does not have, to run at all

8.Interfaces & Channels

A terminal or command line, a web interface, a desktop app, at least one messaging channel such as Telegram, Discord, WhatsApp or Slack, an API I can call from a script, and scheduled or event-triggered runs that happen while I am not there. Each counts when a task completes through it; a channel that only shows output but cannot take a task counts as half, and a half is rounded down in the total.

  • 5: all six
  • 4: 5
  • 3: 3 or 4
  • 2: 2
  • 1: 1

9.Licence & Maintenance

I read the licence of the part I installed, count releases in the last 90 days, record the median time to a maintainer's first reply on the ten most recent issues, check the install and upgrade documentation, and upgrade from the version I installed to the newest one with my data and memory in place. Each point is reported; the grade combines them as below.

  • 5: OSI-approved licence, 3 or more releases in 90 days, median issue reply under 7 days, upgrade kept all data
  • 4: all of the above but one
  • 3: licence has a commercial restriction, or no release in 90 days, or upgrade lost settings
  • 2: issues unanswered for over a month, or upgrade lost memory or data
  • 1: repository archived or unmaintained, or no open-source licence

10.Cost to Run

I add up the software fees for any paid tier the task set needed, the cost of the smallest server that completed the task set at a mainstream host's published price, and the cloud model spend the twenty tasks cost when the cloud model was used, scaled to a year of weekly use. The figure with the local model is the grade; the figure with the cloud model is reported alongside. Features that are open source in name but paid in practice are listed.

  • 5: under $100 a year with the local model
  • 4: $100 to $299
  • 3: $300 to $599, or a paid tier needed for a core feature
  • 2: $600 to $1,199
  • 1: $1,200 or more, or cannot run without a paid service

Documented but not graded

Voice input and output, multi-agent features, browser automation, plugin and skill ecosystems, documentation quality and the community around the project are checked and described in every review. When they work well they count in the written assessment, but they never add points to the overall grade.

Rules that apply to every test

  • Real installs on machines I own, with the network watched, never README claims, star counts or demo videos.
  • Identical machines, model, runtime, tasks, thresholds and sample sizes for every agent in the category.
  • Authorised use only; no bypassing of licence terms or paid tiers, and no agent given access beyond the test machines.
  • Every grade has evidence: a timing, a count, a network capture, a screenshot or a transcript, in the full test results.
  • A grade changes only when a retest produces new evidence; the review records the date, the version and the reason.

Questions about a specific grade go to the contact page. Makers can submit a self-hosted AI agent for testing. No self-hosted AI agent has been through these tests yet; every one I follow is listed on the Self-hosted AI agents page, and reviews appear there as tests finish.