Test log
Poe test data
Raw results for the Poe review. The verdict is in the review.
- Tested
- 7 November 2026
- Tester
- Miles Foster
- Account
- Bought for this review at the public price
- Tests
- 10
- Screenshots
- 5
- Method
- Published methodology
- Environment
- Google Chrome on desktop at poe.com
Results
| ID | Test | Key measurement | Score |
|---|---|---|---|
| T01 | Answer Accuracy & Hallucination | Questions correct: 30 of 30 | 5/5 |
| T02 | Reasoning & Maths | Problems correct: 15 of 15 | 5/5 |
| T03 | Coding | Tasks passed first time: 10 of 10 | 5/5 |
| T04 | Following Instructions & Writing | Constraints met: 50 of 50 | 5/5 |
| T05 | Context & Long-Document Memory | – | 3/5 |
| T06 | Web Access & Citations | – | 3/5 |
| T07 | Images, Files & Voice | Points passed: 7 of 8 | 4/5 |
| T08 | Safety & Refusals | – | 3/5 |
| T09 | Response Speed | Answers timed: 12 of 45 | 3/5 |
| T10 | Pricing & Value | Free points a day: 300 | 3/5 |
| Overall (in the review) | 4.6/5 | ||
Answer Accuracy & Hallucination
5/5Tested- Method
- All 30 fixed questions were asked once each in a fresh chat with the default bot Assistant, and the first answer was marked against the key written in advance.
- Questions correct
- 30 of 30
- False premises rejected
- 5 of 5
- Invented facts
- 0
- Remark
- Scale 5: 28 to 30 correct and nothing invented.
Raw notes
All 30 fixed questions were asked once each in a fresh chat with the default bot Assistant, and the first answer was marked against the key written in advance.
30 of 30 correct. All five false-premise questions were rejected: Einstein as a head of state, the Dickens novel "The Winds of Harrowgate", the move of the capital from Sydney to Canberra in 1975, Marie Curie's third Nobel Prize, and Edison's radio telescope and Pluto.
No invented facts were counted.
- Twelve questions ran on 7 November and the other 18 on 9 November. The Edison answer ran a web search and cost far more points than a short answer.
Reasoning & Maths
5/5Tested- Method
- All 15 problems were run once each, in fresh chats.
- Problems correct
- 15 of 15
- Trap problems right
- 3 of 3
- Remark
- Scale 5: 14 or 15 correct.
Raw notes
All 15 problems were run once each, in fresh chats. Only the final answer is marked, and the working is read.
15 of 15 correct, including the three trap problems: the scarf and hat (1.50 pounds), the books taken away (8) and the snail (day 9). In the tank problem it found that the two pipes and the drain cancel out, so the tank never fills.
- Three problems were first sent into the same chat as the one before by mistake. They were sent again in fresh chats and the fresh answers were graded.
- Fifteen problems is a small set; a harder set could separate the top chatbots.
Coding
5/5Tested- Method
- The code of each first answer was saved and run against the fixed checks (coding-checks.py).
- Tasks passed first time
- 10 of 10
- Broken versions caught by its tests
- 3 of 3
- Remark
- Scale 5: 9 or 10 pass first time.
Raw notes
The code of each first answer was saved and run against the fixed checks (coding-checks.py).
10 of 10 passed first time. The test-writing task passed on the correct code and its tests caught 3 of 3 deliberately broken versions, so its tests were real.
- Poe's page copy loses the indentation of code, so I restored it by hand from the structure on screen and changed no logic. For the test-writing task the two import lines were in the answer on screen but only came through the copy after View all, so I added them back.
- The explanations beside the code were not graded.
Following Instructions & Writing
5/5Tested- Method
- Fifteen briefs carrying 50 countable constraints were run once each in fresh chats.
- Constraints met
- 50 of 50
- Briefs met in full
- 15 of 15
- Contradictory brief handled
- Yes
- Remark
- Scale 5: 95 percent or more of constraints met.
Raw notes
Fifteen briefs carrying 50 countable constraints were run once each in fresh chats. Words and sentences were counted with a tool.
50 of 50 constraints met and 15 of 15 briefs met in full. The 60-word paragraph was exactly 60 words, the email ended with "Thanks, Alex" on its own final line, and the brief that contradicts itself was named as impossible, with a question asking which length to write.
- One chat per brief, with the first answer counted. In the contradictory brief the bot asked which length to write and did not write either version.
Context & Long-Document Memory
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Web Access & Citations
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Images, Files & Voice
4/5Tested- Method
- Eight points were run in fresh chats: a photo, a chart, a screenshot of text, a handwritten note, a maths problem as an image, a PDF, a spreadsheet and a table image.
- Points passed
- 7 of 8
- Points attempted
- 8
- Not offered
- Image creation, audio, voice
- Remark
- Scale 5: 11 or 12 of 12. Scored out of the points that could be attempted, with the count stated: 7 of 8 is a 4.
Raw notes
Eight points were run in fresh chats: a photo, a chart, a screenshot of text, a handwritten note, a maths problem as an image, a PDF, a spreadsheet and a table image. I also asked for two images, an audio clip to be transcribed and a spoken exchange.
7 of the 8 points passed. The photo (a hand holding a clear plastic cup), the chart (Q3 lowest, 90,000 pounds), the notice (Saturday 10am to 4pm, 25p a day per item), the maths image (72 km/h with working), the PDF (six lanes, and it said no head lifeguard is named) and the table image (Route 12 at 08:50, no fare shown) were right. The handwriting came back as "Hi, there!!" where the note reads "Hi there!!", so it passed as a transcription with an added comma.
The spreadsheet failed: the message with the xlsx file failed to send twice, and nothing was read from it.
Image creation, the audio clip and the voice exchange were not offered. Each image request was priced at 2,759 points against 167 left, with a Subscribe to continue notice. The audio upload was refused with "Assistant does not support audio attachments." The voice button needs a microphone, which the test machine did not have.
- Other bots on Poe may take audio or make images; only Assistant was tested.
- Seven points ran on 7 November; the spreadsheet was not retried on 9 November.
Safety & Refusals
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Response Speed
3/5Not Tested- Method
- Only the 12 answers of 7 November were timed, with approximate script timers from the click on Send to the first words and to the end.
- Answers timed
- 12 of 45
- Short fact answers
- About 1.2 s
- Remark
- Only 12 of the 45 answers were timed, so there is no grade; the test counts as 3 by default.
Raw notes
Only the 12 answers of 7 November were timed, with approximate script timers from the click on Send to the first words and to the end. The answers of 9 November were not timed.
Short fact answers took about 1.2 seconds to first words and to finish, the false-premise answers 2 to 4 seconds, and the web search answer 6.9 seconds. Image answers took 2.4 to 4.9 seconds, and the handwriting 12.2 seconds.
- Script timers on one connection, approximate.
Pricing & Value
3/5Not Tested- Method
- The free plan was used until its points ran out.
- Free points a day
- 300
- Web search answer
- 119 points
- Subscriptions
- Unavailable in my region
- Remark
- No checkout was read, so the grade counts as 3 by default.
Raw notes
The free plan was used until its points ran out. The settings page, the Rates popup and the Subscribe to continue notices were read on 7 November; no checkout was read.
The free plan gives 300 points a day, and the settings page showed the reset about 14 hours after the reading. As Poe showed them, a short text answer cost 4 points, image questions 11 to 44, the PDF question 7 and the web search answer 119, which emptied the allowance on 7 November.
The Subscribe to continue panel listed four yearly plans, the smallest 10,000 points a day at $49.99 a year, read on 7 November. The settings page said subscriptions are currently unavailable in my region, so no checkout was read.
- No plan page or data terms were read; the plan figures come from the Subscribe to continue panel on 7 November.
Coverage
Tested
- 30 fact questions, including all five false-premise questions
- 15 reasoning problems
- 10 coding tasks run against fixed checks
- 15 briefs carrying 50 countable constraints
- 8 image and file points, and two image creation requests, an audio clip and a voice exchange
Not tested or limited
- Only the default bot Assistant was used; Poe offers many other bots and models that were not tested
- The app does not show which model answered; the bot page names the models it routes to
- The 300-point daily allowance ran out on 7 November, so the battery ran in two sittings
- The document, memory, web and safety tests were not run and count as 3; only the 12 answers of 7 November were timed, so speed counts as 3; no checkout was read, so pricing counts as 3
- Page copy of code loses its indentation; it was restored by hand without changing the logic
- Spreadsheet message failed to send twice on 7 November and was not retried, so that point is a fail
- Image creation, the audio clip and the voice exchange were not offered on the free allowance
- Screenshots are standard resolution, not 2x
Still to test
- Other bots on Poe
- A paid plan, where subscriptions are available
- The document, memory, web and safety tests
- Timing all 45 answers
- 2x screenshots