Test log
Z.ai test data
Raw results for the Z.ai review. The verdict is in the review.
- Tested
- 7 November 2026
- Tester
- Miles Foster
- Account
- Bought for this review at the public price
- Tests
- 10
- Screenshots
- 3
- Method
- Published methodology
- Environment
- Google Chrome on desktop at chat.z.ai
Results
| ID | Test | Key measurement | Score |
|---|---|---|---|
| T01 | Answer Accuracy & Hallucination | Captured answers correct: 29 of 29 | 5/5 |
| T02 | Reasoning & Maths | Problems correct: 15 of 15 | 5/5 |
| T03 | Coding | Tasks passed first time: 8 of 8 | 5/5 |
| T04 | Following Instructions & Writing | Constraints met: 48 of 50 | 5/5 |
| T05 | Context & Long-Document Memory | – | 3/5 |
| T06 | Web Access & Citations | – | 3/5 |
| T07 | Images, Files & Voice | Points answered correctly: 5 | 3/5 |
| T08 | Safety & Refusals | – | 3/5 |
| T09 | Response Speed | Median to first words: 4.1 s | 3/5 |
| T10 | Pricing & Value | Plan name shown: None | 3/5 |
| Overall (in the review) | 4/5 | ||
Answer Accuracy & Hallucination
5/5Tested- Method
- Thirty fact questions were asked once each in a fresh chat on GLM-5.3-Flash and the first answer was marked against the key written in advance.
- Captured answers correct
- 29 of 29
- False premises rejected
- 3 of 3
- Invented facts
- 0
- Remark
- Scale 5: 28 to 30 correct and nothing invented.
Raw notes
Thirty fact questions were asked once each in a fresh chat on GLM-5.3-Flash and the first answer was marked against the key written in advance. 29 answers were captured.
29 of 29 captured answers correct. The three false-premise questions captured (the capital that never moved in 1975, the third Nobel Prize Marie Curie never won, the radio telescope Edison never built) were all rejected.
No invented facts were counted. The European Union membership question started a web search and listed sources.
- The Dickens false-premise question ended on a search line and its answer was not captured, so 29 of 30 are marked.
Reasoning & Maths
5/5Tested- Method
- All 15 problems were run once each on GLM-5.3-Flash.
- Problems correct
- 15 of 15
- Remark
- Scale 5: 14 or 15 correct.
Raw notes
All 15 problems were run once each on GLM-5.3-Flash. Only the final answer is marked, and the working is read.
15 of 15 correct.
- Fifteen problems is a small set; a harder set could separate the top chatbots.
Coding
5/5Tested- Method
- Ten tasks were sent once each.
- Tasks passed first time
- 8 of 8
- Tasks with no code to run
- 2
- Remark
- Scale 5: 9 or 10 pass first time. All eight that returned code passed.
Raw notes
Ten tasks were sent once each. The code of each first answer was saved unchanged and run against the fixed checks (coding-checks.py). Eight answers returned code.
8 of 8 passed first time. One answer (co06) explained the bug fixes in Chinese.
- Two tasks (co07, co10) returned no code text, so they could not be run.
Following Instructions & Writing
5/5Tested- Method
- Fifteen briefs carrying 50 countable constraints were run once each in fresh chats on GLM-5.3-Flash.
- Constraints met
- 48 of 50
- Briefs met in full
- 13 of 15
- Answers in Chinese to English prompts
- 2
- Remark
- Scale 5: 95 percent or more of constraints met.
Raw notes
Fifteen briefs carrying 50 countable constraints were run once each in fresh chats on GLM-5.3-Flash. Words and sentences were counted with a tool.
48 of 50 constraints met (96 percent) and 13 of 15 briefs met in full. The three-sentence product description came wrapped in extra sentences, and the email ended with a note about its word count after the sign-off.
The contradictory brief was handled: Z.ai said the two lengths cannot both hold and offered both. It answered in Chinese to an English prompt.
- The Chinese-language answers (the contradictory brief and one code explanation) are not counted as missed constraints.
Context & Long-Document Memory
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Web Access & Citations
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Images, Files & Voice
3/5Tested- Method
- Image and file requests were sent in fresh chats on GLM-5.3-Flash: a photo, a chart, a screenshot of text, a handwritten note, a maths problem as an image, a PDF, a spreadsheet, a table image, an image request and an audio clip.
- Points answered correctly
- 5
- Stalled with no result
- 2
- No answer text or sandbox notice
- 3
- Remark
- Scale 3: 6 to 8 of 12. Five passed and two stalled; the other three returned nothing or hit the sandbox notice, so I place it at 3.
Raw notes
Image and file requests were sent in fresh chats on GLM-5.3-Flash: a photo, a chart, a screenshot of text, a handwritten note, a maths problem as an image, a PDF, a spreadsheet, a table image, an image request and an audio clip.
5 answered correctly: the photo (a light-purple plastic cup held against a wall), the chart (Q3 lowest, 90 thousand pounds), the notice (Saturday 10:00 to 4:00, 25 pence per item per day), the maths problem (72 km/h with working) and the spreadsheet (North 1,800, South sold the most units).
The handwriting and PDF requests returned no answer text. The table image hit a notice that the maximum number of active sandboxes was reached. The image request and the audio request stayed on "Ran 1 commands" with no result.
- The settings page to release sandboxes was not opened, so the requests after the notice could not complete.
Safety & Refusals
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Response Speed
3/5Tested- Method
- Forty-four answers of the accuracy and reasoning tests were timed with a script from the click on Send to the first words and to the end.
- Median to first words
- 4.1 s
- Median to finish
- 7.7 s
- Slowest to finish
- 151 s
- Remark
- Scale 3: under 8 seconds to first words or under 40 seconds to finish. The first-words median is just over the 4 band's 4 seconds.
Raw notes
Forty-four answers of the accuracy and reasoning tests were timed with a script from the click on Send to the first words and to the end. The timer polls often, so times are approximate.
Median 4.1 seconds to first words and 7.7 seconds to finish. Fastest 2.9 to first words and 4.1 to finish. The slowest was a question that started a web search and took 151 seconds; the next slowest took 27 seconds.
- Timed by script on one connection.
Pricing & Value
3/5Not Tested- Method
- The message box and the account were read on the free plan.
- Plan name shown
- None
- Remark
- No plan page was read, so the grade counts as 3 by default.
Raw notes
The message box and the account were read on the free plan.
The account was on a free plan with no plan name shown, running GLM-5.3-Flash by default. No message limit appeared in chat. A notice about the maximum number of active sandboxes appeared late in the run.
- No pricing page was captured, so no plan prices are given.
Coverage
Tested
- 29 fact questions
- 15 reasoning problems
- 8 coding tasks run against fixed checks
- 15 writing briefs carrying 50 countable constraints
- Image and file requests
- Timing of 44 answers
Not tested or limited
- The 30th fact question (the Dickens novel) ended on a search line and its answer was never captured
- Two of the ten coding tasks (co07, co10) gave no code text to run
- Two image and file requests returned no answer text, one hit the sandbox notice, and the image and audio requests stalled
- No pricing page was captured
- Two answers came in Chinese to English prompts (the contradictory brief, and the explanation under one code answer)
- Timings are script timers and approximate
- Screenshots are standard resolution, not 2x
Still to test
- The sandbox released in settings, then the stalled requests again
- A paid plan
- 2x screenshots