Test log
Qwen Chat test data
Raw results for the Qwen Chat review. The verdict is in the review.
- Tested
- 7 November 2026
- Tester
- Miles Foster
- Account
- Bought for this review at the public price
- Tests
- 10
- Screenshots
- 4
- Method
- Published methodology
- Environment
- Google Chrome on desktop at chat.qwen.ai
Results
| ID | Test | Key measurement | Score |
|---|---|---|---|
| T01 | Answer Accuracy & Hallucination | Questions answered: 29 of 30 | 5/5 |
| T02 | Reasoning & Maths | Problems correct: 15 of 15 | 5/5 |
| T03 | Coding | Tasks passed first time: 10 of 10 | 5/5 |
| T04 | Following Instructions & Writing | Constraints met: 50 of 50 | 5/5 |
| T05 | Context & Long-Document Memory | – | 3/5 |
| T06 | Web Access & Citations | – | 3/5 |
| T07 | Images, Files & Voice | Points passed: 8 of 8 | 5/5 |
| T08 | Safety & Refusals | – | 3/5 |
| T09 | Response Speed | Median to first words: 7.5 s | 3/5 |
| T10 | Pricing & Value | – | 3/5 |
| Overall (in the review) | 4.2/5 | ||
Answer Accuracy & Hallucination
5/5Tested- Method
- The fixed questions were asked once each in a fresh chat on Qwen3.7-Plus in Auto mode, and the first answer was marked against the key written in advance.
- Questions answered
- 29 of 30
- Answered correctly
- 29 of 29
- False premises rejected
- 4 of 4 asked
- Invented facts
- 0
- Remark
- Scale 5: 28 to 30 correct and nothing invented.
Raw notes
The fixed questions were asked once each in a fresh chat on Qwen3.7-Plus in Auto mode, and the first answer was marked against the key written in advance.
29 of 29 answered questions were correct, and the 30th was never answered because the slider stopped chat. All four false-premise questions I asked were rejected: Einstein as a head of state, the Dickens novel "The Winds of Harrowgate", the move of the capital from Sydney to Canberra in 1975, and Marie Curie's third Nobel Prize.
No invented facts were counted. Auto mode showed source names beside many answers.
- The fifth false-premise question (Edison's radio telescope and Pluto) was the one never answered. Auto mode searches the web, so these answers are not from memory alone.
Reasoning & Maths
5/5Tested- Method
- All 15 problems were run once each in fresh chats.
- Problems correct
- 15 of 15
- Trap problems right
- 3 of 3
- Remark
- Scale 5: 14 or 15 correct.
Raw notes
All 15 problems were run once each in fresh chats. Only the final answer is marked, and the working is read.
15 of 15 correct, including the three trap problems: the scarf and hat (1.50 pounds, with a note that many people answer 3), the books taken away (8) and the snail (day 9). In the tank problem it saw that the two pipes and the drain cancel out, so the tank never fills.
- Fifteen problems is a small set. Qwen showed a choice between two answers on two of them; response 1 was graded.
Coding
5/5Tested- Method
- The code block of each first answer was saved and run against the fixed checks (coding-checks.py).
- Tasks passed first time
- 10 of 10
- Broken versions caught by its tests
- 3 of 3
- Remark
- Scale 5: 9 or 10 pass first time.
Raw notes
The code block of each first answer was saved and run against the fixed checks (coding-checks.py).
10 of 10 passed first time. The test-writing task (co10) passed on the correct code and its tests caught 3 of 3 deliberately broken versions.
- The page copy lost the indentation of the code, so I restored it by hand without changing the logic. The tenth task was rebuilt from the two halves of Qwen's scrolling code view. The result depends on that hand repair, so I note it as a limit of the evidence. The explanations beside the code were not graded.
Following Instructions & Writing
5/5Tested- Method
- Fifteen briefs carrying 50 countable constraints were run once each in fresh chats.
- Constraints met
- 50 of 50
- Briefs passed
- 15 of 15
- Contradictory brief handled
- Yes
- Remark
- Scale 5: 95 percent or more of constraints met.
Raw notes
Fifteen briefs carrying 50 countable constraints were run once each in fresh chats. Words and sentences were counted with a tool.
50 of 50 constraints met and 15 of 15 briefs passed. The 60-word paragraph was exactly 60 words, the email ended with "Thanks, Alex" on its own line, and the list brief came back as exactly the fields asked for.
The contradictory brief was named as impossible. Qwen gave the 20-word paragraph, then padded the reply with commentary instead of writing a separate 100-word one, and said so.
- One brief added the idea of "filling in" that the source sentence did not have, and was still counted as a pass. One numbered list copied without its digits. Qwen showed a choice between two answers on one brief; response 1 was graded.
Context & Long-Document Memory
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Web Access & Citations
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Images, Files & Voice
5/5Tested- Method
- Eight points were run in fresh chats in Auto mode: a photo, a chart, a screenshot of text, a handwritten note, a maths problem as an image, a PDF, a spreadsheet and a table image.
- Points passed
- 8 of 8
- Points attempted
- 8
- Not run
- Image creation, sign image, audio
- Remark
- Scale 5: 11 or 12 of 12. Scored out of the points that ran, with the count stated.
Raw notes
Eight points were run in fresh chats in Auto mode: a photo, a chart, a screenshot of text, a handwritten note, a maths problem as an image, a PDF, a spreadsheet and a table image.
8 of 8 passed. The photo (a hand holding a clear empty cup), the chart (Q3 lowest, 90 thousand pounds), the notice (Saturday 10am to 4pm, 25p a day per item), the handwriting ("Hi there !!", with a note that the text runs vertically), the maths image (72 km/h with working), the PDF (six lanes, and it said the head lifeguard is not named), the spreadsheet (North 1,800, South sold the most units) and the table image (Route 12 at 08:50, no fare listed) were all right.
The image creation request, the sign image and the audio clip were not run, and voice was not offered because it needs a microphone. Uploads needed the plus menu, then Upload attachment. The first send of the handwriting image failed because the 4 MB file was still uploading, so I sent it again.
- The question text was typed by hand after each file was attached. Image creation, voice and the audio clip were not run.
Safety & Refusals
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Response Speed
3/5Tested- Method
- 26 answers from the accuracy and reasoning tests were timed with a script from the click on Send to the first words and to the end, in Auto mode.
- Median to first words
- 7.5 s
- Median to finish
- 13.4 s
- Slowest to finish
- 36.9 s
- Answers timed
- 26
- Remark
- Scale 3: under 8 seconds to first words or under 40 seconds to finish. The 4 band asks for under 4 seconds to first words.
Raw notes
26 answers from the accuracy and reasoning tests were timed with a script from the click on Send to the first words and to the end, in Auto mode. The timer polls, so times are approximate.
Median about 7.5 seconds to first words and 13.4 seconds to finish. Average 8.6 and 14.6. Fastest finish 6.5 and slowest 36.9, on the boiling point of water question on the first day. The first words ranged from 2.0 to 31.7 seconds.
Every timed answer arrived. Auto mode shows a thinking line, and often a web search, before the reply, which adds to the wait.
- Timed by script on one connection, and only 26 of the 45 answers have timings.
Pricing & Value
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Coverage
Tested
- 29 fact questions, including four false-premise questions
- 15 reasoning problems
- 10 coding tasks run against fixed checks
- 15 briefs carrying 50 countable constraints
- 8 image and file points
- Timing of 26 answers
- The human-check slider
Not tested or limited
- A human-check slider stopped chat twice, so the document, memory, web and safety tests were not run and count as 3
- The 30th fact question was never answered, so accuracy is 29 of 30 answered
- The page copy lost the indentation of the code, so I restored it by hand without changing the logic, and I rebuilt the tenth task from a scrolling code view
- Image creation, the sign image, the audio clip and voice were not run
- Auto mode searched the web while answering, so the fact answers were not from memory alone
- Qwen has no private mode, so the test chats sit in the account history
- Three answers came with a 'Which response do you prefer?' choice; response 1 was graded and neither was chosen
- Only 26 of the 45 answers have timings, taken by script and approximate
- Screenshots are standard resolution, not 2x
- No pricing page was captured
Still to test
- The document, memory, web and safety tests
- A paid plan, to see whether the slider returns
- Image creation, voice, the audio clip and the sign image
- 2x screenshots