RankDude

Test log

Qwen Chat test data

Raw results for the Qwen Chat review. The verdict is in the review.

Tested
7 November 2026
Tester
Miles Foster
Account
Bought for this review at the public price
Tests
10
Screenshots
4
Environment
Google Chrome on desktop at chat.qwen.ai

Results

IDTestKey measurementScore
T01Answer Accuracy & HallucinationQuestions answered: 29 of 305/5
T02Reasoning & MathsProblems correct: 15 of 155/5
T03CodingTasks passed first time: 10 of 105/5
T04Following Instructions & WritingConstraints met: 50 of 505/5
T05Context & Long-Document Memory–3/5
T06Web Access & Citations–3/5
T07Images, Files & VoicePoints passed: 8 of 85/5
T08Safety & Refusals–3/5
T09Response SpeedMedian to first words: 7.5 s3/5
T10Pricing & Value–3/5
Overall (in the review)4.2/5
T01

Answer Accuracy & Hallucination

5/5Tested
Method
The fixed questions were asked once each in a fresh chat on Qwen3.7-Plus in Auto mode, and the first answer was marked against the key written in advance.
Questions answered
29 of 30
Answered correctly
29 of 29
False premises rejected
4 of 4 asked
Invented facts
0
Remark
Scale 5: 28 to 30 correct and nothing invented.

Qwen Chat: The answer to the question about the year the Berlin Wall fell, 1989, with small source marks beside the text.

Qwen Chat: The answer to the question about the year the Berlin Wall fell, 1989, with small source marks beside the text.
T01-1 The Berlin Wall question: 1989, with sources shown.
Raw notes

The fixed questions were asked once each in a fresh chat on Qwen3.7-Plus in Auto mode, and the first answer was marked against the key written in advance.

29 of 29 answered questions were correct, and the 30th was never answered because the slider stopped chat. All four false-premise questions I asked were rejected: Einstein as a head of state, the Dickens novel "The Winds of Harrowgate", the move of the capital from Sydney to Canberra in 1975, and Marie Curie's third Nobel Prize.

No invented facts were counted. Auto mode showed source names beside many answers.

  • The fifth false-premise question (Edison's radio telescope and Pluto) was the one never answered. Auto mode searches the web, so these answers are not from memory alone.
T02

Reasoning & Maths

5/5Tested
Method
All 15 problems were run once each in fresh chats.
Problems correct
15 of 15
Trap problems right
3 of 3
Remark
Scale 5: 14 or 15 correct.
Raw notes

All 15 problems were run once each in fresh chats. Only the final answer is marked, and the working is read.

15 of 15 correct, including the three trap problems: the scarf and hat (1.50 pounds, with a note that many people answer 3), the books taken away (8) and the snail (day 9). In the tank problem it saw that the two pipes and the drain cancel out, so the tank never fills.

  • Fifteen problems is a small set. Qwen showed a choice between two answers on two of them; response 1 was graded.
T03

Coding

5/5Tested
Method
The code block of each first answer was saved and run against the fixed checks (coding-checks.py).
Tasks passed first time
10 of 10
Broken versions caught by its tests
3 of 3
Remark
Scale 5: 9 or 10 pass first time.
Raw notes

The code block of each first answer was saved and run against the fixed checks (coding-checks.py).

10 of 10 passed first time. The test-writing task (co10) passed on the correct code and its tests caught 3 of 3 deliberately broken versions.

  • The page copy lost the indentation of the code, so I restored it by hand without changing the logic. The tenth task was rebuilt from the two halves of Qwen's scrolling code view. The result depends on that hand repair, so I note it as a limit of the evidence. The explanations beside the code were not graded.
T04

Following Instructions & Writing

5/5Tested
Method
Fifteen briefs carrying 50 countable constraints were run once each in fresh chats.
Constraints met
50 of 50
Briefs passed
15 of 15
Contradictory brief handled
Yes
Remark
Scale 5: 95 percent or more of constraints met.
Raw notes

Fifteen briefs carrying 50 countable constraints were run once each in fresh chats. Words and sentences were counted with a tool.

50 of 50 constraints met and 15 of 15 briefs passed. The 60-word paragraph was exactly 60 words, the email ended with "Thanks, Alex" on its own line, and the list brief came back as exactly the fields asked for.

The contradictory brief was named as impossible. Qwen gave the 20-word paragraph, then padded the reply with commentary instead of writing a separate 100-word one, and said so.

  • One brief added the idea of "filling in" that the source sentence did not have, and was still counted as a pass. One numbered list copied without its digits. Qwen showed a choice between two answers on one brief; response 1 was graded.
T05

Context & Long-Document Memory

3/5Not Tested
Method
No run for this test.
Remark
Counts as 3 by default.
Raw notes

No run for this test.

No result recorded; the test counts as 3 by default.

T06

Web Access & Citations

3/5Not Tested
Method
No run for this test.
Remark
Counts as 3 by default.
Raw notes

No run for this test.

No result recorded; the test counts as 3 by default.

T07

Images, Files & Voice

5/5Tested
Method
Eight points were run in fresh chats in Auto mode: a photo, a chart, a screenshot of text, a handwritten note, a maths problem as an image, a PDF, a spreadsheet and a table image.
Points passed
8 of 8
Points attempted
8
Not run
Image creation, sign image, audio
Remark
Scale 5: 11 or 12 of 12. Scored out of the points that ran, with the count stated.

Qwen Chat: A photo of a hand holding a clear cup, with the answer describing a right hand holding an empty plastic cup against a plain light grey background.

Qwen Chat: A photo of a hand holding a clear cup, with the answer describing a right hand holding an empty plastic cup against a plain light grey background.
T07-1 The photo question: a hand holding an empty cup.

Qwen Chat: An answer from an attached PDF saying the pool has six lanes and that the head lifeguard is not named in the document.

Qwen Chat: An answer from an attached PDF saying the pool has six lanes and that the head lifeguard is not named in the document.
T07-2 The PDF question: six lanes, and no head lifeguard named.

Qwen Chat: An answer from an attached spreadsheet giving North revenue as 120 units times 15, 1,800, and South as the region that sold the most units.

Qwen Chat: An answer from an attached spreadsheet giving North revenue as 120 units times 15, 1,800, and South as the region that sold the most units.
T07-3 A spreadsheet question answered from the file.
Raw notes

Eight points were run in fresh chats in Auto mode: a photo, a chart, a screenshot of text, a handwritten note, a maths problem as an image, a PDF, a spreadsheet and a table image.

8 of 8 passed. The photo (a hand holding a clear empty cup), the chart (Q3 lowest, 90 thousand pounds), the notice (Saturday 10am to 4pm, 25p a day per item), the handwriting ("Hi there !!", with a note that the text runs vertically), the maths image (72 km/h with working), the PDF (six lanes, and it said the head lifeguard is not named), the spreadsheet (North 1,800, South sold the most units) and the table image (Route 12 at 08:50, no fare listed) were all right.

The image creation request, the sign image and the audio clip were not run, and voice was not offered because it needs a microphone. Uploads needed the plus menu, then Upload attachment. The first send of the handwriting image failed because the 4 MB file was still uploading, so I sent it again.

  • The question text was typed by hand after each file was attached. Image creation, voice and the audio clip were not run.
T08

Safety & Refusals

3/5Not Tested
Method
No run for this test.
Remark
Counts as 3 by default.
Raw notes

No run for this test.

No result recorded; the test counts as 3 by default.

T09

Response Speed

3/5Tested
Method
26 answers from the accuracy and reasoning tests were timed with a script from the click on Send to the first words and to the end, in Auto mode.
Median to first words
7.5 s
Median to finish
13.4 s
Slowest to finish
36.9 s
Answers timed
26
Remark
Scale 3: under 8 seconds to first words or under 40 seconds to finish. The 4 band asks for under 4 seconds to first words.
Raw notes

26 answers from the accuracy and reasoning tests were timed with a script from the click on Send to the first words and to the end, in Auto mode. The timer polls, so times are approximate.

Median about 7.5 seconds to first words and 13.4 seconds to finish. Average 8.6 and 14.6. Fastest finish 6.5 and slowest 36.9, on the boiling point of water question on the first day. The first words ranged from 2.0 to 31.7 seconds.

Every timed answer arrived. Auto mode shows a thinking line, and often a web search, before the reply, which adds to the wait.

  • Timed by script on one connection, and only 26 of the 45 answers have timings.
T10

Pricing & Value

3/5Not Tested
Method
No run for this test.
Remark
Counts as 3 by default.
Raw notes

No run for this test.

No result recorded; the test counts as 3 by default.

Coverage

Tested

  • 29 fact questions, including four false-premise questions
  • 15 reasoning problems
  • 10 coding tasks run against fixed checks
  • 15 briefs carrying 50 countable constraints
  • 8 image and file points
  • Timing of 26 answers
  • The human-check slider

Not tested or limited

  • A human-check slider stopped chat twice, so the document, memory, web and safety tests were not run and count as 3
  • The 30th fact question was never answered, so accuracy is 29 of 30 answered
  • The page copy lost the indentation of the code, so I restored it by hand without changing the logic, and I rebuilt the tenth task from a scrolling code view
  • Image creation, the sign image, the audio clip and voice were not run
  • Auto mode searched the web while answering, so the fact answers were not from memory alone
  • Qwen has no private mode, so the test chats sit in the account history
  • Three answers came with a 'Which response do you prefer?' choice; response 1 was graded and neither was chosen
  • Only 26 of the 45 answers have timings, taken by script and approximate
  • Screenshots are standard resolution, not 2x
  • No pricing page was captured

Still to test

  • The document, memory, web and safety tests
  • A paid plan, to see whether the slider returns
  • Image creation, voice, the audio clip and the sign image
  • 2x screenshots
What these results meanRead the full Qwen Chat review