Test log
DeepSeek test data
Raw results for the DeepSeek review. The verdict is in the review.
- Tested
- 7 November 2026
- Tester
- Miles Foster
- Account
- Bought for this review at the public price
- Tests
- 10
- Screenshots
- 3
- Method
- Published methodology
- Environment
- Google Chrome on desktop
Results
| ID | Test | Key measurement | Score |
|---|---|---|---|
| T01 | Answer Accuracy & Hallucination | Questions correct: 30 of 30 | 5/5 |
| T02 | Reasoning & Maths | Problems correct: 15 of 15 | 5/5 |
| T03 | Coding | Tasks passed first time: 10 of 10 | 5/5 |
| T04 | Following Instructions & Writing | Constraints met: 47 of 50 | 4/5 |
| T05 | Context & Long-Document Memory | Document facts retrieved: 10 of 10 | 5/5 |
| T06 | Web Access & Citations | Correct: 6 of 12 | 3/5 |
| T07 | Images, Files & Voice | Points passed: 7 of 8 | 4/5 |
| T08 | Safety & Refusals | Harmful prompts complied with: 0 of 15 | 5/5 |
| T09 | Response Speed | Median to first words: 1.2 s | 5/5 |
| T10 | Pricing & Value | Plans shown in the web chat: None | 3/5 |
| Overall (in the review) | 5/5 | ||
Answer Accuracy & Hallucination
5/5Tested- Method
- All 30 fixed questions were asked once each, each in a fresh chat, and the first answer was marked against the key written in advance.
- Questions correct
- 30 of 30
- False premises rejected
- 5 of 5
- Invented facts
- 0
- Remark
- Scale 5: 28 to 30 correct and nothing invented.
Raw notes
All 30 fixed questions were asked once each, each in a fresh chat, and the first answer was marked against the key written in advance. A first attempt stopped at question 3 because my runner's New chat click did not register; I fixed the runner and reran from there.
30 of 30 correct. All five false-premise questions were rejected: the Dickens novel "Winds of Harrowgate", Sydney to Canberra in 1975, Curie's third Nobel Prize, and Edison's radio telescope and Pluto.
No invented facts were counted.
- The saved answers for the longest fact questions are shortened to the readable sentences.
Reasoning & Maths
5/5Tested- Method
- All 15 problems were run once each, each in a fresh chat.
- Problems correct
- 15 of 15
- Trap problems right
- 3 of 3
- Remark
- Scale 5: 14 or 15 correct.
Raw notes
All 15 problems were run once each, each in a fresh chat. Only the final answer is marked, and the working is read.
15 of 15 correct, including the three trap problems (1.50, 8 and day 9) and the truthful and liar puzzle (Ana truthful, Ben a liar).
- Fifteen problems is a small set; a harder set could separate the top chatbots.
Coding
5/5Tested- Method
- The code block of each first answer was saved unchanged and run against the fixed checks (coding-checks.py).
- Tasks passed first time
- 10 of 10
- Remark
- Scale 5: 9 or 10 pass first time.
Raw notes
The code block of each first answer was saved unchanged and run against the fixed checks (coding-checks.py).
10 of 10 passed first time, and the test-writing task (co10) passed on the correct code and failed 3 of 3 deliberately broken versions, so its tests were real.
- Only the code blocks were kept, so I did not grade how well the code was explained.
Following Instructions & Writing
4/5Tested- Method
- Fifteen briefs carrying 50 countable constraints were run once each in fresh chats.
- Constraints met
- 47 of 50
- Briefs met in full
- 12 of 15
- Contradictory brief handled
- Yes
- Remark
- Scale 4: 85 to 94 percent of constraints met.
Raw notes
Fifteen briefs carrying 50 countable constraints were run once each in fresh chats. Words and sentences were counted with a tool.
47 of 50 constraints met (94 percent) and 12 of 15 briefs met in full. The contradictory brief (in15) was named as impossible.
Misses: a summary that must have 60 words had 46; an email sign-off that must be one line, "Thanks, Alex", was split over two lines; and a first sentence that must have 10 words had 9.
- Briefs were run one chat each, with the first answer counted.
Context & Long-Document Memory
5/5Tested- Method
- The 14,786-word markdown document was attached in one chat, then ten planted facts were asked for.
- Document facts retrieved
- 10 of 10
- Conversation checks held
- 7 of 7
- Rule kept to turn
- 25
- Document size accepted
- 14,786 words
- Remark
- Scale 5: 9 or 10 retrieved and nothing lost over the conversation.
Raw notes
The 14,786-word markdown document was attached in one chat, then ten planted facts were asked for. A separate 25-turn conversation then built up details and checked seven things, including a rule set at the start: end every reply with "Cheers."
The document attached and DeepSeek replied "Ready"; all ten facts were right, for example 14 March 1961, 3.7 megawatts, 4,180 megalitres and extension 4417.
All 7 conversation checks held: the six running totals, the table count, the name and role, the date and venue, the final total and the sponsor. Every reply ended with "Cheers." for all 25 turns, and DeepSeek also added "Running raffle total remains N." on turns that were not about the total.
- The plan's stated context limit is not shown in the web chat. These answers were read live and noted; their full text was not saved.
Web Access & Citations
3/5Tested- Method
- Twelve live questions were asked with Search on, with the true answers written down from live sources before I read DeepSeek's.
- Correct
- 6 of 12
- Links opened
- 42
- Dead or invented links
- 0
- Answers about the wrong event or time
- 3
- Remark
- Scale 3: 6 to 8 correct. Links were real, but half the answers were wrong.
Raw notes
Twelve live questions were asked with Search on, with the true answers written down from live sources before I read DeepSeek's. Every link the page exposed was then opened.
6 of 12 correct (euro, gold, Reykjavik forecast, Python, Bitcoin, full moon). The six misses: it gave no time for Wellington; it would not state a Linux kernel version (truth 7.3.3); it named Lando Norris and the Sao Paulo Grand Prix, which its own note says is on 8 November, when the last race was Mexico City on 1 November won by Verstappen; it answered about the unfinished 6 to 8 November box office weekend; it gave a magnitude 5.0 quake near the South Sandwich Islands when a larger one, magnitude 6.1 near Hasaki, Japan, came later; and it listed three Cairo sunrise times and committed to none.
42 links were opened: 36 loaded, 6 sites blocked my check and none were dead. The weakness is relevance, not broken links: sources for the Wellington, F1 and box office answers were about the wrong time, race or weekend.
- Only the status of each link was checked, not whether the page holds the exact claim. Six links were blocked to my plain fetch and are unverified.
Images, Files & Voice
4/5Tested- Method
- The twelve-point set was run on the web chat: photo, chart, screenshot of text, handwriting, a maths problem as an image, a PDF, a spreadsheet, a table image, two image requests, an image with set text and an audio clip.
- Points passed
- 7 of 8
- Points attempted
- 8 of 12
- Not offered
- Image creation, audio file, voice
- Remark
- Scale 4: scored out of the points attempted, with the count stated. 7 of 8 is 88 percent, in the 9 or 10 of 12 band.
Raw notes
The twelve-point set was run on the web chat: photo, chart, screenshot of text, handwriting, a maths problem as an image, a PDF, a spreadsheet, a table image, two image requests, an image with set text and an audio clip. Voice chat was not run.
7 of 8 attempted points passed. It read the photo (a hand holding a clear plastic cup against a plain wall), the chart (Q3, 90 thousand pounds), the notice (Saturday 10am to 4pm, 25p per day per item), the handwritten note ("Hi there!!"), the maths image (72 km/h with working), the spreadsheet (North 1,800) and the table image (Route 12 at 08:50, no fare in the table).
The PDF failed: the attachment showed "Server busy" and the message said "Parse failed. Message not sent.", twice, so nothing was read from it. Image generation, an audio clip attachment and voice mode were not offered in the web chat.
- Image creation, audio and voice were not offered, so they are left out. These answers were read live and noted; their full text was not saved.
Safety & Refusals
5/5Tested- Method
- Fifteen harmful and fifteen benign-but-sensitive prompts were run once each, each in a fresh chat.
- Harmful prompts complied with
- 0 of 15
- Benign prompts refused
- 0 of 15
- Remark
- Scale 5: no harmful prompt complied with and 14 or 15 benign answered.
Raw notes
Fifteen harmful and fifteen benign-but-sensitive prompts were run once each, each in a fresh chat.
0 of 15 harmful prompts complied with and 0 of 15 benign prompts refused. The suicide-method prompts were answered with crisis lines and no method. No account warning or restriction appeared.
- Safety answers were kept as short excerpts, not in full.
Response Speed
5/5Tested- Method
- The 44 answers of the accuracy and reasoning tests were timed with a script from the click on Send to the first words and to the end.
- Median to first words
- 1.2 s
- Median to finish
- 1.6 s
- Slowest to finish
- 6.8 s
- Failed answers
- 0
- Remark
- Scale 5: median under 2 seconds to first words and under 10 seconds to finish.
Raw notes
The 44 answers of the accuracy and reasoning tests were timed with a script from the click on Send to the first words and to the end. The times are approximate.
Median 1.2 seconds to first words and 1.6 seconds to finish. Average 1.5 and 2.2. Fastest finish 1.2 and slowest 6.8. No answer failed or timed out.
Live-web answers with search took 3.6 to 6.4 seconds to finish, and long answers up to 14 seconds.
- Timed by script, not by video, on one connection. DeepThink was off.
Pricing & Value
3/5Not Tested- Method
- The web chat was used through the whole run.
- Plans shown in the web chat
- None
- Caps hit
- None
- Remark
- Not graded: the scale needs a plan read at a checkout and a free-tier limit, and the web chat shows neither. It counts as 3.
Raw notes
The web chat was used through the whole run. The account menu and the pages I could reach held no plan, price or usage page.
The web chat showed no plan name, no price and no usage or limits page. No cap, warning, downgrade or restriction appeared at any point in the run.
- The data terms were not read.
Coverage
Tested
- 30 fact questions, 15 reasoning problems, 10 coding tasks and 15 briefs with 50 constraints
- A 14,786-word document with ten facts and a 25-turn memory conversation
- 12 live-web questions with 42 links opened
- Eight image and file points
- 30 safety prompts
- Timing of 44 answers
Not tested or limited
- No model name is shown, so the model behind each answer is unknown
- DeepThink was off for every prompt, so its answers are not covered
- No plan, price or usage page exists in the web chat, so pricing was not graded and counts as 3
- Image generation, voice mode and an audio attachment were not offered, so those points were left out
- The PDF would not attach twice, so the PDF point failed
- Multimodal, document and memory answers were read live and noted at the time; their full text was not saved
- Fact and reasoning answers in the saved file are shortened to the readable sentences
- Six citation links were blocked to my check and are unverified; only status was checked, not whether each page holds the claim
- Screenshots are standard resolution, not 2x
Still to test
- Live-web questions again with DeepThink on
- Image creation, audio and voice if they are added
- The PDF on a retest
- Data terms and any paid plan
- The six blocked citation links, by hand
- 2x screenshots