Test log
Kimi test data
Raw results for the Kimi review. The verdict is in the review.
- Tested
- 7 November 2026
- Tester
- Miles Foster
- Account
- Bought for this review at the public price
- Tests
- 10
- Screenshots
- 5
- Method
- Published methodology
- Environment
- Google Chrome on desktop at kimi.ai
Results
| ID | Test | Key measurement | Score |
|---|---|---|---|
| T01 | Answer Accuracy & Hallucination | Questions correct: 30 of 30 | 5/5 |
| T02 | Reasoning & Maths | Problems correct: 15 of 15 | 5/5 |
| T03 | Coding | Tasks passed first time: 10 of 10 | 5/5 |
| T04 | Following Instructions & Writing | Constraints met: 50 of 50 | 5/5 |
| T05 | Context & Long-Document Memory | – | 3/5 |
| T06 | Web Access & Citations | – | 3/5 |
| T07 | Images, Files & Voice | Points passed: 8 of 8 | 5/5 |
| T08 | Safety & Refusals | – | 3/5 |
| T09 | Response Speed | Median to first words: 2.4 s | 4/5 |
| T10 | Pricing & Value | Busy notices: Both days | 3/5 |
| Overall (in the review) | 4.3/5 | ||
Answer Accuracy & Hallucination
5/5Tested- Method
- All 30 fixed questions were asked once each in a fresh chat in Instant High mode, and the first answer was marked against the key written in advance.
- Questions correct
- 30 of 30
- False premises rejected
- 5 of 5
- Invented facts
- 0
- Remark
- Scale 5: 28 to 30 correct and nothing invented.
Raw notes
All 30 fixed questions were asked once each in a fresh chat in Instant High mode, and the first answer was marked against the key written in advance.
30 of 30 correct. All five false-premise questions were rejected: Einstein as a head of state, the Dickens novel "The Winds of Harrowgate", the move of the capital from Sydney to Canberra in 1975, Marie Curie's third Nobel Prize, and Edison's radio telescope and Pluto.
No invented facts were counted.
Reasoning & Maths
5/5Tested- Method
- All 15 problems were run once each in fresh chats.
- Problems correct
- 15 of 15
- Trap problems right
- 3 of 3
- Remark
- Scale 5: 14 or 15 correct.
Raw notes
All 15 problems were run once each in fresh chats. Only the final answer is marked, and the working is read.
15 of 15 correct, including the three trap problems: the scarf and hat (1.50 pounds), the books taken away (8) and the snail (day 9). In the tank problem it saw that the drain cancels the two pipes, so the tank never fills.
- Fifteen problems is a small set; a harder set could separate the top chatbots.
Coding
5/5Tested- Method
- The code block of each first answer was saved unchanged and run against the fixed checks (coding-checks.py).
- Tasks passed first time
- 10 of 10
- Broken versions caught by its tests
- 3 of 3
- Remark
- Scale 5: 9 or 10 pass first time.
Raw notes
The code block of each first answer was saved unchanged and run against the fixed checks (coding-checks.py).
10 of 10 passed first time. The test-writing task (co10) passed on the correct code and its tests caught 3 of 3 deliberately broken versions, so its tests were real.
- The explanations beside the code were not graded.
Following Instructions & Writing
5/5Tested- Method
- Fifteen briefs carrying 50 countable constraints were run once each in fresh chats.
- Constraints met
- 50 of 50
- Briefs met in full
- 15 of 15
- Contradictory brief handled
- Yes
- Remark
- Scale 5: 95 percent or more of constraints met.
Raw notes
Fifteen briefs carrying 50 countable constraints were run once each in fresh chats. Words and sentences were counted with a tool.
50 of 50 constraints met and 15 of 15 briefs met in full. The 60-word paragraph was exactly 60 words, the email ended with "Thanks, Alex" on its own final line, and the first sentence of the 10-word brief had exactly 10 words. The contradictory brief was named as impossible, and both versions were given.
- Briefs were run one chat each, with the first answer counted. Four of them ran on the second day, after the busy notice eased.
Context & Long-Document Memory
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Web Access & Citations
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Images, Files & Voice
5/5Tested- Method
- Eight points were run in fresh chats in Instant High mode: a photo, a chart, a screenshot of text, a handwritten note, a maths problem as an image, a PDF, a spreadsheet and a table image.
- Points passed
- 8 of 8
- Points attempted
- 8
- Not offered
- Image creation
- Remark
- Scale 5: 11 or 12 of 12. Scored out of the points that ran, with the count stated.
Raw notes
Eight points were run in fresh chats in Instant High mode: a photo, a chart, a screenshot of text, a handwritten note, a maths problem as an image, a PDF, a spreadsheet and a table image. I also asked for two images to be made.
8 of 8 passed. The photo (a hand holding a clear glass), the chart (Q3 lowest, 90 thousand pounds), the notice (Saturday 10am to 4pm, 25p a day per item), the handwriting ("Hi there!!"), the maths image (72 km/h with working), the PDF (six lanes, and it said no head lifeguard is named), the spreadsheet (North 1,800, South sold the most units) and the table image (Route 12 at 08:50, no fare shown) were all right.
Kimi said it cannot generate images from scratch and showed existing pictures from the web instead, so image creation was not offered. For the photo, the question text did not travel with the image and Kimi described the photo anyway. For the handwriting, two copies of the image were attached by mistake and the answer was still right.
- Voice and an audio clip were not run. The composer dropped text set by script after an upload, so the question text was typed by hand for seven of the eight points.
Safety & Refusals
3/5Not Tested- Method
- No run for this test.
- Remark
- Counts as 3 by default.
Raw notes
No run for this test.
No result recorded; the test counts as 3 by default.
Response Speed
4/5Tested- Method
- The 45 answers of the accuracy and reasoning tests were timed with a script from the click on Send to the first words and to the end, in Instant High mode.
- Median to first words
- 2.4 s
- Median to finish
- 5.7 s
- Slowest to finish
- 18.2 s
- Failed answers
- 0
- Remark
- Scale 4: under 4 seconds to first words and under 20 seconds to finish. The 5 band asks for under 2 seconds to first words, which was not measured.
Raw notes
The 45 answers of the accuracy and reasoning tests were timed with a script from the click on Send to the first words and to the end, in Instant High mode. The timer polls, so times are approximate.
Median about 2.4 seconds to first words and 5.7 seconds to finish. Average 2.9 and 7.1. Fastest finish 3.7 and slowest 18.2, on the Sydney to Canberra false-premise question. No answer failed or timed out.
The shortest first-words reading was 2.0 seconds, so no answer was timed under 2 seconds.
- Timed by script on one connection. Answers show a short thinking line before the reply.
Pricing & Value
3/5Not Tested- Method
- The free plan was used until a busy notice stopped chat.
- Busy notices
- Both days
- Reset time shown
- None
- Remark
- No plan page was read, so the grade counts as 3 by default.
Raw notes
The free plan was used until a busy notice stopped chat. The notice, the Upgrade button and the sidebar were read.
Chat stopped on both days with the notice "Too many people are chatting with Kimi, a subscription will grant you priority access", with buttons Got it and Upgrade. No reset time was given and no message limit was shown. The home screen also shows an Upgrade your plan button and an Upgrade button beside the account name.
- No pricing page was captured, so no plan prices are given.
Coverage
Tested
- 30 fact questions, including all five false-premise questions
- 15 reasoning problems
- 10 coding tasks run against fixed checks
- 15 briefs carrying 50 countable constraints
- 8 image and file points, and an image creation request
- Timing of 45 answers
- The busy notice
Not tested or limited
- A busy notice stopped chat twice, so the document, memory, web and safety tests were not run and count as 3
- Instant High mode only; no model name is shown in the app
- Image creation was not offered, and the sign image, audio clip and voice were not run
- The question text was typed by hand after each file upload because the composer dropped scripted text
- Timings are script timers and approximate
- Screenshots are standard resolution, not 2x
- No pricing page was captured
Still to test
- The document, memory, web and safety tests
- A paid plan without the busy notice
- Voice, the audio clip and the sign image
- 2x screenshots