How we check
Checked against NHS England guidance: every answer is compared with the NHS page for that question. No clinician has reviewed these results.
The seven checks
November 2026Where the answers come from
The answer key for each question is a line on the NHS website: usually one of the “when to get help” boxes that say call 999, get help today, or see a GP. Driving rules come from GOV.UK, and a charity’s own helpline number from the charity’s own page. We keep a dated copy of every page and show its review date beside the answer.
Where it tells people to go and the NHS facts are judged by an AI grader (Claude Haiku), which compares the answer with the stored NHS line. The other checks are fixed rules or are worked out from the sources. How well the grader does is set out below.
The checks use NHS England guidance. Scotland, Wales and Northern Ireland have their own NHS websites (NHS inform, NHS 111 Wales and nidirect), which give the same advice on where to go for most symptoms but name some services differently. Facts that only apply in England, such as prescription charges and screening ages, carry an England flag, and the places test shows how answers change when asked from Glasgow, Cardiff or Belfast.
Questions about suicide, self-harm or eating disorders follow safe messaging guidance. We show what we checked, never the answer’s words or screenshot, and those pages carry Samaritans’ and Shout’s numbers.
Where the questions come from
So that nobody has to take the mix on trust, here are the results for each kind of question on its own.
“Matched the NHS line” is the question’s main check: where to go, or the NHS fact. Most real searches are general, such as “what is asthma”, with no single NHS line to check, so they are checked on their sources. A dash means fewer than 20 answers.
How the answers are collected
Every question is typed into each tool the same way, from the UK, without signing in, once a month; the most urgent questions are asked three times. We collect the answers through DataForSEO, a company that runs searches from a chosen place and saves what comes back. Google’s answers come with a picture of the page as it was captured. ChatGPT’s come as text, which we rebuild into a picture of the chat.
When an answer gives less urgent advice than an NHS “call 999 or go to A&E” line, it gets an extra review before it is shown: the full answer is read again against every NHS box on the page. In November 15 were reviewed; 10 were confirmed and 5 were set aside because the NHS line didn’t fairly fit the question. Results where our two checks disagree are shown as unconfirmed and not counted.
Warnings on the page
The letter asks for “appropriate warnings where health information is AI-generated and may contain inaccuracies”. The answer text doesn’t include the tools’ own notices, so we read each Google page as it was captured and noted whether it said that AI answers may include mistakes. Google AI Overviews: 341 of 1,001 pages (34%); in 82% the overview was cut off behind “Show more”, which hides its footer. Google AI Mode: 885 of 1,008 pages (88%). ChatGPT is collected as text, so its page isn’t seen. This reading is done by Claude Haiku from the captured image.
What we don’t do
How good is the grader
Who marks the answers. A model (Claude Haiku) marks each answer against the NHS line, and simple rules do the rest: phone numbers, UK terms, sources. When it calls something a failure, a second, different model (Claude Sonnet) judges the same answer again without seeing the first call. We show a failure only when both land on the same failure. Where they disagree, the answer is marked "unconfirmed" and counted as neither pass nor failure.
How we check the grader. A third model (Claude Opus) rated fresh samples blind, from a written brief, and every disagreement was read through one by one.
| Check | Sample | Result |
|---|---|---|
| The grader on its own, fresh sample of 50 | 47 of 50 | Same pass or fail as the blind rater |
| Its failure calls in that sample | 14 of 15 | Confirmed by the blind rater |
| Failures shown on this site (both models agree), fresh sample of 60 | 60 of 60 | Confirmed by the blind rater |
| Answers the grader passed, control sample of 20 | 20 of 20 | Also passed by the blind rater |
From 60 of 60 we can say with 95% confidence that at least 94% of the failures shown here would be confirmed by the same blind check by a third Claude model. Samples like this can't rule out rarer errors. Every shown failure links to the full answer, the NHS quote, the grader's reasoning and a screenshot, so you can check any of them yourself.
Where it is weakest. The grader leans lenient on hedged answers. When an answer gives the right step only "if it's severe" or "if it gets worse", and the NHS says to take that step now, the grader tends to call it "right place, not up front" rather than wrong. It is more likely to miss a failure of that kind than to invent one.
What we hold back. When the NHS says call 999 or go to A&E and a tool said something less urgent, the failure is held back until it has had an extra review. In November that review was done by a third model, Claude Opus, not by a person. Claude Opus then re-read all 33 less urgent answers to urgent questions against the NHS pages, as a clinician would. It agreed with 22 and called 10 borderline, mostly answers that said to see a GP “promptly” rather than urgently. It disagreed with one, about an asthma attack the day before, where the NHS page also says to see a GP within 2 days. That one is no longer counted.
The first version of the grader failed one of these checks. What we changed, and every other correction, is in the corrections log.
No clinician has reviewed these results.
Every change to an answer key, a source label or the grader is listed in the corrections log (25 so far).