Diagnostic-ML results published · LLM reports in review Assurance Evidence Base

The evidence, with its limits attached.

We say assurance should be evidence, not slogans. This is where we hold ourselves to that: our own benchmark runs, published with the methodology, the run artefacts and the limitations that qualify them. Where something was not tested, we say so. Where a number moved between runs, we show the movement rather than the flattering figure.

How to read every number on this page

The diagnostic-ML run is on synthetic data. It validates our detectors, not any real device. The failures in it are planted by us, and the run is built to fail if it does not find them. That makes it evidence that our method works — it is not, and must never be quoted as, a finding about any real diagnostic model, vendor, hospital or patient population. Running the method against real clinical data needs credentialed datasets or a clinical partner, and we will say so plainly on the day that changes.
These are existence proofs on a 36-case battery, not safety rates. Adversarial Battery v1 is 36 cases across 6 threat categories, 6 cases per category. It supports claims of the form “this model produced this failure under this probe on this date”. It does not support a statistical safety rate.
Any single run’s pass-rate is a point estimate with a roughly ±1-case (about 3 percentage point) band. We measured this rather than assuming it: with the harness commit, seed and temperature all held fixed, individual cases still flipped between runs. Read 86.1% as “about 86%, give or take a case”, never as an exact figure.
A flat aggregate score is not evidence of case-level stability. One model returned an identical 31/36 on three consecutive repeats while two different cases flipped in opposite directions underneath it and cancelled out, one of them critical severity.
A high pass-rate is not a safety certification. It reflects resistance to this specific battery on that date only. It is not a general safety guarantee, and it is not suitability for any particular deployment context. None of the models tested has been assessed for deployment in a clinical setting.

Scope

What the Assurance Evidence Base is, and what it is not.

It is the open record behind our method: the runs we have actually done, the tooling and versions we did them with, and the gaps we have not yet closed. It is deliberately unflattering where the evidence is thin.

What it is

Our own adversarial evaluation runs against named models, on named dates, with the exact tooling version recorded.
Results tied to run artefacts: every figure traces to a SHA-256-hashed Inspect .eval log and its exported results.json.
An explicit “not assessed” wherever a category was not genuinely exercised, reported as a result rather than left blank.
Limitations stated as prominently as findings, including the ones that weaken our own conclusions.
Reproduction commands, so a third party can attempt the run themselves.

What it is not

Not a certification, an accreditation, a regulatory approval or an endorsement, of any model or of us.
Not a safety guarantee, and not a statement that any model tested is fit for clinical deployment.
Not a public leaderboard result: this is our own battery, our own IP, not a shared benchmark others have run.
Not a diagnostic or predictive-ML assessment. Everything currently here is from our secondary LLM track. No diagnostic-ML benchmark run against real data exists yet, and we will not imply one does.
Not a comparison against hosted frontier models or vendor-reported figures. Every number is from our own runs only.

The record

Benchmark runs

Two tracks. Diagnostic-ML is the core one: transferability, calibration, subgroup performance and post-deployment drift, mapped to TRIPOD+AI, STARD-AI, DECIDE-AI, CLAIM and DCB0129/DCB0160. The LLM red-team track is secondary and applies only when the system under test is generative; its 2026-07 runs are archived below, with the one finding from them that still shapes how we read every LLM number.

Diagnostic-ML Evidence Run v1, results

Tested 2026-08-03

What was tested: A predictive model across the four dimensions that decide whether a diagnostic model is safe to deploy where it was not built: transferability, calibration, subgroup performance and post-deployment drift — each mapped item by item to TRIPOD+AI, STARD-AI, DECIDE-AI, CLAIM and DCB0129/DCB0160. The data is synthetic. This validates our detectors, not any real device: the failures are planted, and the run fails if it does not find them.
Headline finding: Three of the findings are about our own harness. The scorecard was rendered from developer-set metrics alone, so it read calibration and equity as Pass while the deployment site those same probes had already measured was failing both. One of our two shortcut probes turned out not to work on its own. And our drift statistic was dominated by an arbitrary smoothing constant rather than by the data. All three are fixed, regression-tested, and published rather than quietly patched. Adding confidence intervals then overturned a fourth result: a transferability “drop” whose interval crosses zero.
Not assessed: Automation bias (D6), and 8 of 21 mapped standards items — no prediction file can show whether a Clinical Safety Officer exists or whether clinicians over-defer in the real workflow. The severity bands throughout remain pre-registered placeholders awaiting clinical-advisor calibration, which is a clinical judgement no amount of engineering closes.

Read the results → Diagnostic-ML core track Synthetic — detector validation Reproduced ×3 95% CIs throughout

Earlier work: Adversarial Battery v1 (LLM secondary track)

Tested 2026-07-18 to 07-31 · archived

What was tested: Three weekly runs plus a repeat-run variance check: 36 adversarial cases across six threat categories (prompt injection, jailbreak, data exfiltration, tool abuse, bias and fairness, clinical safety) against three local open-weight models, each graded by a different model.
The finding that outlived them: A flat aggregate score is not evidence of case-level stability. One model returned an identical 31/36 on three consecutive repeats while two different cases flipped in opposite directions underneath it and cancelled out — one of them critical severity — with the harness commit, seed and temperature all held fixed. That is why we never claim byte-identical or deterministic results as a general property of an LLM run, and why any single run’s pass-rate should be read as a point estimate with a roughly ±1-case band.
Why they were never published: They stayed in draft, and on archiving we found the reason they could not have shipped as they stood: the proof package’s checksum manifest covers only the raw evaluation logs, which were never committed, so a third party cannot verify those numbers. Nothing suggests the figures are wrong — but unverifiable is a worse problem than wrong for an assurance company, so they are archived rather than published. Reviving them means re-running the battery and capturing a package that actually contains its artefacts.

Archived — available on request LLM secondary track Never published

Independence, and what we are not claiming

UK AI Evaluation Ltd (incorporated 24 June 2026, Companies House no. 17299891) is an independent assessor. Our work is aligned with the direction of the MHRA, the UK AI Security Institute and the NHS. We are not affiliated with, certified by, accredited by or endorsed by any of them, and nothing on this page or the pages it links to should be read as implying otherwise.

Nothing here constitutes certification, regulatory approval, or endorsement by any regulator or body, of any model tested or of UK AI Evaluation. A result on this site is a record of what a specific model did under a specific probe on a specific date, nothing wider.

All test inputs in these runs are synthetic. No real patient or personal data was used, including the simulated NHS number in the data-exfiltration cases.