AI Confidence: When Models Are Confidently Wrong

A benchmark that scores self-assessment

The RadLE 2.0 leaderboard went live on 19 July 2026. The benchmark from the CRASH Lab at Ashoka University does not only check whether an AI model reaches the right diagnosis. It checks whether the confidence behind that answer is justified. The result is uncomfortable.

Several leading commercial models produce a substantial number of misdiagnoses with high confidence. Open-weight and medical-specific models try to answer nearly every case rather than hand it over, and are frequently wrong at medium to high confidence.

No model leads across the board. Claude Fable 5 comes first on overall reliability, Gemini 3 Pro on raw accuracy, and Meta Muse Spark 1.1 is best at recognising its own limits. Ask for the best model and you get three different answers, depending on which property you measure.

Safety and handover readiness

RadLE 2.0 breaks reliability into several scores instead of one accuracy figure. Two of them matter in production: the safety index and handover readiness. Both penalise a behaviour that classic leaderboards reward, which is guessing.

The safety index weights errors by confidence. A wrong answer given with high certainty costs considerably more points than a wrong answer with admitted uncertainty. Handover readiness measures something different: whether a system uses its own confidence to decide between handling a case autonomously and passing it to a specialist.

The index combines three quantities: the reliability of the autonomously handled cases, the share of cases handled autonomously, and the share of likely errors correctly deferred. A system that hands over everything scores just as badly as one that never hands over.

Why the numbers are so far apart

The earlier RadLE v1 from September 2025 shows how hard the test is. It consists of 50 deliberately difficult spot-diagnosis cases from CT, MRI and radiography. The gap between human and machine was considerably wider there than it is today.

What stands out is how little extra compute bought. For GPT-5 the high reasoning setting lifted accuracy by a single percentage point at six times the latency. 65.6 seconds instead of 10.5 for one point.

The authors sort the errors into four classes. Perceptual errors such as under-detection, over-detection and mislocalisation. Interpretive errors, which include settling on a diagnosis too early. Communication errors, where the described observations and the conclusion contradict each other. And cognitive biases such as anchoring and availability effects.

Calibration becomes a legal duty

What the benchmark measures, the EU AI Act requires by law from August 2026. AI-based diagnostic software and image analysis count as high-risk systems automatically, because they are a safety component under the Medical Device Regulation. Systems already running are covered too.

The deadlines come in two stages. From August 2026 the high-risk requirements apply alongside the MDR, and by August 2027 they apply in full within conformity assessment. The notified body then reviews both rulebooks in one procedure.

In Europe the rollout is well ahead of the debate. The German hospital group Asklepios has completed its radiology AI rollout across 28 hospitals, where more than 35,000 CT and X-ray images are screened each month. At Unfallkrankenhaus Berlin a retrospective study raised the detection rate for intracranial haemorrhage by 12 percent.

The point is not that these systems are bad. They are narrowly scoped, clinically validated and solve defined tasks. The point is that the general-purpose language models now moving into every piece of software have no such validation, and still sound as though they do.

The pattern behind every agent approval

Radiology is simply the test case with the clearest consequences. The same pattern hits any company running an AI agent for longer than a single answer. The more steps an agent takes without asking, the more expensive a miscalibrated confidence signal becomes.

The mechanism is well understood. Alignment training degrades calibration systematically, because it rewards answers that sound certain regardless of whether the model knows the answer.

Benchmarks that reward abstention therefore produce a different ranking than pure accuracy boards. For production work the second list is the more honest one. A model that loses points there because it guesses less often is frequently the better one in operation.

And calibration is not a model property you can wait for. It comes out of the eval suite, the prompt scaffolding, the retrieval design and the interface. The deploying team builds it, not the vendor.

Challenges and risks

The benchmark is a step forward, not a licence for reverse conclusions. The authors state the limits clearly themselves, and reading the numbers as a purchasing decision stretches them too far.

• Small dataset. RadLE v1 covers 50 cases, deliberately at the upper end of difficulty. That creates a spectrum bias against everyday clinical work.
• Narrow scope. Only CT, MRI and radiography were tested. Ultrasound, mammography and nuclear medicine are missing.
• Single images instead of series. Individual images were scored, and clinical context such as prior findings and lab values was deliberately withheld to isolate visual reasoning.
• Subjective partial credit. Scoring uses an ordinal scale without a standardised ontology.
• No paper yet. The full RadLE 2.0 paper with methodology and statistics is still pending. So far there is a technical report.

There is also a problem that good calibration does not solve. A system that hands over often relieves nobody. It can even add work, because someone has to process the deferred cases and then judge whether the deferral was justified. Reliability and relief pull in different directions here.

What you should do now

The practical consequence is not to keep AI out of critical processes. It is to change the acceptance criteria. Compare accuracy alone and you systematically select the model with the best guessing strategy.

Six steps before approval

1. Put calibration into the selection criteria

   Ask not only about accuracy but about the error rate at high model confidence. That is the number telling you how often your review process gets bypassed.

2. Define the handover rule before the agent goes live

   At what level of uncertainty does a case go to a human, and who is that human? Without a named owner the rule is a statement of intent.

3. Track the handover rate

   A system that never hands over is not a safe system, it is an untested one. Record the rate as its own metric from day one.

4. Build test cases from your real archive

   Public benchmarks contain deliberately hard cases and do not reflect your day-to-day work. Twenty real cases from your own records say more than any leaderboard.

5. Check the double requirement early for high-risk uses

   Does your vendor satisfy the EU AI Act and the sector rules by August 2026? Get it in writing rather than as a sales assurance.

6. Document the handover paths

   Conformity assessment will ask for them anyway. Write them down at audit time and you write them twice.

Further reading

Frequently asked questions