Why AI Diagnostic Models in Healthcare Keep Fooling Themselves: What Engineering Leaders Must Demand Before Deployment

Clinical AI benchmarks are designed to measure how well a model has learned to diagnose. In practice, they frequently measure how well a model has learned to pass a test. The distinction matters enormously in healthcare, where a model that achieves 90% accuracy on a held-out validation set by attending to scan acquisition artefacts rather than pathological anatomy will perform unpredictably the moment it encounters a scanner from a different vendor or a protocol from a different site. Engineering leaders evaluating or commissioning diagnostic AI need to understand the specific mechanisms behind this failure before procurement, not after a regulatory incident forces the question.

Companion piece to our broader work on diagnostic imaging AI. See Building AI for Diagnostic Imaging: What Works, What Breaks, and What Regulators Will Ask for a detailed treatment of why the gap between benchmark and deployment performance is consistently larger for imaging AI than for most other clinical AI categories.

The Anatomical Irrelevance Problem

A classifier trained on MRI or CT scans does not inherently learn to attend to anatomy. It learns to attend to whatever features in the training data correlate with the label. In many clinical datasets, those features include skull shape, background noise patterns, scanner field-of-view boundaries, and patient positioning artefacts. These are not pathological signals, but they are consistent signals, and a neural network will exploit them if they reduce training loss.

The consequences are predictable. A model that has learned to associate a particular background intensity distribution with a positive diagnosis will generalise poorly to any imaging environment where that distribution changes. Because scanner hardware, acquisition protocols, and reconstruction pipelines vary across institutions, this failure mode is nearly guaranteed to appear at external validation.

Recent research on Alzheimer's disease classification from structural MRI makes this concrete. Using Grad-CAM visualisations, Nicolae and Mocanu found that image-only classifiers trained on ADNI data frequently attend to the skull, orbits, and background rather than the medial temporal lobe structures that are clinically implicated in disease progression. Restricting the input to an anatomically constrained crop of the medial temporal lobe improved three-way classification accuracy from 58.7% to 65.1% on the same architecture, without any other modification (Nicolae et al., arXiv 2026). That improvement is not a product of a better model. It is a product of removing the irrelevant signal the model had been using.

Label Leakage in Multimodal Pipelines

Multimodal architectures that fuse imaging with clinical tabular data are increasingly common in diagnostic AI. They are also a reliable source of inflated benchmark performance. The problem arises when the tabular features included in training encode information that was directly used to assign the diagnostic label in the first place. A model that can read a cognitive assessment score in its input features and correlate it with an Alzheimer's diagnosis is not performing imaging-based classification. It is performing a lookup.

This is not a theoretical concern. Nicolae and Mocanu systematically organised clinical variables along what they term a label-leakage spectrum, ranging from features with minimal diagnostic entanglement to cognitive scores that are definitionally entangled with the label. Fusion with cognitive scores produced 87.3% three-way accuracy, which the authors explicitly describe as a leakage-driven upper bound rather than a genuine imaging result. Fusion with regional brain volumes, which are anatomically grounded but not directly used in label assignment, produced 73.0% (Nicolae et al., arXiv 2026). The 14-point gap between those two numbers is not model capability. It is information contamination.

The downstream risk for engineering teams is that a vendor can present multimodal accuracy figures that are technically correct but commercially misleading. Without knowing which tabular features were included and how those features relate to the labelling process, the headline number is uninterpretable.

What Contrastive Training Targets Actually Teach the Encoder

A subtler failure mode appears in contrastive learning architectures, where an image encoder is trained to align its representations with a paired modality. The choice of contrastive target shapes what the image encoder learns, and that effect persists at inference even when the paired modality is removed.

Nicolae and Mocanu demonstrate this directly. An image-only classification head achieved 52.4% accuracy on MCI versus CN classification when the encoder had been aligned to cognitive scores during contrastive training, and 73.8% when the encoder had been aligned to regional brain volumes. No tabular input was used at inference in either case (Nicolae et al., arXiv 2026). The difference is entirely attributable to what the encoder learned to represent during training.

For engineering teams, this means that evaluating a multimodal model by stripping out the tabular input at test time is not a reliable way to assess the quality of the image representations. The contamination from a leaky contrastive target is baked into the encoder weights before inference begins.

The Evaluation Rigour Engineering Leaders Must Enforce

Given these failure modes, the evaluation requirements for clinical AI procurement need to go beyond held-out accuracy on the training distribution. Several specific demands are warranted before any diagnostic model is considered for production deployment.

Explainability as a Validity Check

Saliency maps and attention visualisations should be treated as validity checks, not as post-hoc documentation. If a model's attention is consistently landing on anatomically implausible regions, that is a signal that the accuracy figure was achieved through shortcut learning. Requiring vendors to provide Grad-CAM or equivalent outputs stratified by diagnostic class is a reasonable minimum standard.

External Validation on Prospective Data

Internal validation on a held-out split of the training dataset does not test generalisation across scanner hardware, acquisition protocols, or patient demographics. External validation on prospectively collected data from a different institution is the appropriate standard for any model intended for clinical use. Models that have only been validated on retrospective splits from the same dataset they were trained on should be treated with significant scepticism.

Tabular Feature Auditing in Multimodal Systems

Any multimodal system that includes clinical tabular features requires a documented audit of how those features relate to the labelling process. Features that were used to assign the diagnostic label, or that are definitionally correlated with it, must be excluded from training or clearly accounted for in the accuracy reporting. A vendor that cannot produce this audit has not performed it.

The Procurement Question That Changes the Conversation

Most procurement processes for clinical AI focus on accuracy, regulatory clearance status, and integration compatibility. These are necessary conditions but not sufficient ones. A model can satisfy all three and still be learning the wrong thing.

The question that surfaces the underlying validity is straightforward: what evidence do you have that your model's performance is attributable to clinically relevant features rather than dataset-specific artefacts? A vendor with a well-validated model will have a specific answer involving anatomical grounding, external validation cohorts, and feature attribution analysis. A vendor without one will not.

Asking that question before signing a contract is considerably less expensive than asking it after a regulatory body or a clinical safety review asks it first.