Building Clinical AI That Survives Contact With Real Patient Populations: Lessons From Respiratory and Psychiatric Research Pipelines

Clinical AI systems fail in a specific and recurring pattern: they perform well in controlled validation, then degrade when deployed against the full diversity of real patient populations. Respiratory disease management and psychiatric neuroimaging are two domains where this pattern has become well-documented, not because they are technically unusual, but because the populations they serve are structurally heterogeneous in ways that standard validation pipelines are not designed to surface. The failure modes that emerge from these two domains are not domain-specific. They are architectural, and they will appear in any clinical AI system built without explicit subgroup accountability baked into the validation strategy from the start.

Companion piece to our broader work on clinical AI validation. See AI Diagnostic Models in Healthcare: Hidden Validity Failures for a detailed treatment of label leakage, anatomical irrelevance, and the evaluation rigour engineering leaders must enforce before deployment.

Why Population Heterogeneity Is an Architectural Problem, Not a Data Problem

The instinct when subgroup performance degrades is to call it a data quality issue and request more samples. This misdiagnoses the problem. The underlying issue is that aggregate metrics hide differential performance across subgroups, and most model selection and hyperparameter tuning processes optimise for the aggregate. A model can achieve strong overall AUC while systematically underperforming on women, older patients, or patients with comorbidities that shift the feature distribution.

In respiratory AI, this manifests through spirometry and imaging features that carry different predictive weight depending on body habitus, smoking history, and sex-linked lung physiology. COPD progression models trained predominantly on male cohorts will encode male physiological baselines as implicit priors. When deployed on mixed-sex populations, the model's confidence intervals do not widen to reflect this prior mismatch. They stay narrow, and the errors are invisible until a clinician flags them.

Psychiatric neuroimaging pipelines face the same structural issue through a different mechanism. Functional and structural MRI features used to classify conditions like major depressive disorder or schizophrenia spectrum disorders vary significantly by biological sex, age, and scanner hardware. A model trained on data from two sites using 3T scanners will not generalise cleanly to a third site running 1.5T, even if the diagnostic labels are consistent. These are not edge cases. They are the standard conditions of multi-site clinical deployment.

Biological Sex as a Variable: What Most Pipelines Get Wrong

Feature Distribution Shifts by Sex

The most common error is treating biological sex as a demographic covariate to be controlled for rather than as a variable that modifies the feature space itself. In respiratory AI, lung volume, airway geometry, and diffusing capacity all differ systematically between sexes in ways that affect how imaging and functional features map to disease severity. Controlling for sex in a regression sense does not resolve this. It requires stratified model evaluation and, in some cases, sex-stratified model training.

Neuroimaging and Sex-Linked Signal Variation

In psychiatric neuroimaging, cortical thickness, white matter microstructure, and resting-state connectivity patterns all show sex-linked variation that is large enough to affect classification boundaries. A model trained without sex-stratified validation will often appear to perform well on aggregate metrics while encoding a systematic bias toward whichever sex is more represented in the training cohort. The commercial risk here is not just clinical harm. It is regulatory exposure, because regulators are increasingly requiring subgroup performance breakdowns as part of SaMD submissions.

What a Defensible Subgroup Testing Protocol Actually Looks Like

A subgroup testing protocol that can survive regulatory scrutiny and clinical audit needs to be specified before training begins, not retrofitted after aggregate performance looks acceptable. The subgroups to be tested should be defined based on known sources of biological and operational variation: biological sex, age band, disease severity stratum, comorbidity profile, and, in multi-site deployments, acquisition site.

Performance metrics should be reported per subgroup, not just overall. The relevant metrics are not identical across subgroups. Sensitivity and specificity thresholds may need to be calibrated differently for a subgroup where the base rate of the condition differs from the training distribution. If the model cannot meet minimum performance thresholds across all pre-specified subgroups, that is a deployment gate, not a footnote.

The protocol should also specify what happens when a new deployment site introduces a patient population that was not represented in the original training cohort. This is not a hypothetical. It is the standard trajectory of any clinical AI system that achieves adoption. A prospective monitoring plan, with defined drift detection thresholds and a re-evaluation trigger, is a required component of any governance framework that takes generalisability seriously.

Validation Strategy for Multi-Site and Multi-Population Deployments

Site-Specific Calibration

Aggregate validation on a held-out test set does not replicate the conditions of multi-site deployment. Each site introduces variation in patient demographics, clinical workflows, and data acquisition parameters. The validation strategy should include at least one site that was withheld entirely from training and internal validation. Performance on this held-out site is the closest proxy available for true generalisability before prospective deployment.

Federated Evaluation as a Governance Tool

Where data sharing between sites is constrained by privacy regulation, federated evaluation offers a practical alternative. Rather than centralising patient data for validation, model inference can be run locally at each site and performance metrics aggregated centrally. This approach has been used in oncology research to validate models across multi-national hospital networks without moving patient data across jurisdictions. The same architecture applies to respiratory and psychiatric AI validation, and it has the added benefit of producing site-specific performance reports that are directly usable in regulatory submissions.

Governance Requirements That Technical Leaders Cannot Delegate

The governance requirements for clinical AI operating across heterogeneous populations are not optional additions to a deployment checklist. They are the conditions under which a system can be considered fit for clinical use. Technical leaders need to own three things specifically.

First, the subgroup analysis plan must be a formal artefact, version-controlled and reviewed before training begins. Second, the model card or equivalent documentation must include per-subgroup performance metrics, not just aggregate figures. Third, there must be a defined owner for post-deployment monitoring, with authority to trigger re-evaluation or withdrawal if drift thresholds are breached.

The temptation at scale is to treat governance as a compliance exercise that runs in parallel to engineering. The systems that fail in deployment are almost always the ones where governance was treated as documentation rather than as a constraint on model selection and deployment decisions. Clinical AI that survives contact with real patient populations is built by teams where the governance requirements shape the architecture, not the other way around.