Procurement Checklist for Clinical AI: What to Validate Before a Healthcare Model Goes Near a Patient

Clinical AI procurement decisions are too often made on the strength of a headline AUC score from a paper the vendor selected themselves. The number looks credible, the clinical use case is compelling, and the regulatory documentation arrives as a well-formatted PDF. What rarely arrives is a clear account of where the model was trained, who was in that dataset, and what happens to performance when the model encounters a patient population that looks nothing like the one it learned from. That gap between benchmark and bedside is where clinical AI deployments fail, and it is the gap this checklist is designed to close.

Companion piece to our broader work on clinical AI evaluation rigour. See AI Diagnostic Models in Healthcare: Hidden Validity Failures for a detailed breakdown of label leakage, anatomical irrelevance, and the evaluation design failures that inflate reported accuracy.

Dataset Representativeness: The First Question Is Always "Who Was in the Training Data?"

A model's training dataset is its worldview. If that dataset was drawn from a single academic medical centre, a narrow age band, or a population with a specific ethnic composition, the model has learned patterns that may not generalise beyond those boundaries. The vendor's job is to tell you this upfront. If they cannot, that is itself a finding.

Ask for a structured breakdown of the training data by site of acquisition, scanner or device type, patient demographics, and clinical subgroup. Lung cancer screening AI, for example, has historically been evaluated on datasets with inconsistent clinical metadata and fragmented nodule annotations, which makes it difficult to assess whether reported performance holds across the range of presentations a real screening programme would encounter (Topalis et al., arXiv 2026). The same structural problem applies across radiology, pathology, and any domain where image acquisition conditions vary by institution.

Pay particular attention to whether the training data includes the specific demographic and comorbidity profiles present in your patient population. A model trained predominantly on non-smoking populations will have learned a different prior distribution for nodule malignancy risk than one trained on a high-prevalence screening cohort.

Out-of-Distribution Performance: What the Benchmark Does Not Show You

Benchmark performance is measured on a held-out test set drawn from the same distribution as the training data. That is not the distribution your model will encounter in production. The question to ask is not "what is your AUC on your test set?" but "what is your performance on data from institutions that contributed nothing to training or validation?"

Request external validation results specifically. If the vendor has only evaluated the model on internal splits of their own dataset, you have no evidence of generalisation. If they have external validation results, examine the sites involved: a model validated at three institutions in the same city, using the same scanner vendor, is not externally validated in any meaningful sense.

Also probe for subgroup performance disaggregation. Aggregate accuracy can mask substantial performance disparities across age groups, sexes, or disease severity strata. A model that performs well on average but poorly on the subgroup most likely to benefit from early detection is not a model that serves your clinical purpose.

Regulatory Documentation: What the Filing Actually Covers

Regulatory clearance is not a quality certificate. It is a statement that the device meets the evidentiary standards required for the specific intended use described in the submission. Understanding the boundaries of that intended use is essential before assuming the clearance covers your deployment context.

Intended Use Scope

Read the intended use statement carefully. A model cleared for use as a decision support tool for radiologists reviewing low-dose CT scans in a specific age range may not be cleared for autonomous flagging, different imaging protocols, or use outside that age range. Deploying outside the cleared intended use creates both patient safety and liability exposure.

Algorithm Version Control

Confirm which version of the algorithm received clearance, and whether the vendor has a documented process for managing performance drift and revalidation after updates. A model that was cleared at version 1.2 and has since been retrained without regulatory notification is operating in a grey area that your legal and clinical governance teams need to understand before sign-off.

Evaluation Design Red Flags: Questions That Separate Rigorous Validation from Marketing

The structure of a validation study tells you almost as much as its results. Several design choices reliably indicate that a vendor has optimised for a persuasive number rather than a clinically meaningful one.

Reader Study Design

Ask whether the clinical performance study compared the model against radiologists reading with and without AI assistance, or simply compared AI output against a retrospective ground truth label. The latter tells you nothing about how the model changes clinical decision-making in practice.

Ground Truth Quality

Understand how ground truth labels were constructed. Labels derived from a single reader, from radiology reports rather than pathology confirmation, or from retrospective record linkage without prospective follow-up all introduce noise that inflates apparent model accuracy. The LUCQUE dataset work highlights that inconsistent annotation standards across existing lung cancer screening datasets have historically made it difficult to draw reliable conclusions from published evaluations (Topalis et al., arXiv 2026). Your vendor's validation data may have the same problem.

Temporal Validation

Check whether the validation data was collected contemporaneously with training data or prospectively after model development was complete. A model validated on data from the same time window as its training set has not been tested under the conditions it will actually face in deployment.

Integration, Monitoring, and Ongoing Governance

Procurement is not the end of the technical risk exposure. A model that performs adequately at go-live can degrade as patient populations shift, imaging protocols change, or upstream data pipelines are modified. Your vendor agreement should specify what monitoring is contractually required, not just what is technically possible.

Define performance thresholds that trigger mandatory review, and establish who is responsible for detecting and acting on drift. Ask whether the vendor provides performance dashboards tied to your specific deployment, or whether monitoring is limited to aggregate telemetry across their entire customer base. The latter gives you no signal about your population.

Finally, ensure your clinical governance framework includes a clear escalation path for cases where the model output conflicts with clinician judgement. Clinical AI tools that lack a defined override and audit mechanism create accountability ambiguity that regulatory bodies and clinical risk committees will scrutinise closely.