Why CT Foundation Models Fail at the Organ Level and What Adaptive Architectures Do About It

CT foundation models have made genuine progress on population-level tasks: zero-shot disease classification, report generation, and cross-modal retrieval. But when clinical teams attempt to use these models for anatomy-specific decisions, a structural problem surfaces. The whole-volume representations that make these models efficient at scale are the same representations that make them imprecise at the organ level. This article explains why that tension exists, what architectural approaches resolve it, and what engineering leaders should be asking vendors before committing to a medical imaging AI platform.

Companion piece to our broader work on medical imaging AI in production. See Building AI for Diagnostic Imaging: What Works, What Breaks, and What Regulators Will Ask for a detailed treatment of why benchmark performance diverges from deployment performance across imaging modalities.

The Whole-Volume Representation Problem

Most CT foundation models compress a full 3D volume into a single embedding before any downstream task sees it. This design choice is not arbitrary. Training on large-scale datasets demands computational tractability, and whole-volume representations make contrastive pre-training feasible at scale.

The cost is signal dilution. When a model encodes a full thoracic CT into one vector, the embedding must simultaneously represent the lungs, mediastinum, liver, and spine. Fine-grained anatomical signals, the kind that distinguish a subtle pulmonary nodule from a vascular structure, get averaged into a representation optimised for global coherence rather than local precision.

The practical consequence is that these models perform well on coarse classification tasks but degrade on anatomy-specific queries. A health system evaluating a vendor model on whole-volume benchmarks may be measuring a capability that does not transfer to the organ-level tasks their radiologists actually need.

Fine-Grained Approaches and Why They Introduce a Different Problem

Fine-grained vision-language pre-training (FVLP) attempts to fix this by aligning anatomy-specific visual embeddings with anatomy-specific text. Instead of one embedding per volume, the model learns separate representations for each anatomical region, matched to the corresponding sentences in a radiology report.

This approach does recover local signal. But it does so by discarding the global context that whole-volume models provide. An FVLP model trained to focus on the liver has no architectural mechanism to recognise that the liver finding is clinically significant because of concurrent changes in the spleen. Cross-anatomy context is not a peripheral concern in radiology; it is often the diagnostic signal.

There is also a practical barrier. Existing fine-grained methods train from scratch on anatomy-segmented data, which is computationally expensive and requires annotated datasets that most health systems do not have. The choice between global context and anatomical precision has historically required accepting a real trade-off, not resolving it.

What the ACA Framework Does Differently

Anatomy-Level Decomposition Without Retraining

Anatomy Contextualized Adaptation (ACA) approaches this problem by keeping the foundation model frozen and inserting lightweight adaptation layers on top of it (Kenia et al., arXiv 2026). The foundation model's weights are not touched. Instead, ACA uses TotalSegmentator to decompose each CT volume into per-anatomy embeddings, which are then passed through a transformer that models cross-anatomy relationships.

This design separates two concerns that previous approaches conflated. The foundation model handles global feature extraction. The adaptation layer handles anatomy-level alignment. Neither component has to do both jobs, and neither has to be trained from scratch.

The alignment target is dual: per-anatomy text extracted from radiology reports, and scan-level text that preserves global context. The model is therefore incentivised to be precise at the organ level while remaining aware of the broader clinical picture.

Efficiency as an Engineering Requirement

ACA requires less than one hour of training once embeddings are cached (Kenia et al., arXiv 2026). That figure matters for enterprise deployment. A framework that demands weeks of compute to adapt to a new anatomy or a new institution is not practically deployable across a health system with heterogeneous scanner hardware and varying scan protocols.

The caching step is architecturally important. By pre-computing foundation model embeddings once and training only the adaptation layers, ACA makes iterative experimentation feasible. Teams can adjust the adaptation strategy without rerunning the expensive upstream computation.

What the Attention Weights Reveal About Clinical Reasoning

One of the less-discussed findings from ACA is that the inter-anatomy transformer's attention weights reflect plausible clinical relationships (Kenia et al., arXiv 2026). The model learns to route context between anatomical regions in ways that align with how radiologists reason about multi-system findings.

This matters beyond interpretability. In regulated medical AI, the ability to inspect model behaviour at the anatomical level is increasingly relevant to regulatory submissions and clinical governance. A model that can show which anatomical regions influenced a classification decision is more auditable than one that produces a scalar confidence score from an opaque global embedding.

It also provides a mechanism for clinical validation teams to catch failure modes before deployment. If attention weights indicate implausible cross-anatomy routing on a specific pathology subtype, that is a testable signal rather than a black-box anomaly.

What Engineering Leaders Should Ask Vendors

Representation Architecture

Ask whether the model produces whole-volume embeddings or anatomy-level representations, and whether those are distinct concerns in the architecture. A vendor who cannot answer this question cleanly has not thought carefully about where their model will degrade in clinical use.

Ask whether global context is preserved when anatomy-level representations are used. The ACA framework demonstrates that these goals are not mutually exclusive, but most vendor architectures have not resolved the tension.

Adaptation Cost and Institutional Fit

Ask what it costs to adapt the model to your institution's scanner protocols, patient population, and report vocabulary. A model trained on a different health system's data distribution will have systematic biases that only surface after deployment. We have written in detail about why this gap between validation and deployment performance is consistently larger for imaging AI than for other clinical AI categories.

Ask whether adaptation requires retraining from scratch or whether lightweight fine-tuning is architecturally supported. The answer determines whether the model can realistically be updated as clinical requirements change without a multi-month engineering cycle each time.

Anatomy Coverage and Segmentation Dependency

Ask which anatomical structures the model has been validated on and what happens on structures outside that set. ACA's dependency on TotalSegmentator means its coverage is bounded by TotalSegmentator's segmentation classes. That is a known and manageable constraint, but it needs to be understood before procurement.

Ask for zero-shot performance figures on anatomy-specific classification tasks, not just whole-volume benchmarks. The two sets of numbers will diverge, and the anatomy-specific figures are the ones that predict clinical utility.