Foundation Models vs Task-Specific Training in Medical Imaging: How to Make the Architectural Call That Actually Holds Up in Production

The architectural choice between adapting a foundation model and training a task-specific network from scratch is one of the highest-leverage decisions an ML team makes in medical imaging. Get it wrong and you are either over-engineering a problem that a well-configured U-Net solves adequately, or you are shipping a model that degrades the moment it encounters a scanner it has never seen. Most published comparisons do not give you the evidence to make this call well. They test on in-distribution data, use a single baseline whose configuration is never justified, and report a headline metric that tells you nothing about what happens at a new clinical site. This article sets out a framework for making the decision rigorously.

Companion piece to our broader work on transfer learning strategy in medical imaging. See Transfer Learning in Medical Imaging for practical guidance on dataset-informed pretrained model selection for clinical cohorts.

Why In-Distribution Benchmarks Mislead Production Planning

A model that scores well on held-out data from the same institution it trained on has demonstrated memorisation of a data-generating process, not generalisation across one. The distinction matters because clinical deployment is almost always cross-institutional. Scanners differ, acquisition protocols differ, and patient populations differ in ways that shift the input distribution enough to degrade a model that was never exposed to that variance.

Kutlusan and Li (arXiv 2026) make this explicit in their experimental design: every model is evaluated both on held-out in-domain data and on independent external datasets from other institutions. That second condition is the one that predicts production behaviour. Teams that only report the first condition are not lying, but they are answering a different question than the one that matters for deployment.

The practical implication is that your evaluation protocol needs to be part of your architecture decision. If you cannot assemble external validation data before selecting an architecture, you are making the call without the most important evidence.

What the Cross-Institution Evidence Actually Shows

When foundation models and task-specific architectures are tested under genuine distribution shift, the performance ordering changes. Kutlusan and Li (arXiv 2026) found that across colonoscopy, breast ultrasound, and dermoscopy, a DINOv2-based foundation model adapted with various fine-tuning strategies generalised better on every external dataset. Task-specific models, specifically U-Nets and ResU-Nets trained from scratch, remained competitive within the training distribution when data were abundant, and produced sharper boundary delineation. The foundation model more frequently missed small lesions.

This is not a blanket verdict in favour of foundation models. It is a conditional result. The task-specific architecture wins on in-domain precision; the foundation model wins on cross-site stability. Your production context determines which of those properties is the binding constraint.

If your model will be deployed at a single institution with consistent imaging protocols and you have sufficient labelled data, a well-configured task-specific network is a defensible and often preferable choice. If the model must transfer across sites, scanners, or patient populations, the generalisation advantage of foundation model adaptation is not marginal.

Fine-Tuning Depth: How Much of the Foundation Model Do You Actually Need to Adapt

One of the more practically useful findings in the literature concerns how much of a foundation model needs to change during adaptation. The intuition that more fine-tuning always produces better results does not hold. Kutlusan and Li (arXiv 2026) show that only a small fraction of the model required adaptation to reach competitive performance, and that the capacity needed was determined by the imaging domain rather than by the adaptation method chosen.

Bias-Only and Parameter-Efficient Strategies

Strategies at the lighter end of the spectrum, such as tuning only bias terms or using adapter modules, can reach performance close to full fine-tuning in domains where the visual structure is not radically different from the pretraining distribution. This matters for compute budgets and for regulatory traceability, since a model with fewer changed parameters is easier to document and audit.

Full Fine-Tuning

Full fine-tuning is warranted when the target domain is genuinely distant from the pretraining distribution, for example when moving from natural images to modalities like ultrasound or dermoscopy where texture statistics differ substantially. The cost is higher compute and a larger surface area of model behaviour to validate. Teams should treat full fine-tuning as a last resort justified by domain distance, not a default.

The operational implication is that fine-tuning depth should be selected on the basis of the target imaging domain, not on the basis of which method is easiest to implement or most familiar to the team.

The Baseline Configuration Problem

The second major flaw in most published comparisons is that the task-specific baseline is poorly configured. A U-Net trained with default hyperparameters, no data augmentation strategy tuned to the modality, and no preprocessing pipeline matched to the scanner is not a fair opponent for a foundation model that has been carefully adapted. It is a strawman.

This problem is not academic. When teams run internal evaluations to justify an architectural choice, the same bias enters. The foundation model gets careful attention; the task-specific baseline gets the defaults. The resulting comparison then drives a budget and engineering commitment that may not survive contact with a properly configured alternative.

The corrective is to treat the baseline as a first-class engineering effort. A well-configured nnU-Net, for instance, automates many of the preprocessing and hyperparameter decisions that otherwise introduce noise into the comparison. If your task-specific baseline has not had equivalent engineering investment to your foundation model adaptation, your evaluation is not informative.

Making the Architectural Decision: A Practical Framework

The decision reduces to three questions that should be answered in order before any architecture is selected.

First, what is the deployment scope? A model that must operate across multiple institutions, scanner vendors, or acquisition protocols has a generalisation requirement that strongly favours foundation model adaptation based on current evidence.

Second, how much labelled data is available at the target site? Task-specific models remain competitive in-domain when training data are abundant. Foundation model adaptation provides the larger benefit when labelled data are scarce, because the pretrained representations carry useful structure that reduces the data requirement for fine-tuning.

Third, what are the precision requirements for the target pathology? If the clinical task requires precise delineation of small lesions, the tendency of adapted foundation models to miss small structures is a material risk. That characteristic, documented by Kutlusan and Li (arXiv 2026), may tip the decision toward a task-specific architecture even where generalisation is also a requirement, or it may motivate a hybrid approach where the foundation model is used for coarse localisation and a task-specific head handles fine boundary estimation.

Answering these three questions with evidence, rather than defaulting to whichever architecture the team has most recently worked with, is the minimum standard for a production-grade architectural decision.