AI Strategy , Data science & AI Aug 20, 2026

Why High-Quality Domain Data Is Your Real Scaling Bottleneck, Not Compute

VECTOR Labs Team

Last updated on: Aug 20, 2026

Most enterprise AI teams budget for GPUs first and think about data second. That instinct made sense when the primary challenge was getting foundation models to work at all. But as organisations move from general-purpose models to domain-specific ones, the constraint shifts. Compute is purchasable on demand. High-quality proprietary data is not. And new research confirms that how you manage that data during pretraining determines more about your model's ceiling than the size of your training cluster.

The Tokens-Per-Parameter Ratio and Why It Creates a Data Problem

When you scale a model, you cannot simply add parameters and call it done. The tokens-per-parameter ratio (TPP) must remain within an appropriate range to maintain training efficiency and final model quality. As model size grows, the total token budget must grow proportionally.

For general web data, this is manageable. The internet is large, and crawling more of it is an engineering problem with a known cost curve. For domain-specific data, the situation is fundamentally different. Clinical notes, legal filings, proprietary engineering documentation, and financial records are finite, expensive to curate, and often legally constrained in how they can be used.

The practical consequence is that as your model scales, the fraction of high-quality domain data in your training mixture tends to shrink relative to the total token budget. Your model sees proportionally less of the data that actually matters for your use case. This is the structural problem that repetition strategies are designed to address.

What the Research Says About Repetition

The intuition most teams carry is that repeating training data is a necessary evil at best and a source of overfitting at worst. Recent research complicates that picture in useful ways.

Li et al. (arXiv 2026) studied data repetition under practical LLM scaling conditions, where the token budget grows proportionally with model size. Their first finding is counterintuitive: at a fixed TPP, the optimal repetition count increases mildly as model size grows. Larger models are more tolerant of repeated data than smaller ones, not less. The mechanism is that larger models have greater capacity to extract signal from each pass over the data without memorising surface-level patterns.

Their second finding is more immediately actionable. Across domains, optimal repetition count is strongly negatively correlated with the final validation loss of that domain. Domains where the model already achieves low loss can sustain more repetitions without overfitting. Domains with high loss benefit less from repetition and more from acquiring additional unique data.

What This Means by Domain Type

High-quality, well-structured domains such as curated code repositories or formal mathematical text tend to have lower validation loss and can tolerate higher repetition counts. The model has more to learn from each re-exposure because the signal density is high and the distribution is consistent.

Noisier or more heterogeneous domains, such as informal customer support transcripts or lightly edited field reports, reach diminishing returns faster. Repeating low-quality domain data amplifies its noise rather than reinforcing its signal. The practical implication is that your data curation effort should be front-loaded for these domains before you consider any repetition strategy.

The Proxy Model Shortcut

One of the more practically useful findings from Li et al. (arXiv 2026) is that repetition counts tuned on smaller proxy models with the same TPP transfer reasonably well to larger models. This matters because it means you do not need to run expensive large-scale ablations to calibrate your repetition strategy.

The workflow this enables is to run systematic repetition experiments at a smaller model scale, identify the optimal repetition count per domain at a fixed TPP, and then apply those counts when training at full scale. The transfer is not perfect, but it is close enough to be a practical tool for teams with constrained experimentation budgets.

This also changes how you should think about your small-scale training infrastructure. Rather than treating smaller models purely as proof-of-concept vehicles, they become calibration instruments for data strategy decisions that will apply at scale.

Allocating Curation Resources Against the Repetition Curve

The finding that unique data volume is only weakly related to optimal repetition count is worth sitting with. Teams often assume that acquiring more raw domain data is always the right investment. The research suggests the relationship is more nuanced.

If your domain already achieves low validation loss, you are likely in a regime where strategic repetition of existing curated data is more efficient than acquiring marginal new data of similar quality. If your domain loss is high, you need better data rather than more passes over what you already have. The decision about where to invest in data pipelines should be driven by domain loss metrics, not by raw data volume targets.

This reframes the data curation budget conversation. The question is not "how many tokens can we collect?" but "what is the validation loss of our current domain corpus, and does that put us in a repetition-tolerant or data-hungry regime?"

Practical Guidance for Enterprise ML Teams

The operational steps that follow from this research are specific enough to act on.

First, instrument your domain-specific validation loss early in your training runs and track it across domains separately. This is the leading indicator that determines your repetition strategy, and most teams do not monitor it at sufficient granularity during pretraining.

Second, design your data pipeline to support variable repetition counts by domain rather than applying a single global repetition multiplier. A uniform repetition strategy will over-repeat high-loss domains and under-repeat low-loss ones simultaneously.

Third, run your repetition calibration experiments at smaller scale before committing to a full training run. The proxy model approach is not a workaround; it is a legitimate experimental methodology that reduces the cost of getting your data mixture right.

The broader point is that compute efficiency and data efficiency are not the same optimisation target. Teams that treat them as interchangeable will consistently underperform teams that manage them separately.

Where Vector Labs Fits

We build domain-specific AI models for regulated and data-constrained environments, where getting the training data strategy right is as important as the architecture. Our work developing a cardiovascular AI model for wearable ECG signals, detailed in our AI model development and certification for cardiovascular medicine case study, required training on a combination of public and proprietary datasets with strict data quality controls, ultimately achieving clinical-grade accuracy and Class 2A medical device certification. If your team is working through a similar data strategy challenge, speak to us directly.

FAQs

How do we know when our domain corpus is large enough to support pretraining rather than fine-tuning?

There is no universal token threshold, but the practical signal is whether your domain data is large enough to maintain a reasonable TPP at your target model size without excessive repetition. If you are already repeating data more than five to ten times just to meet your token budget, you are likely better served by continued domain data acquisition or by shifting to a fine-tuning approach on top of a strong general foundation model. The right answer depends on how much domain-specific capability you need versus what the foundation model already provides.

What repetition counts are typically safe before overfitting becomes a real risk?

The research from Li et al. (arXiv 2026) does not prescribe a single safe number because the optimal count varies by domain loss and model size. What it does establish is that the optimal count is higher than most practitioners assume and increases with model size. The practical approach is to treat this as an empirical question specific to your domain, run calibration experiments at smaller scale, and use validation loss on a held-out domain set as your overfitting signal rather than assuming a fixed repetition cap applies universally.

Should we prioritise data acquisition or data curation for our existing corpus?

This depends on your current domain validation loss. If your domain loss is high, the data you have is not teaching the model effectively, and adding more of the same quality will not solve that. Curation investment, meaning filtering, deduplication, and quality scoring, addresses the signal-to-noise problem that repetition cannot fix. If your domain loss is already low, you are likely in a regime where strategic repetition of well-curated existing data is more efficient than marginal new acquisition. Measure first, then decide.

Can we use the proxy model approach if our target model is significantly larger than what we can train experimentally?

Yes, and this is precisely the scenario the proxy model methodology is designed for. The key requirement is that the proxy model and the target model share the same TPP, not the same absolute token count. As long as that ratio is held constant, the repetition counts identified at smaller scale transfer reasonably well to larger models, according to Li et al. (arXiv 2026). The transfer is an approximation rather than a guarantee, so build in some validation budget at full scale to confirm the calibration holds for your specific domain.

How should we handle domains with very different data volumes within the same training mixture?

Apply domain-specific repetition counts rather than a single global multiplier. Domains with lower validation loss and smaller corpora can be repeated more aggressively to maintain their presence in the mixture. Domains with higher loss and larger corpora should be sampled at lower repetition rates to avoid overfitting on noisy data. This requires your data pipeline to support per-domain repetition configuration, which is worth building early because the alternative, a uniform repetition strategy, will systematically misallocate your effective token budget.

Does this research apply to continued pretraining and domain adaptation, or only to training from scratch?

The findings from Li et al. (arXiv 2026) are framed in the context of pretraining, but the underlying logic applies to continued pretraining and domain adaptation as well. When you are adapting a foundation model to a new domain by continuing training on domain-specific data, you face the same repetition trade-off: too few passes and the domain signal does not register; too many and you risk degrading general capabilities through overfitting. The proxy model calibration approach is particularly useful here because the stakes of getting the repetition count wrong are higher when you are working from a pretrained checkpoint rather than random initialisation.

A team that understands you

With 20+ years of experience in the world's leading consultancy companies, implementing AI and ML projects in industry-specific contexts, we are ready to hear your challenges.

Talk with an AI expert