Agentic AI , AI Strategy , Data science & AI Jul 23, 2026

Pretraining Choices You Made Six Months Ago Are Constraining Your RL Gains Today

VECTOR Labs Team

Last updated on: Jul 23, 2026

Most teams evaluating RL post-training treat it as an independent optimisation stage: pick a base model, design a reward signal, run GRPO or PPO, and measure the delta. The implicit assumption is that more RL compute can compensate for a weaker starting point. Recent research suggests that assumption is structurally wrong, and the consequences for organisations planning post-training roadmaps are significant enough to warrant a clear-eyed look at what pretraining decisions actually determine.

The Pretraining Ceiling Is Quantifiable, Not Theoretical

The relationship between pretraining and RL returns is not a soft dependency. Work from Shen et al. (arXiv 2026) using chess and mathematical reasoning as controlled testbeds found that post-RL performance at a given RL compute budget is well-predicted by the pretraining loss. The slope of RL reward curves improved approximately linearly with the number of pretraining tokens. That is a scaling law relationship, not a qualitative observation.

The commercial implication is direct. If your base model was pretrained on a compressed token budget, you are not looking at a model that needs more RL compute to catch up. You are looking at a model that will asymptote earlier, at a lower ceiling, regardless of how much reward signal you throw at it.

This matters especially for teams who adopted a smaller, cheaper foundation model with the intention of closing the gap through fine-tuning. The gap does not close in the way the framing implies.

What RL Actually Does to a Pretrained Model

Understanding the ceiling requires understanding the mechanism. RL post-training does not rewrite a model's knowledge from scratch. On problems that fall within the model's existing competence, RL amplifies correct moves the supervised policy already preferred. On harder problems, it surfaces behaviours that were present but suppressed under the SFT distribution (Shen et al., arXiv 2026).

In both cases, RL is working with what pretraining deposited. The harder the problem class, the more dependent the RL signal is on latent capability that either exists in the pretrained weights or does not.

This is why RL post-training on a weak base model tends to produce confident but shallow reasoning improvements. The model learns to favour outputs that score well on the reward function, without the underlying representational capacity to generalise those improvements to novel problem structure.

Spectral Inheritance and the Weight-Space Constraint

A separate line of work makes the pretraining dependency concrete at the level of weight geometry. Zhu et al. (arXiv 2026) identified what they term spectral inheritance: RLVR reuses the singular value spectra of the base model's weight matrices, acquiring new behaviour through changes in the associated singular frames rather than through changes to the spectrum itself.

This has a precise implication. The directions in weight space that RL can efficiently explore are shaped by the singular structure inherited from pretraining. A model pretrained on low-quality or narrow data will have a weight spectrum that reflects those constraints. RL operating on top of that spectrum is not free to move arbitrarily through parameter space.

Zhu et al. also showed that keeping either the input or output singular subspace fixed during RL leaves a substantial portion of the checkpoint change unexplained. Both frames need to remain adaptable for RL to fully express the reward signal. This finding supports building optimisation stacks that are designed around the structure of reward-driven adaptation from the outset, rather than inheriting pretraining optimisation wholesale.

What This Means for Foundation Model Selection

The practical consequence for teams evaluating foundation models is that pretraining data quality and compute are not just factors affecting out-of-the-box benchmark performance. They determine the gradient of improvement available from every subsequent post-training stage.

Model size interacts with this in a non-trivial way. Larger models pretrained on more tokens carry richer singular structure and more latent capability for RL to surface. Smaller models may reach acceptable SFT performance on narrow tasks, but their RL improvement curves flatten earlier and more sharply.

For organisations selecting a base model with a post-training roadmap already in mind, the right question is not which model performs best on current benchmarks. The right question is which model's pretraining compute and data quality give the post-training pipeline the most headroom to work with.

Planning Post-Training Roadmaps With the Constraint in View

The honest framing for technical leaders is this: your post-training budget is not buying capability from scratch. It is buying access to capability that pretraining either deposited or did not.

If your current model stack is showing diminishing returns from RL, the diagnostic question is not whether the reward function is well-specified or whether the RL algorithm is appropriate. The first question is whether the pretraining loss is low enough to support further improvement at the RL compute level you are operating at.

Organisations that have the option to influence base model selection should treat pretraining compute and data curation as upstream infrastructure decisions with downstream post-training consequences. Those already committed to a specific foundation model need an accurate picture of where that model's RL ceiling sits before allocating further post-training budget against it.

FAQs

How do we know whether our base model's pretraining is sufficient for the RL improvements we are targeting?

The most direct signal is the pretraining loss on your target domain. Work by Shen et al. (arXiv 2026) shows that post-RL performance at a given compute budget is well-predicted by pretraining loss, so models with lower pretraining loss on relevant data distributions will consistently reach higher post-RL ceilings. If you do not have access to the pretraining loss for your chosen foundation model, running short RL compute sweeps and observing how quickly the reward curve flattens gives you a practical proxy for the headroom available.

Does using a larger model always give better RL returns, or does data quality matter more?

Both matter, and they interact. Scale without data quality produces a model with broad but shallow representational coverage, which limits RL's ability to surface correct behaviour on hard problems. Data quality without sufficient scale limits the singular structure of the weight matrices that RL operates on. The evidence from Shen et al. (arXiv 2026) points to pretraining tokens as a strong predictor of RL reward curve slope, which implicates both the volume and the quality of the training distribution. In practice, a smaller model pretrained on high-quality, domain-relevant data will often outperform a larger model pretrained on noisier data when both are subject to the same RL post-training regime.

What is spectral inheritance and why does it matter for our RL fine-tuning pipeline?

Spectral inheritance, as identified by Zhu et al. (arXiv 2026), refers to the finding that RLVR reuses the singular value spectra of the base model's weight matrices during post-training. New behaviour is acquired through changes to the singular frames rather than to the spectrum itself. This matters because it means the directions in weight space that RL can efficiently explore are constrained by the structure deposited during pretraining. An optimisation stack that accounts for this structure, fixing the spectrum and optimising the frames, reaches equivalent accuracy with substantially fewer training steps than a standard approach.

We are already committed to a specific foundation model. Is there anything we can do to extend its RL ceiling?

Within a fixed base model, the most productive interventions are on the data side of the RL pipeline rather than the compute side. Ensuring that the reward signal is well-calibrated to the difficulty distribution of your target tasks matters more than increasing RL compute once you are approaching the pretraining ceiling. Domain-specific continued pretraining, where the model is trained on additional in-domain text before RL begins, can also shift the effective pretraining loss on your target distribution and provide meaningful headroom. This is not a complete substitute for adequate original pretraining, but it is the most tractable lever available when base model selection is fixed.

How should we factor this into foundation model procurement or selection decisions going forward?

The key shift is to evaluate foundation models not only on their current benchmark performance but on the pretraining conditions that produced them. Relevant questions include the total pretraining compute, the token count and data curation approach, and whether the pretraining distribution overlaps meaningfully with your target task domain. Models where this information is not disclosed present a real evaluation risk: strong out-of-the-box performance can mask a shallow pretraining regime that will constrain post-training returns earlier than expected. Where possible, run short RL compute sweeps on candidate models before committing to a post-training roadmap.

Does this analysis apply equally to RLHF and RLVR, or is it specific to verifiable reward settings?

The pretraining ceiling argument applies to both, but the evidence base is currently stronger for RLVR settings where reward signals are verifiable and the training dynamics are easier to measure cleanly. RLHF introduces additional variance from human preference modelling, which can obscure the pretraining dependency without removing it. The spectral inheritance findings from Zhu et al. (arXiv 2026) were developed in RLVR contexts, but the underlying weight geometry argument is not specific to the reward mechanism. The practical implication is the same in either case: the representational capacity deposited during pretraining sets the boundary within which any reward-driven optimisation operates.

A team that understands you

With 20+ years of experience in the world's leading consultancy companies, implementing AI and ML projects in industry-specific contexts, we are ready to hear your challenges.

Talk with an AI expert