When your tabular pipeline mixes structured fields with text embeddings from product descriptions or image embeddings from document scans, the dimensionality reduction step sitting between your encoder and your learner is making decisions about what information matters. Principal component analysis makes those decisions without consulting your target variable. That is not a minor statistical nuance - it is a systematic mechanism by which predictive signal gets discarded before your model ever sees it.
Companion piece to our broader work on multimodal data pipelines for prediction tasks. See AI screening tool for recruitment software for how we approached combining structured candidate data with unstructured text fields at production scale.
What PCA Actually Optimises For
PCA finds the directions of maximum variance in your embedding space. It is a mathematically principled answer to a specific question: how do we compress a high-dimensional representation while retaining the most statistical spread?
The problem is that variance in an embedding space and variance relevant to your prediction target are not the same thing. A text encoder trained on a general corpus will produce embeddings where the dominant axes of variation reflect linguistic diversity across the training distribution, not the signal that separates your outcome classes or drives your regression target.
When you reduce a 768-dimensional BERT embedding to 50 PCA components and feed it into XGBoost, you are handing the model the 50 directions that explain the most variance in the text. Whether those directions contain the signal that predicts churn, fraud, or product return rate is largely a matter of luck.
How Partial Least Squares Fills the Gap
Partial least squares takes a different approach. Rather than maximising variance in the input, it finds projections that maximise covariance between the input representation and the target variable. The components it extracts are, by construction, oriented toward what you are trying to predict.
This is not a new idea in statistics, but its application to frozen encoder embeddings in multimodal tabular pipelines is recent and practically significant. LightTAR, introduced by Shapira et al. (Shapira et al., HuggingFace 2026), combines a standard PCA reduction of the final encoder layer with a small set of PLS-derived features extracted from the concatenation of the final and an intermediate encoder layer. The intermediate layer is included because task-relevant signal is not always most concentrated at the final representation.
The result is a hybrid representation: PCA handles the bulk of the compression, preserving general encoder structure, while a small number of PLS components carry supervised signal that PCA would have ranked low and discarded. Across 21 tabular models tested on the MulTaBench benchmark suite, this augmented representation improved average predictive performance for every model class evaluated (Shapira et al., HuggingFace 2026).
What This Means for Tree Ensembles and Tabular Foundation Models
Tree ensembles and tabular foundation models handle features differently, but both are affected by the same upstream problem. If the features they receive have already had target-relevant variance compressed out, no amount of downstream model capacity recovers it.
Tree Ensembles
For gradient boosted trees, the issue is particularly direct. Trees split on individual features, so they are sensitive to whether the dimensions they receive carry discriminative signal. A PCA component that captures general semantic variation in product descriptions may not split cleanly on any outcome-relevant boundary. A PLS component extracted against your target variable will, by construction, have a higher probability of producing useful splits.
Tabular Foundation Models
Tabular foundation models introduce a different consideration. These models are pretrained on structured numerical and categorical data and have developed internal representations suited to that domain. When you feed them embedding columns, those columns are treated as numerical features. The quality of what the foundation model can do with them depends entirely on whether the upstream reduction preserved the signal it needs. Task-aware reduction is therefore not just a preprocessing convenience - it determines whether the foundation model's capacity is applied to signal or noise.
The Runtime Trade-off Is Smaller Than You Expect
The practical objection to any supervised preprocessing step is that it adds training overhead and complicates the pipeline. PLS fitting requires access to labels at encoding time, which means it cannot be applied at inference time without storing the fitted transform, and it introduces a dependency on training data statistics.
These are real engineering constraints, but the cost is bounded. PLS fitting on a compressed intermediate representation is not computationally expensive relative to the encoder forward pass. Shapira et al. report that LightTAR adds a small increment to total runtime compared to PCA-only pipelines (Shapira et al., HuggingFace 2026). The fitted PLS transform is a fixed linear projection that applies at inference with the same cost as PCA. The pipeline complexity increase is real but modest.
The more substantive engineering question is whether your current pipeline architecture makes it straightforward to store and apply separate fitted transforms per column type. If your preprocessing is already modular, this is a configuration change. If it is not, this is a reasonable prompt to make it so.
Making the Infrastructure Decision
Treating dimensionality reduction as a research question rather than an infrastructure decision is where most ML teams leave performance on the table. The choice of reduction method is upstream of everything else in your pipeline. Changing it after the fact requires re-encoding, re-fitting, and re-evaluating, which is expensive if your data volumes are large.
The practical recommendation is to build your encoding pipeline with the reduction method as an explicit, swappable component rather than a hardcoded PCA call. This makes it straightforward to run controlled comparisons between PCA and PLS-augmented representations on your specific datasets and targets, which is the only way to quantify the trade-off in your context.
For pipelines that already incorporate text or image fields, running that comparison on a held-out validation set is a low-cost diagnostic. If PLS-augmented representations improve your validation metric, the case for adopting them as the production default is straightforward. If the gain is negligible, you have confirmed that PCA is sufficient for your specific embedding and target combination, which is also useful information.
Where Vector Labs Fits
We build production tabular prediction pipelines that combine structured data with unstructured text and image fields, with careful attention to what happens between the encoder and the learner. In our recruitment AI work, we integrated semantic analysis of resumes and free-text profiles with structured candidate data to power classification models that enabled hiring teams to efficiently filter candidates at scale. If you want to audit your current dimensionality reduction choices or redesign your multimodal encoding pipeline, contact us at vector-labs.ai/contacts.
FAQs
Yes. The issue is not specific to the encoder architecture - it applies to any frozen pretrained encoder whose embedding space was not trained against your prediction target. Sentence transformers optimise for semantic similarity, which correlates with your target only incidentally. PLS-augmented reduction addresses this regardless of which encoder produces the embeddings.
It can, which is why regularised PLS is the appropriate implementation. LightTAR uses regularisation specifically to limit the risk of the supervised reduction fitting to label noise rather than genuine signal. Additionally, the PLS step extracts only a small number of components relative to the full embedding dimensionality, which limits the degrees of freedom available for overfitting. Standard cross-validation discipline applies: fit the PLS transform on training data only and apply it as a fixed transform at validation and test time.
The number of PLS components is a hyperparameter that should be tuned on your validation set. In practice, the marginal gain from additional components diminishes quickly, and a small number - often in the range of five to twenty - captures most of the available supervised signal. The LightTAR approach keeps this number small by design, so that the PLS features augment rather than replace the PCA representation.
PLS has a natural multi-output extension that extracts components maximising covariance with a vector of targets simultaneously. If your pipeline predicts multiple targets from the same table, multi-output PLS can produce a single shared representation that is supervised against all targets at once. Whether that shared representation outperforms target-specific reductions depends on how correlated your targets are, and is worth evaluating empirically.
In most cases, yes. The PLS transform is a fitted linear projection, which means it can be serialised and stored alongside your PCA transform as a standard preprocessing artefact. At serving time, it applies as a matrix multiplication with no dependency on training data. The main infrastructure requirement is that your feature store can version and serve multiple fitted transforms per column, which is standard in mature feature store implementations.

