Why Your Data Lake Format Choices Are Now an MLOps Architecture Decision

Data lake format decisions have historically lived in the data engineering backlog, surfaced during storage cost reviews or migration projects, and rarely escalated to ML leadership until something breaks in production. That pattern is no longer acceptable. Changes to open table formats like Apache Parquet and Apache Iceberg now directly shape how ML pipelines version features, prune training partitions, and reproduce dataset snapshots at scale. If your Head of ML is not in the room when these format decisions are made, they will be handed a set of constraints they did not choose and cannot easily reverse.

What Format-Level Changes Actually Mean for ML Pipelines

The distinction between a data engineering concern and an MLOps concern is not as clean as most organisations assume. When Parquet introduced support for decoupled row group boundaries, the change was framed as a storage efficiency improvement. In practice, it altered how downstream readers materialise column chunks, which affects the granularity at which feature pipelines can selectively load training windows without full file scans.

This matters because feature stores built on top of Parquet-backed lakes often rely on row group statistics to skip irrelevant partitions during training data retrieval. If the row group boundaries no longer align with the partitioning scheme the feature store was designed around, predicate pushdown becomes unreliable. The result is silent performance degradation: queries complete, but they read far more data than intended, and training jobs slow down in ways that are difficult to attribute to the format change.

Iceberg's Stats-Only Fields and Training Data Versioning

Apache Iceberg's metadata layer is where the MLOps implications become most concrete. Iceberg maintains per-column statistics at the manifest level, including null counts, value bounds, and row counts, without requiring a full table scan to access them. These stats-only fields were designed for query optimisation, but they carry significant weight for partition-aware training data versioning.

Snapshot Isolation and Reproducibility

Iceberg's snapshot model gives every table state a unique identifier, which is a genuine asset for training data reproducibility. A training job can be pinned to a specific snapshot, and that snapshot's manifest files encode the column statistics at the time of writing. This means that data quality checks and partition filters applied during training can be reconstructed from the metadata alone, without re-reading the underlying Parquet files.

The risk arises when teams evolve their partition strategies without accounting for how snapshot metadata is inherited. If a new partition column is added mid-project, older snapshots do not carry statistics for that column. A pipeline that relies on stats-only pruning for the new column will fall back to full partition scans against historical snapshots, which breaks the performance assumptions that were valid for recent data.

Manifest List Evolution and Feature Store Layout

Feature stores that organise features by entity and timestamp need their physical layout to align with how Iceberg constructs manifest lists. When manifest list ordering diverges from the feature store's lookup pattern, the overhead of manifest file traversal compounds across every training batch. This is not a theoretical concern: it is the kind of issue that appears as intermittent latency spikes during large-scale training runs and is rarely traced back to the format layer without deliberate instrumentation.

Partition Pruning as a First-Class MLOps Concern

Partition pruning is typically treated as a query optimisation technique. In ML contexts, it is also a correctness mechanism. Training pipelines that use time-based partitions to enforce training and validation cutoffs depend on the format layer to honour those boundaries precisely.

When format changes alter how partition metadata is written or cached, a pipeline that previously excluded future-leaking data may silently include it. The model trains without error, but the evaluation metrics no longer reflect real-world performance. This class of failure is particularly difficult to detect because it does not produce exceptions; it produces subtly optimistic validation scores.

What Engineering Leaders Need to Track

The format-level changes that carry the highest MLOps risk share a common pattern: they improve general-purpose query performance while introducing assumptions about read patterns that ML workloads do not always satisfy. The items worth tracking before they become production bottlenecks include:

• Row group size defaults and whether they are being tuned per-table or inherited globally
• Iceberg manifest compaction schedules and their interaction with snapshot retention policies
• Column statistics coverage across partition evolution events
• The version of the Parquet and Iceberg readers embedded in your training framework versus your serving infrastructure

Mismatches between the reader versions used during training and those used during inference are a common source of subtle data skew. Format upgrades that introduce new encoding types or statistics fields may be read differently by older client libraries, and the difference is rarely surfaced by standard data validation checks.

Making Format Decisions a Shared Architecture Concern

The practical fix is not to make ML engineers responsible for data lake format governance. It is to establish a shared review process where format changes are evaluated against ML pipeline contracts before they are deployed. That means defining, explicitly, what each training pipeline expects from the format layer: partition granularity, statistics coverage, snapshot retention windows, and reader version compatibility.

When those contracts are written down, format changes become testable against ML workloads before they reach production. Without them, the data engineering team optimises for storage cost and query latency, and the ML team inherits whatever assumptions those optimisations introduced. The gap between those two sets of priorities is where production incidents originate.

Companion piece to our broader work on ML data pipeline architecture. See Your ML Data Pipeline in 2026: A New Architecture for how GPU-native processing and real-time lakehouse serving are reshaping the full pipeline stack.