The Enshittification Problem: Why Enterprise AI Products Are Repeating Big Tech's Worst Mistakes

The consumer internet spent two decades optimizing itself into user hostility. Platforms that began as genuinely useful tools gradually reoriented around engagement metrics, behavioral capture, and data retention, until the product experience became a byproduct of the surveillance infrastructure rather than the point of it. Enterprise AI is now following the same trajectory, faster and with less public scrutiny. CTOs who assume that operating in a B2B context insulates them from this dynamic are misreading where the pressure is coming from and what it will cost them.

The Structural Pressure That Produces Bad Defaults

The mechanism here is not malice. It is incentive misalignment operating at the architectural level. AI product teams need data to improve models, product managers need engagement signals to justify roadmap decisions, and the path of least resistance is to capture everything by default and sort out the policy later. Each individual decision looks reasonable in isolation. The aggregate produces a system that treats user behavior as a resource to be mined rather than a trust relationship to be maintained.

This dynamic has a name in the consumer context: Cory Doctorow's "enshittification" describes how platforms degrade by first serving users, then advertisers, then extracting value from both. Enterprise AI is not running the same playbook on the same timeline, but the structural logic is identical. The difference is that enterprise users have procurement leverage, legal teams, and the ability to walk.

Where the Architecture Goes Wrong

Consent as Friction Reduction

The first failure point is consent architecture. Most enterprise AI products treat consent flows as onboarding friction to be minimized rather than as a meaningful communication of data handling intent. Opt-out defaults for training data usage, vague references to "product improvement" in terms of service, and the burial of data retention settings in admin panels that most users never open are not neutral design choices. They are choices that favor the vendor's data accumulation interests over the user's right to understand what is being done with their inputs.

The commercial risk is not abstract. Enterprise procurement teams at regulated industries are now routinely asking vendors for explicit documentation of what employee-generated data is retained, for how long, and whether it is used to train shared models. Vendors who cannot answer these questions clearly are losing deals to those who can.

Behavioral Capture by Default

The second failure is the normalization of behavioral telemetry as a background process that users neither see nor control. Feature usage tracking, query logging, and session recording serve legitimate product development purposes. The problem is when these systems are architected so that the data flows are opaque, the retention periods are indefinite, and the user has no meaningful way to inspect or limit what is collected.

In enterprise contexts, this creates a specific governance problem. When an employee uses an AI assistant to draft a sensitive internal document, the question of whether that content is retained, where it is stored, and who at the vendor can access it is not a hypothetical. It is a question that legal and security teams are already asking, and the answer "we collect it for model improvement" is increasingly treated as a disqualifying response.

The Platform Optimization Trap

AI product teams face a version of the same tension that consumer platforms faced with algorithmic feeds: optimizing for the metric that is easy to measure tends to degrade the outcome that actually matters. For AI assistants, the easy metric is engagement. The metric that matters is whether the user accomplished something useful and trusted the system enough to use it again on something sensitive.

When product decisions are driven by engagement optimization, the result is an AI product that nudges users toward more interactions rather than more efficient ones, that surfaces suggestions designed to extend sessions rather than resolve tasks, and that treats user dependency as a success indicator rather than a design failure. This is not a hypothetical pattern. It is already visible in how some AI writing and productivity tools are instrumented and how their success metrics are reported internally.

What Engineering Leaders Must Build Differently

The architectural commitments that prevent this outcome are not complicated, but they require treating privacy and user control as first-class design constraints rather than compliance additions bolted on after the product ships.

Data minimization by default means collecting only what is necessary for the immediate task and requiring an explicit, affirmative decision to retain anything beyond that. Transparent data flows mean that any user or administrator can inspect, at a granular level, what has been collected, where it is stored, and what it has been used for. User-controlled retention means that deletion requests are honored at the data layer, not just at the interface layer, and that this is verifiable.

These are engineering decisions, not policy decisions. A system architected to make data minimization the default is structurally different from one that adds a privacy settings page to a system designed for maximum capture. The former requires less remediation when regulation tightens. The latter requires a rewrite.

The Regulatory and Commercial Timeline

The EU AI Act's requirements around transparency and human oversight are already in force for high-risk systems, and enforcement posture across European data protection authorities has hardened considerably since 2023. US state-level privacy legislation is expanding the scope of what counts as sensitive data in ways that directly affect AI-generated behavioral profiles. The regulatory window for treating data architecture as an afterthought is closing, and it is closing faster for enterprise vendors whose customers are themselves regulated entities.

The commercial signal is already visible in procurement behavior. Enterprise customers in financial services, healthcare, and legal services are writing data handling requirements into vendor contracts at a level of specificity that was rare three years ago. Vendors who have built these commitments into their architecture can respond to these requirements with documentation. Vendors who have not are discovering that "we can configure that" is not a sufficient answer when the customer's legal team wants to audit the implementation.

CTOs who make these architectural commitments now are building a defensible position for a market that is moving in a predictable direction. Those who treat it as a future compliance problem are accumulating technical and commercial debt simultaneously, and the cost of resolving both at the same time is considerably higher than addressing either proactively.