The Inference Gap: Why Encrypting AI Prompts at Rest and in Transit Is Not Enough for Enterprise Security Teams

Enterprise security teams have spent years building layered encryption strategies, and most have done it well. TLS in transit, AES-256 at rest, key management policies that would satisfy any auditor. What those controls do not address is the moment a hosted AI model actually reads your data. At inference time, the prompt is decrypted into plaintext, loaded into GPU or CPU memory, and processed in an environment that the enterprise does not control. That window is where the real exposure sits, and closing it requires a different class of architectural decision than encryption policy alone can provide.

What Encryption Does Not Cover

Encryption protects data while it moves between systems and while it sits on disk. It is not designed to protect data while it is being used. This distinction is fundamental to cryptography and is not a vendor limitation. It is a property of how computation works.

When a hosted model receives a prompt containing regulated data, that prompt must be decrypted before the model can process it. The decrypted content lives in memory on infrastructure owned and operated by the model provider. Hypervisors, orchestration layers, and privileged operators on that infrastructure can, in principle, access that memory. Whether any given provider has strong controls against that access is a separate question from whether the structural exposure exists. It does.

The Plaintext Memory Problem at Scale

The exposure surface grows with usage volume. A single enterprise deployment sending thousands of prompts per day creates a persistent plaintext memory footprint across shared inference infrastructure. Each prompt represents a discrete window of exposure, and those windows accumulate.

GPU memory is particularly relevant here. Large language models run on GPU clusters where memory is pooled and reused across inference requests. Without explicit memory isolation and scrubbing between requests, residual data from one inference run can persist in memory regions allocated to the next. Most hosted inference APIs do not publish their memory isolation guarantees at this level of specificity, which means security teams are making implicit trust assumptions they may not have consciously evaluated.

The commercial implication is straightforward. If your organisation processes HIPAA-covered health data, PCI-scoped payment records, or proprietary deal information through a hosted model, the question is not whether your TLS configuration is correct. The question is what happens to that data after decryption and before the response is returned.

The Identity Federation Attack Surface

The inference gap does not begin at the model API. It begins at authentication. Most enterprise AI deployments federate identity through SSO, connecting corporate identity providers to model provider accounts via SAML or OIDC. That chain introduces a category of risk that sits upstream of any inference-time control.

SSO Token Exposure

A compromised SSO token grants access to the model provider account with the same permissions as the federated user. If that user has broad prompt access or can retrieve conversation history, the attacker inherits that access without touching the model provider's own authentication systems. The OpenAI internal systems breach, which involved a threat actor accessing internal discussion forums through a compromised employee account, illustrates how identity federation creates lateral movement paths that bypass infrastructure-level controls entirely.

Privilege Scope and Least-Access Configuration

Federation configurations in enterprise AI deployments frequently inherit overly broad permission scopes from initial setup. Access reviews that cover SaaS applications often do not extend to AI API credentials or model provider accounts with the same rigour. The result is a set of standing privileges that exceed operational requirements and expand the blast radius of any credential compromise.

Confidential Computing as a Structural Response

Confidential computing addresses the plaintext memory problem at the hardware level by executing computation inside a trusted execution environment (TEE). A TEE isolates the memory used during inference from the hypervisor, the host operating system, and privileged operators. The model provider cannot read the contents of the enclave, even with root access to the underlying host.

TEE-Based Inference

TEE-based inference is available from a small number of providers and is an active area of development across cloud infrastructure. The trade-off is performance. TEEs introduce overhead in memory bandwidth and context switching, which translates to higher latency and lower throughput at inference time. For latency-sensitive applications, that cost is material.

Single-Tenant Inference Deployments

Single-tenant inference deployments offer a different approach. Rather than shared infrastructure with isolation guarantees, the enterprise runs the model on dedicated compute that no other customer's workloads share. This eliminates cross-tenant memory exposure and gives the security team a clearer boundary to audit. The trade-off is cost and operational overhead. Single-tenant deployments are significantly more expensive and require the enterprise to manage more of the infrastructure stack, including model updates and availability.

Neither approach eliminates all risk. TEEs protect memory but require trusting the TEE implementation and the attestation process. Single-tenant deployments reduce shared-infrastructure exposure but shift operational responsibility to the enterprise team.

The Security Decisions That Must Precede Deployment

Before regulated data touches a hosted model, security teams need answers to a specific set of questions that are distinct from standard vendor security reviews.

The first is memory isolation. Does the provider offer documented, auditable guarantees about memory scrubbing between inference requests, and at what layer of the stack are those guarantees enforced?

The second is operator access. What access do privileged operators at the model provider have to in-flight inference data, and is that access logged and auditable by the enterprise?

The third is identity chain scope. What permissions does the federated identity configuration grant, and when were those permissions last reviewed against the principle of least access?

The fourth is data residency during inference. In which geographic regions does inference execution occur, and does that create regulatory jurisdiction exposure for the data being processed?

These questions do not have universally correct answers. They have answers that are either acceptable or unacceptable given the sensitivity classification of the data involved and the regulatory framework the organisation operates under. The security decision is matching data classification to inference architecture, not assuming that a provider's SOC 2 certification covers the inference-time exposure described here.