Running AI Agents on Kubernetes: The Security Architecture Gaps Most Teams Discover Too Late

Kubernetes was designed to schedule and isolate stateless workloads. An AI agent is neither stateless nor isolated in any meaningful sense: it holds credentials, calls external APIs, executes code, and makes decisions that propagate across systems. When a standard web service is compromised, the attacker inherits the service's network socket and its process memory. When an agent is compromised, the attacker inherits its tool surface, its delegated authority, and its ability to act autonomously on downstream systems. The controls that protect a REST API do not transfer to this threat model, and the gap between them is where most production incidents originate.

Companion piece to our broader work on agent identity and access design. See Scoped Credentials & Short-Lived Tokens for AI Agents for a detailed treatment of per-task credential scoping and short-lived token architecture.

The Expanded Blast Radius of Agent Compromise

A compromised agent does not behave like a compromised container. Standard container breakout scenarios concern themselves with host filesystem access and lateral movement through the cluster network. An agent adds a second attack surface on top of that: every tool registered to the agent becomes an action the attacker can invoke, every API credential the agent holds becomes an exfiltration vector, and every downstream system the agent has write access to becomes a target.

The practical consequence is that the blast radius of an agent compromise scales with the agent's capability, not with the size of the container. An agent authorised to write to a database, send emails, and call a payment API presents a fundamentally different risk profile than a container serving HTTP traffic, even if both run in the same namespace with the same resource limits.

This means the threat model must be defined before the deployment architecture is finalised. Teams that treat agent security as a hardening pass after deployment are working backwards: the permissions an agent holds, the tools it can invoke, and the credentials it carries should be determined by a formal threat model, not inherited from whatever the development environment happened to grant.

Network Egress Controls

By default, pods in Kubernetes can initiate outbound connections to any reachable address. For a stateless web service this is often acceptable. For an agent that may be manipulated through prompt injection to exfiltrate data to an attacker-controlled endpoint, unrestricted egress is a direct path to data loss.

Kubernetes NetworkPolicy resources can restrict egress to named namespaces and CIDR ranges, but they operate at Layer 3 and 4. They cannot inspect the content of a TLS-encrypted request to a legitimate external API. For agents that must call external services, the appropriate control is an explicit egress allowlist enforced at the DNS level, combined with a forward proxy that terminates and re-inspects outbound TLS. Tools such as Squid or a service mesh with egress gateway configuration can provide this, though they add operational overhead that must be planned for.

The commercial implication is straightforward: a data exfiltration event through a compromised agent is a regulatory incident under GDPR and similar frameworks, not just an operational failure. The cost of implementing egress controls before an incident is bounded and predictable. The cost of a reportable breach is not.

Pod Security Admission and Privilege Constraints

Admission Configuration

Kubernetes Pod Security Admission (PSA), which replaced PodSecurityPolicy in Kubernetes 1.25, enforces security profiles at the namespace level using three built-in policy levels: privileged, baseline, and restricted. For agent workloads, the restricted profile should be the starting point, not an aspirational target. It disallows privilege escalation, requires non-root execution, drops all Linux capabilities, and requires a read-only root filesystem where possible.

The reason teams defer this is that LLM inference containers and agent frameworks are often built against assumptions of root access inherited from development environments. Remediating those assumptions takes engineering time. The alternative is running production agents with privilege escalation enabled, which means a container breakout gives the attacker root on the node.

Service Account Scoping

Every pod in Kubernetes is assigned a service account, and by default that service account's token is mounted into the pod's filesystem. An agent process that reads its own filesystem can find its own Kubernetes API credentials. Those credentials, if the service account has broad RBAC permissions, give an attacker access to the cluster control plane.

The fix is to set automountServiceAccountToken: false on pods that do not require Kubernetes API access, and to create dedicated service accounts with minimal RBAC grants for those that do. Token expiry should be set explicitly using the expirationSeconds field on projected volume tokens rather than relying on the default, which is not short-lived.

Sandboxed Runtimes for Code-Executing Agents

Agents that execute generated code, whether through a code interpreter tool or a sandboxed shell, present a qualitatively different risk than agents that only call APIs. Code execution is arbitrary by definition: the agent can be manipulated into running commands that escape the intended tool boundary.

The standard container runtime, runc, provides namespace and cgroup isolation but shares the host kernel. A kernel exploit from within a container running runc can affect the host. For code-executing agents, this is an unacceptable risk surface. The appropriate runtime is a VM-based or kernel-isolated alternative: gVisor (runsc) interposes a user-space kernel between the container and the host, while Kata Containers runs each pod inside a lightweight VM with a separate kernel.

Both options carry a performance overhead. gVisor adds syscall interposition latency that is measurable for I/O-intensive workloads. Kata Containers adds VM startup time that matters for short-lived tasks. For agents that execute code infrequently as part of a longer reasoning loop, the overhead is generally acceptable. The decision should be made against a measured baseline, not estimated.

Credential Lifecycle Design

Agents in production almost always require credentials: API keys for LLM providers, database connection strings, credentials for the tools they operate. The failure mode that appears most frequently in post-incident reviews is long-lived shared credentials stored as Kubernetes secrets, mounted into pods, and never rotated. A secret that has existed for eighteen months and is shared across multiple agent deployments cannot be revoked without disrupting all of them simultaneously.

The structural fix is to issue short-lived, per-task credentials through a secrets management system such as HashiCorp Vault or AWS Secrets Manager, using Kubernetes service account tokens as the authentication mechanism for the secrets backend. The agent authenticates to the secrets backend at runtime, receives a time-bounded credential scoped to the specific operation, and that credential expires automatically when the task completes. This limits the window of exposure for any single credential to the duration of a single task.

Credential design also intersects with audit requirements. A shared API key that is used by multiple agent instances produces an audit trail that cannot be attributed to a specific agent action. Per-instance, per-task credentials make attribution exact, which matters both for internal incident investigation and for regulatory compliance in sectors where action-level audit trails are mandated.

We have covered the full architecture for scoped credential issuance and short-lived token design in detail in our published work on runtime access for agentic systems.

Observability as a Security Control

Kubernetes-native logging captures container stdout and cluster events. It does not capture what an agent decided, which tools it called, what arguments it passed to those tools, or what the tool returned. Without that data, detecting anomalous agent behaviour requires inferring intent from network logs and API call records, which is slow and unreliable.

The practical requirement is an agent-level audit log that records each reasoning step, each tool invocation, the full argument payload, and the result. This log should be written to an append-only sink that the agent process cannot modify or delete, and it should be ingested into the same SIEM pipeline as cluster security events. Correlating a suspicious outbound connection with the specific tool call and reasoning trace that preceded it reduces investigation time from hours to minutes.

The secondary benefit is that this audit infrastructure is also what regulators in financial services and healthcare will ask for when they review an agentic system deployment. Building it as a security control from the start avoids the cost of retrofitting it under compliance pressure.