Cost Exhaustion, Agent Swarms, and Evidence Custodians: The AI Risk Categories Your Governance Framework Is Not Built For

Enterprise AI governance has spent the better part of three years refining its response to bias, explainability, and data privacy. Those are real problems, and the frameworks built around them have genuine value. But the risk landscape is shifting faster than most governance teams have acknowledged, and the next category of failures will not arrive as a discriminatory output or an unexplainable decision. They will arrive as a drained cloud budget, a public-facing system overwhelmed by coordinated agent traffic, and a legal team asking who is responsible for a decision that no human reviewed. Gartner's predictions for the 2027 to 2030 window identify these structural failure modes explicitly. Most enterprise governance frameworks are not built to absorb any of them.

Cost Exhaustion Is an Attack Surface, Not Just a Budget Problem

Agentic systems are designed to act autonomously over extended task horizons. That autonomy is precisely what makes them useful, and precisely what makes them expensive to attack. A cost exhaustion attack does not require compromising a model or stealing data. It requires triggering an agent into a high-cost execution loop, either through adversarial prompt injection, a malformed input that causes recursive tool calls, or simply an edge case the agent was not designed to exit gracefully.

The financial exposure here is not theoretical. Agents that invoke external APIs, spawn sub-agents, or iterate through large context windows can accumulate costs at a rate that outpaces any manual monitoring threshold. The mechanism is straightforward: because agentic systems are built for persistence, they will keep running until a termination condition is met, and if that condition is poorly specified, the default behaviour is to continue.

Governance frameworks that treat compute cost as an infrastructure concern rather than a security concern are structurally unprepared for this. Budget controls need to sit at the agent policy layer, not the finance reporting layer. Hard token limits, cost-per-task ceilings, and automated circuit breakers are not optimisation tools. They are risk controls, and they belong in the governance architecture.

Agent Swarms and the Fragility of Public-Facing Systems

The term "agent swarm" describes what happens when multiple autonomous agents operate in parallel against shared resources or interfaces. In internal deployments, this is a coordination challenge. In public-facing systems, it is a load and integrity problem that most organisations have not modelled.

Consider a customer service environment where agents are handling inbound queries, escalating to human review, triggering backend workflows, and logging decisions simultaneously. At low volume, the system behaves predictably. At swarm scale, the interaction effects between agents competing for shared state, shared APIs, and shared human review queues produce failure modes that do not appear in any single-agent evaluation. The mechanism is emergent: individual agents behave correctly, but their aggregate behaviour degrades system integrity.

Governance frameworks built around evaluating individual models have no natural vocabulary for swarm-level risk. Organisations deploying agentic pipelines at scale need to run adversarial load simulations that model agent-to-agent interaction, not just agent-to-user interaction. That requires a different testing discipline and a different set of questions at the architecture review stage.

The Evidence Custodian Problem

When an AI system makes or materially influences a consequential decision, and that decision is later disputed, someone has to produce the record. In regulated industries, that obligation is already arriving in the form of audit requirements, subject access requests, and litigation discovery. The question of who holds that responsibility is the evidence custodian problem, and most organisations have not assigned it.

The difficulty is not just organisational. Agentic systems often do not produce clean, human-readable decision logs. They produce token streams, tool call histories, and intermediate reasoning states that are technically recoverable but practically opaque without purpose-built logging infrastructure. The mechanism is architectural: because agents are designed for task completion rather than auditability, the audit trail is a secondary output that has to be deliberately engineered.

Governance frameworks need to designate an evidence custodian function before deployment, not after a dispute arises. That function requires defined log retention standards, chain-of-custody documentation for AI-influenced decisions, and the technical infrastructure to reconstruct a decision trace on demand. Treating this as a post-incident concern is the same category of error as treating data backup as a post-breach concern.

Disposable Application Economics and the Governance Gap They Create

Gartner's framing of disposable AI applications describes a world where short-lived, task-specific AI agents are spun up, used, and discarded at a rate that makes traditional software governance cycles irrelevant. The economic logic is sound: if an agent can be built in hours and retired in days, the overhead of a full governance review becomes a competitive liability. The risk is that governance gets skipped entirely rather than adapted.

The structural problem is that disposable applications are not consequence-free simply because they are short-lived. An agent that processes customer data for 48 hours and is then retired has still processed customer data. If it produced outputs that influenced downstream decisions, those outputs persist even after the agent does not. The mechanism is one of temporal asymmetry: the agent's lifespan is short, but the consequences of its operation are not.

Governance frameworks need a lightweight, rapid-review track that is proportionate to deployment duration and consequence severity, without defaulting to either a full review cycle or no review at all. The calibration criteria should be consequence-based: what decisions does this agent influence, what data does it touch, and what happens if it fails. Those three questions can be answered in a structured intake form. The absence of that intake form is the governance gap.

Rebuilding Governance Architecture Around Operational Resilience

The common thread across cost exhaustion, agent swarms, evidence custodians, and disposable applications is that they are all operational and financial risks, not primarily ethical or compliance risks. The governance frameworks built to address bias and explainability are not wrong. They are simply pointed at a different category of problem.

Rebuilding governance architecture to absorb these failure modes requires treating agentic deployment as an operational resilience question from the first design review. That means cost controls at the policy layer, swarm-level load testing before public deployment, evidence custodian designation as a precondition for go-live, and a rapid-review intake process for short-lived agents. None of these are exotic requirements. They are the engineering discipline that production AI systems already demand, applied to a governance context that has not yet caught up.

The organisations that will absorb these risks most effectively are not those with the most sophisticated ethics committees. They are those that have embedded governance checkpoints into their deployment pipelines at the point where architectural decisions are still reversible. That is a process design problem, and it has a tractable solution.

Companion piece to our broader work on AI governance at scale. See Shadow AI Agents in Enterprise: Governance Gaps for an analysis of how undetected agents already operate within enterprise infrastructure and where existing governance frameworks fail to surface them.