The gap between what enterprises believe about their AI agents and what is actually happening in production is not a minor calibration issue. Gartner's 2025 survey of infrastructure and operations leaders found that 70% believe their agents operate safely, while 95% of the same group reported agent-related incidents in the preceding year. That is not a rounding error in self-assessment. It is a structural failure of observability, and it is the exact kind of failure that external auditors are trained to find.
Source: Gartner, AI Infrastructure and Operations Survey, 2025. Survey of I&O leaders across enterprise organisations on agentic AI deployment and incident reporting. gartner.com
Companion piece to our broader work on agent operational readiness. See What Long-Running Agents Expose About Engineering Team Readiness for a practical analysis of the workflow and trust-calibration gaps that surface when teams move from short-context AI assistance to autonomous agents.
The Embedded Assessor Model Is a Signal, Not a Curiosity
OpenAI's embedded assessor programme places independent evaluators inside deployments to observe model behaviour under real operating conditions. The frontier labs built this because internal red-teaming and pre-deployment benchmarks consistently missed failure modes that only appear at scale, under adversarial prompting, or when agents interact with tools and APIs in sequences that no test harness anticipated.
Enterprise leaders should read this not as a frontier-lab housekeeping exercise but as a preview of what external scrutiny will look like when it reaches commercial deployments. The embedded model exists because point-in-time audits against static documentation are insufficient for systems that learn, adapt, and take actions in the world. Regulators and insurers studying the frontier labs are watching the same evidence.
The practical implication is that the audit model coming toward enterprise AI is continuous and behavioural, not periodic and documentary. Organisations that prepare only a governance policy document will be exposed the moment an assessor asks to observe live agent behaviour.
Why the Confidence Gap Persists
The 70/95 disconnect from Gartner is not primarily a cultural problem. It is an instrumentation problem. Most enterprise agentic deployments lack the telemetry to distinguish between an agent completing a task correctly and an agent completing a task in a way that produces the right output through a sequence of actions that should have triggered a human review.
When teams cannot observe the decision path, only the outcome, they conflate output quality with operational safety. An agent that files the correct purchase order after querying three internal systems and one external API may have operated within policy. It may also have exfiltrated data, escalated its own permissions, or triggered a downstream workflow that no one intended. Without action-level logging, you cannot tell.
The mechanism here is straightforward: safety confidence tracks output monitoring, but safety risk tracks action monitoring. Enterprises that have invested in the former without the latter are measuring the wrong thing.
The Governance Infrastructure That Assessors Will Expect
When external assessment becomes standard practice, whether through EU AI Act enforcement, insurance underwriting requirements, or sector-specific regulation, assessors will arrive with a consistent set of questions. Organisations that can answer them with instrumented evidence rather than policy assertions will be in a materially different position.
Action-Level Observability
Every action an agent takes against an external system, internal API, or data store should produce a structured log entry that captures the action type, the authorisation state at the time of execution, the data accessed, and the agent's reasoning trace where the model supports it. This is not a nice-to-have. It is the minimum evidentiary record an assessor needs to reconstruct what happened in an incident.
Permission Boundary Enforcement
Agents should operate under dynamically scoped permissions that are granted per task and revoked on completion, not persistent credentials that accumulate over time. Static service accounts with broad access are the single most common source of agent-related incidents, because they allow an agent operating outside its intended scope to succeed where it should have been stopped.
Policy-as-Code With Version History
Governance policies that exist only as prose documents cannot be tested, diffed, or audited against a specific deployment version. Encoding agent behavioural constraints as machine-readable policy, with version control tied to deployment history, gives assessors the ability to verify that the policy in place at the time of an incident was the policy that should have been in place.
The 24-Month Horizon and What It Demands Now
The EU AI Act's provisions for high-risk AI systems are already in force for new deployments, and the scope of what qualifies as high-risk is being interpreted more broadly than most enterprise legal teams initially anticipated. Sector regulators in financial services, healthcare, and critical infrastructure are developing their own agentic AI guidance in parallel. The convergence of these timelines puts meaningful external scrutiny of enterprise agent deployments within a 24-month window for most large organisations.
The organisations that will handle that scrutiny well are not the ones that start preparing when the first audit letter arrives. They are the ones that treat the embedded assessor model as a design requirement today, building the observability, permission, and policy infrastructure into their agent platforms while they still have the freedom to architect it correctly.
Retrofitting audit capability into a production agent fleet is significantly more expensive and disruptive than building it in from the start. The cost of instrumentation is a fraction of the cost of a regulatory finding or an insurance claim that cannot be defended because the logs do not exist.
What to Prioritise in the Next Quarter
The practical sequence for most enterprises is not to rebuild their agent infrastructure from scratch. It is to close the most exposed gaps first, in order of the evidence an assessor would seek earliest.
Start with action logging. If your agents are taking actions against external systems without structured, queryable logs, that is the highest-priority gap. Move next to permission scoping, replacing persistent credentials with task-scoped access where your architecture allows. Then formalise your policy layer, converting prose governance documents into versioned, testable constraints that can be mapped to specific deployment configurations.
None of these steps require waiting for a regulatory mandate. They are the same steps that any organisation serious about operational reliability would take regardless of external pressure. The difference is that the external pressure is now on a defined timeline, and the organisations that treat it as a future problem are accumulating a governance debt that will be called in.
Where Vector Labs Fits
We build AI systems with audit and certification requirements embedded from the design stage, not added as a compliance layer after deployment. In our cardiovascular certification work, we structured validation to meet medical device software standards from the outset, achieving Class 2A certification within the product launch timeline. If you are preparing your agent infrastructure for external assessment and want to close the gap before it is forced, contact us at vector-labs.ai/contacts.
FAQs
Internal red-teaming tests behaviour against anticipated threat models, usually before deployment. Independent assessment observes behaviour under real operating conditions, including edge cases and tool-use sequences that internal teams did not anticipate. The embedded assessor model is specifically designed to surface failure modes that only appear in production, which is why it produces findings that internal processes consistently miss.
The EU AI Act is the most immediate, with high-risk system provisions already applying to new deployments and a broader scope than many organisations initially planned for. Sector regulators in financial services and healthcare are developing parallel guidance, and insurance underwriters are beginning to condition coverage on demonstrable governance instrumentation. The pressure will arrive from multiple directions, not a single regulatory event.
At minimum, each agent action against an external system or data store should produce a structured log entry capturing the action type, the identity and permission state of the agent at execution time, the data accessed or modified, and a timestamp. Where the model supports it, a reasoning trace or chain-of-thought record should be appended. These logs need to be queryable and tamper-evident to serve as evidentiary records in an incident review.
It is feasible at a basic level without a dedicated platform team, but it requires deliberate tooling choices. Several agent orchestration frameworks support policy constraint definitions in structured formats that can be version-controlled alongside deployment configuration. The key requirement is that the policy representation is machine-readable and tied to a specific deployment version, so that any assessor can verify what constraints were active at the time of a given incident.
Sequence by the evidence an assessor would seek first. Action logging against external systems is the highest priority because it is the gap most likely to prevent you from defending an incident. Permission scoping comes next, because persistent broad-access credentials are the most common source of agent-related incidents and the easiest for an assessor to identify. Policy formalisation can follow once the observability and access control layers are in place.

