Wearable AI Assistants in the Enterprise: What MLLMs Can and Cannot Actually Do in Physical Workflows

Enterprise interest in wearable AI assistants has accelerated well ahead of the evidence base for deploying them. Headsets and smart glasses that stream a worker's first-person view to a multimodal large language model, which then narrates the next step in a procedure, sound operationally compelling. The problem is that most evaluations of these systems happen in controlled vendor demonstrations, not under the conditions that define real operational value: a real person, under time pressure, in a noisy environment, receiving an instruction that may be ambiguous, incomplete, or simply wrong. Before committing capital to a wearable AI programme, engineering leaders need a clearer picture of where today's models actually stand.

The Architecture That Makes Wearable AI Different

Wearable AI assistants are not chatbots with cameras attached. The architecture that matters is closed-loop: the assistant observes the scene through the worker's egocentric view, issues an instruction, the worker acts on it, the environment changes, and the assistant must update its reasoning based on what it now sees. Each instruction shapes the next state of the world.

This is categorically different from a one-shot visual question-answering task. In a single-turn query, a model can fail gracefully by hedging or asking for clarification. In a multi-turn physical workflow, a bad instruction at step three cascades into a corrupted state at step seven. The assistant cannot simply retry.

The distinction matters commercially because most vendor benchmarks evaluate models on single-turn or offline tasks. A model that scores well at describing what it sees does not necessarily score well at guiding a person through a twelve-step assembly procedure where the outcome of each step determines what the next instruction should be.

What Rigorous Evaluation Actually Looks Like

The research community has begun to address this gap directly. InteractiveWorld, developed by researchers at Meta and Duke University, is the first simulation environment designed specifically for closed-loop human-AI assistant interaction, as opposed to robot-centric simulators where the AI controls physical actuators (Wang et al., HuggingFace 2026). The distinction is important: in InteractiveWorld, the human agent observes the scene, interprets the instruction, and acts independently. The assistant's job is perception, memory, reasoning, and communication, not motor control.

The associated benchmark, InteractiveBench, covers 120 multi-turn tasks across six categories, scored by verifiable state-based rewards rather than human raters. This matters because human rater evaluations introduce subjectivity and cannot scale to the volume of interaction data needed to characterise model behaviour across task types.

The results from testing twelve frontier MLLMs are instructive. The strongest model, Claude Fable 5.1, achieved at most 20% on long-horizon Guidance tasks and 25% on Planning tasks (Wang et al., HuggingFace 2026). These are not edge-case failures. They represent the ceiling of current capability on tasks that are structurally similar to what wearable assistants would face in a manufacturing or logistics environment.

Where Today's Models Fall Short

Long-Horizon Reasoning

The performance gap on Guidance and Planning tasks is not primarily a perception problem. Models can generally identify objects and describe scenes with reasonable accuracy. The failure mode is temporal coherence: maintaining an accurate model of task state across many turns, tracking what has already been done, and issuing instructions that are consistent with the current world state rather than the initial one.

This is a fundamental architectural challenge. Current MLLMs do not have persistent working memory in the way a human expert does. They reconstruct context from the conversation history and the current frame, which means errors accumulate as task length increases.

Instruction Precision Under Ambiguity

A second failure mode is instruction quality under spatial ambiguity. When a scene contains multiple similar objects, or when the correct action depends on a detail that is not clearly visible in the egocentric frame, models tend to issue instructions that are technically plausible but operationally incorrect. A human expert would ask a clarifying question or flag the ambiguity. Current models frequently do not.

Feedback Interpretation

The closed-loop architecture also requires the assistant to interpret what the worker has done from the updated egocentric view and adjust accordingly. Models that perform adequately on static scene understanding often degrade when required to infer the outcome of an action from a post-action frame. The assistant needs to determine not just what it sees, but what changed and whether that change was the intended one.

What to Verify Before Deployment

Engineering leaders evaluating wearable AI vendors should treat any benchmark that does not involve closed-loop multi-turn interaction with a human agent as insufficient evidence of operational readiness. The specific questions worth pressing on are:

• What is the model's task completion rate on multi-step procedures, not scene description accuracy?
• How does performance degrade as task length increases, and at what turn count does it fall below an acceptable threshold?
• How does the system behave when the worker takes an incorrect action, and can it recover the task state?
• What latency does the instruction loop introduce, and is that latency acceptable given the pace of the workflow?

These are not questions that vendor demos are designed to answer. They require structured evaluation under conditions that approximate the target deployment environment, which is precisely the methodological contribution of frameworks like InteractiveWorld.

The Simulation-First Evaluation Strategy

Simulation-based evaluation is not a workaround for the absence of real data. It is the appropriate first stage of a deployment evaluation for any system where failure has operational consequences. The cost of discovering that a model loses task coherence after eight turns is very different depending on whether you discover it in simulation or on a production floor.

The practical implication is that engineering teams should insist on access to evaluation frameworks that allow them to run their own task scenarios, not just review vendor-provided results. If a vendor cannot provide that access, the evaluation is incomplete by definition.

Once simulation-based thresholds are met, controlled field trials with a small number of workers on non-critical tasks provide the next layer of evidence. The progression from simulation to limited field deployment to broader rollout is not conservative, it is the minimum defensible approach given what the current performance data shows.