Physical AI Is Harder Than It Looks: What Robotics Research Tells Enterprise Buyers Before They Commit

Enterprise buyers evaluating physical AI platforms are, in most cases, making seven-figure commitments based on vendor demonstrations that have been optimised for a controlled lab setting. The gap between a polished pick-and-place demo and a production deployment in a real warehouse is not a matter of engineering time. It is a matter of fundamental unsolved problems that current research is still actively working through. Understanding where those problems sit, and what they cost, is the prerequisite for any serious due diligence process.

Companion piece to our broader work on physical AI deployment risk. See Physical AI Deployments: Why Robots Fail Where Software Succeeds for a wider treatment of why VLA systems break in production and what the hardware trade-offs look like.

The Precision Problem Starts Below the Policy Layer

Most enterprise buyers focus their evaluation on the learned policy: the vision-language-action model that decides what the robot should do. The more consequential failure point is often one layer below that, in the inverse kinematics controller that converts pose commands into actual joint motion.

Why Millimetre Errors Compound

IK solvers must handle kinematic redundancy and singularities, and classical controllers frequently leave residual position errors that are small in isolation but catastrophic in assembly or precision logistics tasks. Research from New York University Abu Dhabi shows that even well-tuned classical IK controllers fail to hold end-effector positions within one millimetre reliably, and that reinforcement learning approaches can substantially close this gap (Liu et al., arXiv 2026). The commercial implication is direct: any VLA policy that assumes the IK layer will faithfully execute its pose commands is implicitly assuming a level of precision that many deployed controllers do not deliver.

The practical question for buyers is whether a vendor's benchmark success rates were measured end-to-end on physical hardware, or whether they assume a simulation-grade IK solver. These are not the same thing, and vendors rarely volunteer the distinction.

Training Data Collection Is More Expensive Than Quoted

Imitation learning policies require large volumes of teleoperation demonstrations, and the quoted data collection cost in a vendor proposal typically reflects the raw time cost of operating the robot. It does not reflect the cost of collecting data that is actually useful.

The Uninformative Demonstration Problem

Research from the University of Texas at Austin demonstrates that unguided teleoperation, where operators collect demonstrations based on intuition rather than a principled assessment of what the policy needs, routinely produces datasets where a significant portion of trajectories contribute little to downstream performance (Dass et al., arXiv 2026). The problem is not operator incompetence. It is that without a feedback mechanism linking collected data to policy performance gaps, operators have no reliable signal about which demonstrations are redundant and which are critical.

The GUILD framework addresses this by using predictive data attribution to evaluate collected trajectories before committing to further collection, then using that signal to direct subsequent sessions (Dass et al., arXiv 2026). The implication for buyers is that a vendor quoting a data collection timeline without describing a curation or attribution methodology is likely underestimating the iteration cycles required to reach production-grade performance.

Benchmark Rankings Do Not Transfer Across Embodiments

A vendor whose system ranks highly on a published benchmark for one robot arm should not be assumed to perform comparably on a different embodiment. This is not a theoretical concern. It is a measured empirical finding.

The RoboColosseum distributed evaluation study, which ran 5,699 physical trials across 124 sites covering four distinct embodiments, found that policy rankings showed weak agreement across embodiment-specific task suites, with a Kendall's W of 0.24 (Nguyen et al., HuggingFace 2026). Rankings that stabilise for one robot type showed substantial reversals when the same policies were evaluated on a different platform. Open Track rankings also proved to be poor predictors of fine-tuned performance, with rank reversals observed across all three embodiments shared between evaluation tracks.

For buyers, this means that a benchmark result is only meaningful if the evaluation embodiment matches the production hardware. A strong result on a Franka arm tells you relatively little about what will happen on a different manipulator in your facility.

Partial Progress Metrics Reveal What Pass Rates Hide

Standard task success rates are binary: the robot either completed the task or it did not. In production, a system that consistently reaches 80% of task completion before failing has a very different remediation path than one that fails immediately. Binary metrics obscure this distinction entirely.

RoboColosseum introduced partial progress measurement specifically to surface this information. On the G1 Humanoid, no policy completed any of the five evaluated tasks, yet average progress across policies ranged from 13.3% to 23.2% (Nguyen et al., HuggingFace 2026). A buyer relying solely on task completion rates would see a uniform failure signal. The partial progress data reveals meaningful differentiation in where each system breaks down and what the engineering path forward looks like.

When reviewing vendor benchmarks, buyers should ask explicitly whether partial progress or sub-task completion data is available. A vendor that only reports binary success rates is either not measuring at the resolution needed to guide development, or is choosing not to share results that would complicate the sales narrative.

What a Rigorous Evaluation Process Actually Looks Like

Before committing to a platform, the evaluation process should be structured around a small number of high-signal questions rather than a broad feature checklist.

The first is whether the vendor can demonstrate IK precision on your target hardware under your operating conditions, not in simulation. The second is what data curation methodology governs their teleoperation pipeline, and whether there is any feedback loop between data evaluation and collection direction. The third is whether benchmark results were produced on the same embodiment you intend to deploy, under conditions that reflect your environment rather than a controlled lab.

A vendor that can answer all three questions with specifics, including failure cases and the engineering responses to them, is operating at a different level of production readiness than one that leads with aggregate success rates on a generalised benchmark. The research literature is clear that these problems are real, active, and not yet fully solved. The appropriate posture for a buyer is to treat production readiness as something to be verified through structured evaluation, not inferred from demo performance.