Deploying Visual AI at the Edge: What Lightweight Depth Models Mean for Your Hardware Strategy

Most engineering teams evaluating visual AI for edge deployment start in the wrong place. They benchmark foundation models against accuracy targets, confirm the numbers look good, and then discover that the compute requirements of those models are incompatible with the hardware they actually have to ship on. The emergence of compact, distillation-trained depth estimation models capable of running at real-time rates on power-constrained devices makes this sequencing problem visible in a useful way. It also forces a more disciplined question: what does deployment efficiency actually require, and how should that reshape hardware procurement before a single model is trained?

Why Foundation Models Create a False Baseline

Foundation models for visual perception have made genuine progress on zero-shot generalisation. A model trained across diverse domains and scene types can produce plausible depth estimates on inputs it has never seen, which matters enormously for production systems that encounter distribution shift in the field.

The problem is that these models carry parameter counts that make them impractical outside of server-side inference. Running a 300M-parameter depth model at acceptable latency requires hardware that most embedded and mobile platforms cannot provide, either in compute headroom or power budget. Engineering teams that benchmark against foundation model accuracy without modelling the inference cost of that accuracy are setting a target they cannot hit on their target device.

This creates a systematic gap in how edge AI projects are scoped. The accuracy benchmark becomes the north star, the hardware is treated as a constraint to be solved later, and the compression or distillation work required to close the gap is underestimated at the planning stage.

The Distillation Approach and What It Actually Transfers

Knowledge distillation offers a principled path to closing that gap. The core idea is to train a compact student network to reproduce the output behaviour of a much larger teacher model, rather than learning from ground-truth labels alone. When the teacher has strong zero-shot generalisation, the student can inherit some of that capability at a fraction of the parameter cost.

ZipDepth demonstrates what this looks like in practice. At 6.1M parameters, it achieves real-time inference on a 15W Jetson Orin NX at 34 FPS in FP32, rising to 77 FPS with TensorRT FP16 quantisation, while approaching the zero-shot accuracy of foundation models with 50 times more parameters (Tosi et al., arXiv 2026). The training regime combines a reparameterisable encoder-decoder architecture with large-scale distillation over a multi-domain dataset, which is what allows the model to generalise across scene types rather than collapsing under domain shift.

The mechanism matters here. Distillation over a diverse training set is not the same as distillation from a single-domain teacher. The generalisation capability being transferred is partly a function of the teacher's training distribution, and partly a function of how many domains the student sees during distillation. A compact model trained only on indoor scenes will still fail silently outdoors, regardless of how capable its teacher was.

Zero-Shot Generalisation Has a Ceiling You Need to Measure

Zero-shot generalisation is often presented as a binary property: a model either has it or it does not. In practice, it is a gradient, and the edges of that gradient are where production systems tend to break.

Lightweight models that have been trained in single-domain, self-supervised paradigms exhibit exactly this failure mode. They perform well within their training distribution and degrade unpredictably outside it. The degradation is not always visible in aggregate benchmark scores, which can average over the failure cases rather than exposing them.

For engineering leaders, this means that the evaluation protocol matters as much as the model choice. Testing on held-out domains that are genuinely outside the training distribution, and measuring failure rates rather than average accuracy, gives a more honest picture of what the model will do in deployment. A model that achieves strong mean accuracy but has a long tail of catastrophic failures on out-of-distribution inputs is a production risk, not a deployment candidate.

What Deployment Efficiency Metrics Should Actually Cover

Accuracy scores alone are not sufficient to make a hardware procurement decision. The metrics that matter for edge deployment span several dimensions, and they interact in ways that are not always obvious from datasheet comparisons.

Energy Per Frame

Energy consumption per inference is the binding constraint on battery-powered and thermally limited devices. A model that runs at 30 FPS but consumes 800mJ per frame will drain a mobile platform faster than the application can tolerate. The ZipDepth energy-accuracy trade-off analysis, benchmarked on the Jetson Orin NX at 15W, demonstrates that sub-400mJ per frame operation is achievable at near-foundation-model accuracy levels (Tosi et al., arXiv 2026). That figure should be a reference point when evaluating competing architectures, not an afterthought.

Quantisation Sensitivity

FP16 and INT8 quantisation can double or triple throughput on hardware with dedicated low-precision compute units. However, not all architectures respond to quantisation equally. Models with residual connections and batch normalisation layers that have been reparameterised at inference time tend to quantise more cleanly than architectures with irregular activation distributions. Evaluating quantisation sensitivity before finalising hardware selection avoids the situation where a model that benchmarks well in FP32 degrades unacceptably at INT8 on the target accelerator.

Memory Bandwidth Constraints

Parameter count is a proxy for memory footprint, but it is an imperfect one. Activation memory during inference can dominate on devices with limited SRAM, particularly for encoder-decoder architectures processing high-resolution inputs. A 6M parameter model running at 640x480 may have a smaller peak memory footprint than a 3M parameter model running at 1280x720. Hardware procurement decisions that are based on parameter count without modelling activation memory will produce surprises at integration time.

Rethinking the Build-Versus-Compress Decision

The conventional framing of this decision is: build a custom model from scratch, or take an existing model and compress it. That framing misses a third option that is increasingly viable: distil from a foundation model into a compact architecture designed for the target hardware class from the outset.

This approach front-loads the architectural constraint. Rather than training a capable model and then attempting to compress it into a form factor it was not designed for, the compact architecture is specified first based on the deployment envelope, and the distillation process is used to maximise accuracy within that envelope. The result is a model that is genuinely deployable, rather than a compressed version of something that was designed for a different context.

The hardware procurement implication is that this decision needs to be made earlier than most teams make it. Specifying the target device class, its power budget, its memory constraints, and its available accelerators before the model architecture is chosen changes which architectures are worth evaluating. It also changes which accuracy benchmarks are relevant, since benchmarks run on server GPUs do not predict inference behaviour on edge accelerators with different memory hierarchies and compute characteristics.