Enterprise deepfake detection is quietly accumulating technical debt. Most deployed systems were trained against a fixed catalogue of manipulation techniques, and they perform well against those techniques in controlled evaluations. The problem surfaces when a novel face-swapping or diffusion-editing method enters the wild: accuracy drops, and the detection layer that compliance teams relied on stops providing meaningful signal. Understanding why this happens, and what architectural choices address it, is now a practical concern for engineering leaders in financial services, identity verification, and media.
Why Rule-Based and CNN-Based Detectors Fail on Unseen Manipulations
Traditional deepfake detectors learn to recognise the artefacts produced by specific manipulation pipelines. A model trained on a particular face-swap implementation learns the blending boundary patterns, frequency anomalies, and compression signatures associated with that tool. When a new tool produces different artefacts, the model has no learned representation to match against.
This is not a data volume problem. Adding more training images from the same manipulation types does not help a model generalise to structurally different forgeries. The failure is architectural: the model is learning to detect a fingerprint rather than to assess whether a face is visually plausible in the first place.
The commercial consequence is that vendor benchmarks, which typically measure accuracy on held-out samples from known manipulation methods, are a poor predictor of production performance. A system that scores well on a standard benchmark can still fail badly when a new generative model reaches the threat landscape.
The Part-Level Decision Architecture
The more durable approach shifts the question the model is trying to answer. Rather than asking "does this image match the distribution of forged images in my training set?", a part-level architecture asks "is each facial component visually plausible on its own terms?". This reframes detection as a visual reasoning task rather than a pattern-matching task.
FaceJev, developed by researchers at CASIA and IIE, implements this directly using a Multimodal Large Language Model backbone. The system poses eight independent binary questions about distinct facial regions, reads anomaly probabilities from the model's native answer logits, and classifies an image as forged if any single part crosses a probability threshold of 0.5 (Peng et al., arXiv 2026). The detection decision is therefore a direct logical consequence of the part assessments, not an opaque aggregate score.
This architecture matters for generalization because visual implausibility in a facial region tends to be consistent across manipulation methods. A diffusion-edited eye region and a GAN-swapped eye region may produce different pixel-level artefacts, but both can exhibit structural inconsistencies that a model trained to assess visual plausibility can identify. FaceJev outperformed the strongest baseline by 7.25 percentage points on unseen manipulations (Peng et al., arXiv 2026), which is a meaningful margin when the evaluation is specifically designed to test generalization.
Training Data Requirements for Part-Level Supervision
Shifting to part-level decisions requires part-level supervision, which is a non-trivial data engineering problem. The FaceJev team constructed PartForensics, a dataset of approximately 160,000 images with binary anomaly labels assigned at the level of individual facial regions, specifically to provide the training signal the architecture requires (Peng et al., arXiv 2026).
This has a direct implication for teams evaluating vendor tooling. A vendor claiming MLLM-based detection should be able to describe what supervision signal their model was trained on. A model fine-tuned on image-level labels cannot reliably produce part-level decisions, regardless of the underlying architecture. The label granularity in training determines the granularity of what the model can actually reason about.
The reinforcement learning component in FaceJev, FS-GRPO, optimises each part decision against an exact reward comparison with its alternative answer. This approach ensures that part-level accuracy is a direct training objective rather than an emergent side effect of image-level supervision.
Production Latency Trade-offs in Identity Verification Pipelines
MLLM-based architectures carry a real inference cost. Running eight independent visual reasoning queries through a large multimodal model is substantially more expensive than a single forward pass through a CNN-based classifier. For synchronous identity verification flows where a user is waiting for a decision, this creates a latency budget problem that engineering teams need to model explicitly before committing to an architecture.
Several mitigation strategies are worth evaluating. Shared image encoding across all eight part queries, as FaceJev implements, reduces the per-query cost significantly compared to independent model calls. Asynchronous verification flows, where the deepfake check runs in parallel with other identity checks rather than blocking them, can absorb higher per-check latency without affecting user-facing response time.
The right trade-off depends on the risk profile of the use case. A high-value account opening flow in financial services can justify higher latency for a more accurate decision. A high-volume media content moderation pipeline may require a tiered approach, using a faster first-pass classifier to triage and routing only flagged content to the more expensive MLLM-based check.
What Generalizable Detection Infrastructure Actually Requires
The gap between benchmark performance and production performance in deepfake detection is primarily a generalization gap, not a model capacity gap. Closing it requires architectural choices that prioritise visual reasoning over artefact memorisation, training data that provides supervision at the right level of granularity, and an honest evaluation protocol that tests against manipulation methods not present in training.
Engineering leaders evaluating vendors should ask specifically how the system was evaluated against unseen manipulation types, what label granularity was used in training, and what the inference cost looks like under production load conditions. A vendor that cannot answer those three questions concretely is likely selling benchmark performance rather than production reliability.
The broader lesson from part-level MLLM architectures is that interpretability and generalization are not in tension here. A system that can explain which facial region triggered a forgery decision is also more likely to generalise, because its reasoning is grounded in visual structure rather than statistical correlation with a training distribution.
Where Vector Labs Fits
We build production fraud detection systems that combine image recognition with structured reasoning pipelines, designed to hold up under real-world distribution shift. In our fraud detection build, we deployed parallel NLP and image recognition teams to construct a detection system from the data engineering layer upward, with results surfaced through a Kibana-based visualisation dashboard for operational teams. If you are evaluating or rebuilding deepfake detection infrastructure and want an independent technical assessment of your current architecture, contact us at vector-labs.ai/contacts.
FAQs
Ask the vendor to provide evaluation results specifically on manipulation methods that were excluded from training data. Standard benchmark scores measured on held-out samples from known methods do not test generalisation. If the vendor cannot provide cross-manipulation evaluation results, treat their accuracy claims as upper-bound estimates that will not hold in production as the threat landscape evolves.
The latency cost depends on model size, hardware, and whether image encoding is shared across part queries. Architectures that share a single image encoding pass and read multiple decisions from that encoding, as FaceJev does, are meaningfully faster than systems that run independent model calls per facial region. For synchronous flows, we recommend benchmarking end-to-end latency under your expected concurrent load before committing to an architecture, and modelling whether an asynchronous verification design would remove the latency constraint entirely.
Fine-tuning is possible, but the value of the fine-tuning depends entirely on the label granularity of your training data. Fine-tuning on image-level forgery labels will improve performance on manipulation types represented in your data, but it will not give the model genuine part-level reasoning capability. To benefit from part-level architectures, you need part-level annotations, which requires an annotation pipeline designed around facial region labelling rather than binary image classification.
A tiered approach routes content through a fast, lightweight classifier first and passes flagged items to a more expensive MLLM-based check. The first-pass classifier should be tuned for high recall rather than high precision, meaning it will over-flag but will rarely miss a genuine forgery. The MLLM layer then provides the high-precision decision on the flagged subset. The right split between tiers depends on your expected forgery base rate and the cost of false negatives versus false positives in your specific use case.
Deepfake detection outputs that inform identity verification decisions may be subject to explainability requirements under applicable financial regulation, depending on jurisdiction. Part-level architectures have a practical advantage here because they produce a traceable decision path: a specific facial region triggered the forgery classification. This is more defensible in an audit context than a single confidence score from an opaque classifier. We recommend engaging your compliance and legal teams early to map the explainability requirements before selecting an architecture, not after.

