Most enterprise teams building retrieval-augmented systems have adopted LLM-based relevance judges without seriously auditing what that choice costs them over time. The per-call fee to a frontier model looks manageable at first. It stops looking manageable when you are re-evaluating thousands of documents across dozens of query topics every time your retrieval stack changes. This article examines the structural weaknesses of LLM judges at scale, the case for executable program-based alternatives, and what the trade-offs look like for teams that need evaluation infrastructure they can audit, repeat, and defend to stakeholders.
Companion piece to our broader work on LLM judge cost and quality trade-offs. See LLM-as-Judge Systems: Cost vs Quality Trade-offs for how to calibrate judge models across structured evaluation tasks.
The Real Cost Structure of LLM Judges
The headline cost of an LLM judge is the API call. The actual cost is the API call multiplied by every document in every pool, across every evaluation run, across every system variant you want to compare.
For teams running continuous evaluation against a live retrieval system, this compounds quickly. A retrieval pipeline that ingests new content, handles query drift, or supports A/B testing of ranking models will need re-evaluation frequently. Each of those runs resets the cost clock.
There is also a reproducibility cost that does not appear on any invoice. LLM outputs are non-deterministic by default. Two evaluation runs against the same document pool can produce different relevance labels, which means your system rankings can shift without your retrieval system having changed at all. For teams trying to demonstrate quality improvements to engineering leadership or regulators, that instability is a structural liability.
What Executable Program Judges Actually Do
The ExecJudges approach, proposed by Agichtein et al. (arXiv 2026), inverts the standard LLM judge pattern. Instead of calling a frontier model once per document, you call it once per query topic to generate an executable program. That program then applies lexical matching, standard NLP library operations, and pool statistics to label documents without any further model calls.
The programs are readable. An engineer can open one, understand the logic it applies, and trace why a specific document received the label it did. That inspectability is not a minor convenience; it is the difference between an evaluation pipeline you can audit and one you have to take on faith.
The cost structure is also fundamentally different. Agichtein et al. report a one-time program generation cost of $0.17 to $0.49 per topic, with no incremental cost per document graded thereafter. For a collection with hundreds of topics and thousands of documents per topic, the economics shift substantially in favour of the program-based approach as evaluation volume grows.
Agreement Quality: Where the Trade-offs Land
The natural question is whether cheaper and more inspectable means less accurate. On TREC Deep Learning 2019 through 2021, ExecJudges agrees with trained NIST assessors at roughly the same level as Claude Sonnet 5.5 used as a direct LLM judge (Agichtein et al., arXiv 2026). That is not a marginal result. It means the program-based approach is competitive with one of the strongest available LLM judges on a well-established benchmark.
The ExecJudges-L variant, which incorporates a small number of human labels per topic, extends this further. Agichtein et al. show it exceeds the agreement of per-topic classifiers trained on the same labels across all seven TREC collections tested, and either matches or comes within approximately 0.05 Krippendorff's alpha of LLM judges given the same label budget.
The system-ranking fidelity is also strong. The produced labels rank official TREC runs with Kendall's tau between 0.87 and 0.94, which means the relative ordering of retrieval systems under evaluation is preserved at a level that supports meaningful engineering decisions.
Lexical Matching: Limitation or Feature
A common objection to lexical approaches is that they cannot capture semantic relevance. A document that uses synonyms or paraphrases the query without sharing surface tokens will be penalised unfairly. This is a real limitation and should not be dismissed.
The practical question is whether that limitation matters more than the reproducibility and cost problems introduced by LLM judges. For many enterprise retrieval tasks, particularly those with well-defined vocabularies, structured content, or domain-specific terminology, lexical matching captures a substantial portion of relevance signal. The programs ExecJudges generates can also incorporate parser-based features and pool statistics, which extends their reach beyond simple keyword overlap.
Where semantic gap is genuinely a problem, the ExecJudges-L variant's use of human labels provides a calibration mechanism that keeps the approach grounded without requiring full LLM coverage. The appropriate choice depends on the query distribution and content domain of the specific system under evaluation.
Building Evaluation Infrastructure That Holds Up
The deeper issue for ML engineering leaders is not which judge achieves the highest agreement score on a benchmark. It is whether the evaluation infrastructure supporting their retrieval system can be maintained, explained, and trusted as the system evolves.
An LLM judge pipeline accumulates hidden dependencies: on a specific model version, on a specific API provider's pricing and availability, and on temperature settings that may not be documented consistently across runs. When something changes in that stack, it is difficult to distinguish genuine retrieval quality changes from evaluation drift. Program-based judges do not have this problem. The programs are versioned artifacts. They run deterministically. Their logic can be reviewed by anyone with basic programming literacy.
For teams operating under audit requirements, serving regulated industries, or simply trying to build evaluation they can explain to a VP who asks why a metric moved, that auditability has direct commercial value. Evaluation infrastructure that produces defensible, repeatable results is infrastructure you can build a quality roadmap on top of.
Where Vector Labs Fits
We design and implement evaluation pipelines for production AI systems, including retrieval and ranking infrastructure where cost governance and auditability are first-class requirements. In our LLM judge cost analysis, we examined how cheaper models can match frontier model performance on structured evaluation tasks, providing a practical framework for calibrating judge selection against your actual quality and cost constraints. If you are building or rearchitecting retrieval evaluation infrastructure, contact us at vector-labs.ai/contacts.
FAQs
The crossover depends on your topic count, pool depth, and frontier model pricing. Because ExecJudges incurs a one-time program generation cost per topic rather than a per-document API call, the economics favour it whenever you are evaluating large document pools or running repeated evaluations against the same topic set. For most enterprise retrieval systems with continuous evaluation cycles, that threshold is reached quickly.
This is the primary limitation of the approach. For queries with high synonym density or paraphrase-heavy content, pure lexical matching will miss relevant documents that do not share surface tokens with the query. The ExecJudges-L variant, which incorporates a small number of human labels per topic, partially addresses this by calibrating program logic to observed relevance patterns. For domains with severe semantic gap, a hybrid strategy combining program judges with targeted LLM calls on uncertain cases may be more appropriate.
Yes. The programs generated by ExecJudges are executable artifacts that run independently of any model API. They can be version-controlled, containerised, and called as part of a standard evaluation harness. The main integration work involves the one-time program generation step per query topic and ensuring your document pool format is compatible with the lexical and NLP operations the programs apply.
The standard approach is to collect a small set of human labels per topic and measure agreement between the program outputs and those labels using a metric like Krippendorff's alpha or Cohen's kappa. Agichtein et al. demonstrate that even a small human label budget per topic is sufficient to calibrate the ExecJudges-L variant to competitive agreement levels. Running this calibration exercise before committing to program judges at scale gives you a principled basis for trusting the outputs.
Program judges are topic-specific artifacts, so a significant shift in query distribution or content domain may require regenerating programs for affected topics. The one-time generation cost per topic remains low, and the regenerated programs are immediately inspectable. This is actually an advantage over LLM judges, where domain shift can silently degrade label quality without any visible signal that the evaluation has become unreliable.

