LLM Evaluation: Methods, Judge Bias, and Shrinking the Surface
Every guide to LLM evaluation recommends using a model as a judge. Almost none cite the research on how those judges fail. Here is what the papers found, and why the size of your evaluation problem is an architectural choice.

How do you evaluate LLM response quality?
Assign every property of an output to exactly one method. Code-based assertions check anything you can write as a pass/fail predicate, such as schema, required fields, resolvable citations and latency. A model judge scores the semantic properties, like groundedness, relevance and tone. Human labels calibrate the judge and cover high-risk decisions. A judge score with no measured agreement rate against human labels is not a metric, so report the agreement or drop the number.
Evaluation, as used here, means measuring whether a model-backed application produces acceptable outputs and actions across inputs that look like production, with a stated method for each property. The part most teams skip is the scoping question that comes before metrics: how much of the workflow is probabilistic, and did anyone decide that on purpose? On Major, repeatable steps move out of the model into deterministic apps, so they get assertions instead of judges and the evaluation surface stays small.
Model evals, application evals and agent evals are three jobs
A model eval scores a model against a public dataset such as MMLU, HumanEval or GPQA. It says something real about the model and almost nothing about your application, because your prompt, retrieval layer and tools never appeared in any benchmark.
An application eval scores your system on your inputs. An agent eval scores the path the system took, including tool choice, order and recovery after an error.
- Model eval: answers how this model compares on public tasks. Cannot say whether your prompt, retrieval and tools work on your traffic.
- Application eval: answers whether your system produces acceptable outputs on your inputs. Cannot say whether intermediate steps were sound or repeatable.
- Agent eval: answers whether the trajectory was sound and consistent across repeated runs. Still needs output-level scoring to say whether the answer helped anyone.
Benchmarks are a fair starting point for routing between models before you have data. They are not evidence your feature works.
The four evaluation methods and what each is for
Code-based checks are the cheapest and most reliable method, and the one teams skip. Schema validity, every citation URL resolving, p95 latency inside budget, no PII pattern in the output, no illegal state transition on a write. They run in CI with no model call. Teams chase semantic failures and then find malformed JSON and a dead link. If a property can be a predicate, write the predicate and never send it to a judge.
Model judges cover what assertions cannot: groundedness against retrieved evidence, relevance, tone against a rubric, whether a refusal was appropriate. They work under the conditions in the next section.
Human review supplies calibration labels and covers high-risk decisions. Reviewers disagree with each other, so treat their labels as a sampling instrument, not a gate on every release.
Production signals catch the failures your dataset never contained: escalation rate, retry rate, tool error rate, abandonment, and trajectory logs that let you reconstruct a bad run months later.
What the research says about LLM judges
Judges work. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (arXiv 2306.05685), report GPT-4 judges reaching over 80% agreement with human preferences, matching human-to-human agreement on the same comparisons. The same paper names three failure modes, and later work put numbers on them.
- Position bias: Wang et al., "Large Language Models are not Fair Evaluators" (arXiv 2305.17926), found that swapping candidate order moved the verdict. With ChatGPT as evaluator, one ordering gave Vicuna-13B wins over ChatGPT on 66 of 80 questions. Mitigation: score both orderings and treat disagreement as a tie.
- Verbosity bias: Dubois et al., "Length-Controlled AlpacaEval" (arXiv 2404.04475), apply a regression adjustment and report correlation with Chatbot Arena rising from 0.94 to 0.98. Mitigation: control for length.
- Self-preference: Panickssery et al., "LLM Evaluators Recognize and Favor Their Own Generations" (arXiv 2404.13076), found GPT-4 and Llama 2 recognize their own outputs above chance, with a linear relationship between self-recognition and self-preference. Mitigation: judge with a different model family.
These papers date from 2023 and 2024, and the models in them trail what you run today. Read them as failure modes to test on your own judge. The test is cheap: reorder your candidate pairs, measure the flip rate, and you have a position-bias number by the end of the afternoon.
How do you evaluate RAG and agents?
Retrieval and generation fail separately, so measure them separately. Retrieval asks whether the answer was in the retrieved set and at what rank. Generation asks whether the output stayed inside that evidence. A fluent answer built over the wrong documents scores well on every output-level metric and is still wrong.
For agents, final-answer scoring hides mid-trajectory failure and hides inconsistency completely. Yao et al., "tau-bench" (arXiv 2406.12045), scores the end state of the database instead of the chat transcript and introduces pass^k, which asks whether an agent succeeds on the same task across repeated trials. In the retail domain, pass^8 fell below 25%. Consistency is a different property from correctness, and it decides whether a workflow can run unattended.
What a working evaluation program looks like
Take a support-reply drafting agent grounded in a knowledge base. Split the properties before choosing tools.
- In CI, as assertions: schema matches, required fields present, every citation resolves to a document in the retrieved set, no PII pattern, latency in budget, legal ticket status transition.
- On a versioned dataset, with a judge: groundedness, relevance, tone against a versioned rubric.
- Sampled weekly by a person: escalation decisions, plus a labeled slice used only to measure judge-human agreement.
Build the dataset from real traffic and add every production failure as a test case. Block merges on code-based checks, since judges are too slow and noisy to gate a pipeline. Define in advance what a moved score triggers, because a number that produces no decision is not being used.
NIST's AI Risk Management Framework (NIST AI 100-1) files this under its Measure function, and the Generative AI Profile (NIST AI 600-1) extends it to generative systems. Neither prescribes metrics. Both expect a documented method and a retained record that the measurement was taken.
Build an evaluable support agent on Major
Major is the enterprise platform where agents build the software they run on. That changes what you have to evaluate. A step that re-reasons on every run can fail differently tomorrow and needs judges, rubrics and drift monitoring for as long as it exists. A step that runs as code fails the same way each time, and an assertion catches it.
For the support example, the build looks like this:
- Connectors: Zendesk for tickets, Slack for escalations, and your knowledge base source, each with scoped credentials.
- The app that holds state: a drafts app with tables for tickets, retrieved passages, drafts and eval results, in Major's managed database.
- Deterministic steps as app code: schema validation, citation resolution, PII scan and status transitions run as code the agent built once. Your CI assertions test them directly.
- The agent's judgment: drafting the reply and deciding when to escalate. These are the steps that need a judge, and the only ones.
- Workflow trigger: a new-ticket event starts the run, and low-confidence drafts route to a person in Slack for approval.
- Governance: RBAC on who can change the app or rubric, and audit logs recording each action under its credential, which gives you the retained record NIST expects.
Evaluation does not disappear. The judgment calls still need real evaluation, judges included, with every bias caveat above. What changes is how many steps fall into that category, and whether the count grows with every release. Start building free and see which of your steps can move out of the model. Reason once. Run forever.
This piece does not cover red-teaming or safety evaluation, which are separate disciplines with their own methods.
Start building on Major
Put the repeatable steps of your agent workflow into deployed apps with assertions, and spend your judges and rubrics only on the judgment calls that remain.
Related articles
Frequently asked questions
- How do you evaluate LLM response quality?
- Split the output's properties by method. Code-based assertions handle the deterministic ones: schema, required fields, resolvable citation links, latency. A model judge scores semantic properties like groundedness, relevance, and tone. Human labels calibrate the judge and cover high-risk decisions. If a property can be written as a predicate that passes or fails, never send it to a judge.
- What is the difference between LLM evals and benchmarks?
- Benchmarks score a model against public datasets like MMLU or HumanEval, which tells you about the model. Evals score your application against your data, your prompts, your retrieval layer, and your requirements. A model that tops a benchmark can still fail your traffic, because no public dataset contains your customers' questions. Benchmarks inform model selection. Evals decide whether your feature ships.
- Is LLM-as-a-judge reliable?
- Reliable enough to use, not reliable enough to trust unmeasured. Zheng et al. (https://arxiv.org/abs/2306.05685) report GPT-4 judges reaching over 80% agreement with human preferences, matching human-to-human agreement. The same line of work documents position bias (https://arxiv.org/abs/2305.17926), verbosity bias (https://arxiv.org/abs/2404.04475), and self-preference (https://arxiv.org/abs/2404.13076). Randomize candidate order, control for length, avoid self-judging, and report agreement against human labels.
- What are rubrics in LLM evaluation?
- A rubric is the explicit scoring criteria handed to a judge or a human reviewer: what each score means, what counts as grounded, what counts as an appropriate refusal. Judge scores are sensitive to how a rubric is worded, so treat it as a versioned artifact. Changing the rubric changes your numbers, so version it alongside the dataset.
- How does Major reduce the evaluation surface?
- On Major, when an agent works out a repeatable step, it builds an app for that step and runs the app instead of reasoning again. The app is deterministic code with its own database and logs, so you test it with pass/fail assertions. Judges and rubrics are then reserved for the judgment calls that still need a model.