What Is LLM Evaluation?
LLM evaluation is the practice of measuring whether a language model performs a specific task correctly, using a defined set of test cases and a scoring method rather than general impressions. For teams running models in production, it answers a narrower question than public benchmarks do: not whether a model is capable in general, but whether it still does your task as well as it did last month.
The distinction matters because every frontier release arrives with a scorecard — reasoning benchmarks, coding benchmarks, a chart showing the new version ahead of the old. None of it tells you whether the model got better at reading a scanned rent roll or abstracting a forty-page lease.
This is part of a series of articles about AI quality
Why Don't Public Benchmarks Answer the Question?
Public benchmarks measure general capability on standardized tasks. If your workload is document extraction, a general reasoning benchmark is a proxy at best. Models that improve on reasoning benchmarks sometimes regress on extraction, and nothing in the release notes distinguishes the two cases.
There is also a selection effect. Benchmarks that become widely cited become targets, and performance on a targeted benchmark drifts away from performance on everything else. A score that rises because a model was optimized for the test tells you little about the task you care about.
That gap is why teams running LLMs in production end up building internal evaluation. Not because public benchmarks are wrong, but because they answer a different question.
Are You Measuring Quality or Measuring Change?
Two similar-sounding questions pull an evaluation suite in opposite directions, and choosing deliberately avoids confusion later.
"How good are we?" wants a representative sample of real work, cases weighted by how often they occur, and a score that approximates the error rate a user would experience.
"Are we getting better?" wants close to the reverse: the hardest cases available, weighted by how much they discriminate between systems, and an absolute number nobody should read too literally.
A suite built for the second question reports a lower score than the first, by design. Trivial cases get retired once everything passes them, and what remains is the difficult residue. Reporting that number as a quality claim would mislead. Using it to detect regression is exactly right.
Preference Ranking vs. Scoring Against Ground Truth
The best-known public evaluations measure preference. Two anonymized responses are shown side by side, a human or model picks the better one, and votes aggregate into an Elo-style ranking.
Preference is the right instrument when there is no correct answer — summarization style, tone, helpfulness. There is no key to grade against, and aggregated judgment is the best available signal.
Document extraction is not that. A tenant's name is transcribed exactly as printed or it is not. An escalation date is the one on the page. Nobody's preference is relevant, and a preference vote cannot tell you a model read a value out of the wrong row — only which output someone liked better.
For tasks with a correct answer, score fields against ground truth. The cost is that ground truth has to be built, which is the expensive part of any serious evaluation.
How Do You Build Ground Truth That Means Something?
If a model generates the ground truth and it is accepted unreviewed, the evaluation measures agreement between two models rather than correctness. That is a real failure mode and an easy one to fall into, because generating keys is fast and reviewing them is not.
For a forty-page lease with hundreds of values, signatures, and checkboxes, a usable key is built by hand and verified by a second reader. Attempts to shortcut this tend to encode the same errors the system under test would make.
Two consequences follow. Suites grow slowly, so they should be designed for longevity rather than rebuilt each quarter. And the genuinely ambiguous cases always require human adjudication, because they are ambiguous precisely where a model would guess.
Why Score Three Outcomes Instead of Two?
Binary pass/fail discards the information that matters most. A table with 47 of 50 rows correct, a date in the wrong format, a value that is right but typed as a string instead of a boolean — these are not failures in the same sense as a fabricated number, and collapsing them into "incorrect" hides where behaviour is changing.
A three-way outcome — correct, partial, incorrect — surfaces drift earlier. Changes in model behaviour tend to appear in the partial column before the failure column, making the partial count a leading indicator rather than a footnote.
Reporting counts alongside the aggregate also matters. "500 / 66 / 69" says considerably more than "86.35%".
When Should a Model Act as the Judge?
Some comparisons need a model. "Acme Holdings LLC" and "ACME Holdings, L.L.C." are the same company, and a string comparison will insist otherwise. Semantic equivalence, paraphrase, and formatting variation are genuine judgment calls.
Most comparisons do not. Comparing two numbers to four decimal places does not require a language model, and routing it through one introduces a failure mode where none existed.
The risk is that model-based judging fails quietly. A grader that degrades looks exactly like a system that degraded — the score drops, and the obvious conclusion is the wrong one. Scoping the judge to comparisons that genuinely need judgment limits the damage and makes the failure easier to isolate.
Do Pinned Model Versions Behave Consistently?
Providers tag stable models with version identifiers, and it is reasonable to assume a pinned version behaves consistently. In practice that assumption does not always hold: the same version string, same configuration, and temperature at zero where available can still produce measurably different results week to week.
This has a direct design consequence. Testing a model once, at adoption, is not sufficient. Re-running the same suite against the same pinned version on a schedule is what turns silent drift into something visible.
It also reframes what the evaluation is for. It is not only a selection tool used when choosing a model. It is a monitoring tool for the model already in production.
Capability Suites and Workflow Suites
Two groupings answer different questions, and running only one leaves a gap.
Capability suites isolate behaviours independent of business context: numeric operations, table handling at scale, long-context retrieval, tool use, format parsing. They work like unit tests — when one regresses, it points at a specific mechanism.
Workflow suites run a real end-to-end process from a real domain. They work like integration tests — they catch the case where every individual capability improved but the composed workflow got worse.
That case is not hypothetical. A model update can improve table extraction and arithmetic while degrading a workflow depending on both, because the interaction changed. Only the workflow suite sees it.
How AI Agents Can Automate the Evaluation Process
Evaluation is itself a document-heavy workflow, and several of its most laborious steps can be automated.
1. Generate Candidate Ground Truth for Human Review
An agent can produce a first-pass extraction from a source document and present it field by field for a reviewer to confirm or correct. This does not replace human construction of the key — it changes the task from transcription to verification.
The distinction is important: a reviewer confirming or correcting a proposed value is doing judgment work, whereas a reviewer typing hundreds of values is doing data entry that introduces its own errors.
Key actions:
Produce candidate field values with a citation to the source page.
Present each value for explicit human confirmation or correction.
Record which values were corrected, as a signal about case difficulty.
2. Route Comparisons to the Right Check
An agent can classify each field comparison by the kind of check it requires — exact match, numeric tolerance, date normalization, or semantic equivalence — and route accordingly, invoking a model-based judge only where judgment is genuinely needed.
Routing limits the blast radius of a degraded judge and makes grader failures easier to isolate from system failures.
Key actions:
Classify each comparison by the type of check required.
Apply deterministic checks wherever the comparison allows one.
Reserve model-based judging for semantic equivalence cases.
3. Classify Failures by Cause
An agent can group failing cases by apparent cause — wrong row, missing page, format mismatch, arithmetic error — turning a list of failures into a short list of hypotheses about what changed.
This is the step that converts a score movement into an investigation with a starting point rather than a general alarm.
Key actions:
Group failures by failure mode rather than by test case.
Surface the modes that changed most since the last run.
Link each group to representative cases and source documents.
4. Monitor Pinned Versions on a Schedule
An agent can re-run a frozen suite against a pinned model version on a cadence and report only the deltas, making drift visible without a person remembering to check.
Because provider-side changes arrive without notice, a scheduled re-run is the only mechanism that reliably catches them before a user does.
Key actions:
Re-run a frozen suite against pinned versions on a schedule.
Report deltas rather than absolute scores.
Alert on movement outside the suite's established noise band.
Evaluating AI Output Quality with Kolena
Kolena is an AI platform for document workflows that applies the same evaluation discipline to customer-defined rubrics. Kolena agents read generated output alongside its source documents, score each field or section against the criteria you define, classify failures by cause, and produce the review record. Every judgment carries a page citation and a reasoning log, so quality, operations, and engineering teams can trace a score to the evidence behind it rather than accepting a number on trust.
Key capabilities of Kolena:
Rubric-based scoring: Scores output against criteria you define rather than a fixed benchmark, so the measurement matches the task you actually run.
Three-way outcome reporting: Reports correct, partial, and incorrect counts alongside the aggregate, surfacing drift in the partial column before it reaches failures.
Cited judgments: Cites the exact page or clause behind every score, producing a reasoning log a reviewer can follow and dispute.
Deterministic and semantic checks: Routes each comparison to the appropriate check, reserving model-based judgment for cases that genuinely require it.
Failure-mode classification: Groups failing cases by cause rather than by test, turning a score movement into a short list of hypotheses.
Scheduled re-evaluation: Re-runs a frozen suite against pinned model versions on a cadence and reports deltas, making silent drift visible.
High-volume parallel processing: Processes hundreds of cases at a time and routes exceptions to reviewers rather than requiring full manual review.
Enterprise security and governance: PCI, SOC 2, and HIPAA compliant, with no training on customer data, encryption in transit and at rest, role-based access control, and a complete audit trail.
Kolena's LLM Output-Quality Evaluation agent is pre-built for this workflow. The agent library page shows the fields it scores, the inputs it takes, and how it is configured.
Learn more about reliable and accurate AI at Kolena
