Benchmarking LLMs on large corporate documents

Note: This was written by a human. All em-dashes were intentionally inserted.

New foundational model versions ship every month, and each one comes with a blog post claiming to be the new SOTA. Nobody publishes a good answer to whether Gemini 3.8 or Claude 5 got better at reading a scanned rent roll or a thousand-page package of loan files and spreadsheets.

Extraction at Kolena is a pipeline rather than a model call, and most of it runs before any extraction happens. Documents reach us as native PDFs, scans, photographs, TIFFs, spreadsheets, Word files and emails with attachments nested inside archives. Making any of that readable is its own engineering problem even before the involvement of LLMs.

Data preprocessing is a core part of the pipeline to make raw documents consumed by LLMs more effectively. Scanned lease agreements, for example, arrive rotated and skewed – we straighten them with classical computer vision. Many of the PDFs we process have no embedded text layer, and so how well we parse those scans caps everything downstream: no amount of prompting recovers a number the OCR missed. We decide how a million-cell workbook is divided and rendered, because until something decides, it is not a friendly document to handle.

Then extraction starts. We filter documents down to the ones relevant for the task at hand, crop and re-examine regions where one pass leaves a detail unresolved, handle arithmetic over large tables with generated code, and anchor every result with a page or positional citation. Each of those stages has its own model or methodology, chosen after significant validation.

We hold every part of the pipeline to the same standard: the best solution available for that specific job. Upholding that standard is harder than it sounds, which is why we’ve made our own benchmark. There are two main challenges to maintaining that standard:

1. Model selection. Before putting a model behind any of the document processing stages we need to know how it handles our documents, not how it handles generic conversations. In the past nine months we changed or improved our main extraction model six times while the model doing document filtering did not change once, because nothing we benchmarked was better at that specific job.
2. Model stability. LLM providers tag their stable models with a version string, but that doesn’t guarantee stable behaviour, even when temperature is set to zero (when possible). We know providers push fixes into production models without bumping the version or transparent communication. I wrote about model stability in 2023, outlining how OpenAI's latest GPT model at the time was worse than its original March snapshot on some tasks and better on others. Re-testing the same model version regularly is part of our routine. For example, the same gemini-3.5-flash model may perform differently from week to week, even under the exact same configuration.

Kolena has always been an AI quality company. We built model testing tooling, published research on LLM evaluation, and hosted AIQCon in 2024, a conference on AI quality standards. These principles are the foundation of our AI automation product.

Benchmarking is not just for performance

The benchmark answers one question: are we getting better? It is not built to answer a different question that sounds almost identical: how good are we?

Those pull in opposite directions on nearly every design decision. A suite meant to report how good you are wants a representative sample of real work, cases weighted by frequency, and a score that approximates a production error rate. A suite meant to detect improvement wants close to the reverse: the hardest documents you have, weighted by how much they teach you, and an absolute value nobody should read too closely.

As part of this effort, we intentionally make our benchmarks harder over time, as the trivial, solved-by-general-use-case scenarios are rather unimportant for the benchmark for actually tracking if we're getting better. Since its inception, roughly half of our test cases have been retired to an easy test suite (gemini-2.5-pro consistently scores 100% on these tests). The easy test suite is reserved only for new model releases, to identify if there are any critical regressions from a new frontier model.

Not a benchmark based on preference

Most public model benchmarks that people have heard of measure preference. Rank-based benchmarks, such as LMArena, put two anonymous responses side by side and ask a human/model which is better. The system aggregates the votes into an Elo-style ranking.

That is a good instrument for what it measures, but preference is what you fall back on when there is no right answer. A rent roll has a correct answer. The tenant's name is either transcribed exactly as shown or it is not, and the escalation date is the one printed on the page. Nobody's preference is relevant, and a vote does not tell us if a model has read the value out of the wrong row. So we score individual extracted fields against a ground truth, on real documents, in the formats that customers actually send.

Our test suites

We group our test suites in two ways.

Capability suites isolate behaviours the system must exhibit, outside of specific business contexts. The platform needs to handle numeric operations, regardless of whether they’re used for bookkeeping, or predictive analytics.

Domain suites run a real workflow from a real vertical, end to end.

We maintain both capability and domain suites because a model can improve on every capability suite but regress on particular domain-specific workflows. A model update could improve individual operations, but worsen the overall business flow. The capability suites allow us to test functionality at a “unit” level, while the domain suites are more end-to-end.

On customer data

We do not directly use customer data in our benchmarks. We do, however, use layouts and representative files when designing our hard test cases. What we keep is the structural difficulty that made the document worth keeping — the layout, the density, the ambiguity, the characteristics that challenge the system. A benchmark does not need anyone's name or address, but it needs the shape of a hard loan package.

That costs us something. Anonymized documents are occasionally easier for the system compared to the originals, because whatever made a document difficult was its messy format or embedded content.

How scoring works

Every case resolves to correct, partial or incorrect, and we report the overall counts of that triple alongside its weighted score, since (500/66/69) says more than 86.35% does. The partial category exists because this is where we see changes in model behaviour: a table with 47 of 50 rows, a date in the wrong format, a value that is right but typed as text rather than a boolean.

Nothing enters the benchmark un-reviewed. A person manually reads the documents and constructs or reviews ground truths, field by field. If an LLM generates a ground truth and naively gets merged, this would measure agreement rather than correctness. The cases that matter most will always require some human adjustment or input.

A model judges only when a model is needed. "Acme Holdings LLC” and “ACME Holdings, L.L.C.” are the same company, but a raw string comparison would disagree. In the arithmetic test suite, comparing numbers to four decimal places doesn’t require AI to verify.We route each comparison to the right kind of check, because a model-based judge once broke quietly and took reported performance down with it. We’ll revisit this in Part 2.

The challenges behind the benchmark

Every ground truth has to be made by hand. For a forty-page lease with hundreds of values, signatures, and checkboxes, a ground truth must be made manually and verified by a reviewer. Suites have to be large enough so real regressions clear the noise from ambiguous test cases. There is no benchmark that scales in terms of breadth and depth the way we would like.

The documents have to be real, which means they aren’t LLM-friendly. Documents may be rotated, skewed, watermarked, or simply be a huge workbook that is difficult to hand to an LLM.

The evaluation has to be trustworthy. A test case’s result should not change if the system is unchanged, but should highlight regressions if a system becomes worse. A perfect extraction pipeline and a degraded grader produces noise in the results. Scoping an AI judge to only the comparisons that genuinely need AI is the only defence we have found.

No part of the benchmark is sexy or glamorous. It is a large amount of tedious work that has to be done correctly, to produce one overall number a week to tell us whether to investigate certain capabilities or workflows in more detail.

Stay tuned for Part 2 where we share how a year of running the same benchmark has taught us, misled us, and changed how we apply our LLMs in production.