---
title: "Benchmarking LLMs on large corporate documents - Kolena"
url: "/engineering-blog/benchmarking-llms-on-large-corporate-documents/"
description: "How Kolena benchmarks LLMs on real corporate documents — 800+ test cases, 40+ suites, run weekly — and why the goal is detecting improvement, not scoring accuracy."
author: "Mark Chen"
updated: 2026-09-11T00:28:22.756784+00:00
---

# Benchmarking LLMs on large corporate documents

How Kolena benchmarks LLMs on real corporate documents — 800+ test cases, 40+ suites, run weekly — and why the goal is detecting improvement, not scoring accuracy.

_Note: This was written by a human. All em-dashes were intentionally inserted._

New foundational model versions ship every month, and each one comes with a blog post claiming to be the new SOTA. Nobody publishes a good answer to whether Gemini 3.8 or Claude 5 got better at reading a scanned rent roll or a thousand-page package of loan files and spreadsheets.  

|     |     |
| --- | --- |
|     | Fun Facts (YTD) |
| Frequency | Weekly (at a minimum) |
| Regular Test Case Count | 400+ |
| Total Test Case Count | 800+ |
| Test Suites | 40+ |
| Cost Per Run | $15 to $450 (depending on model) |
| Time Per Run | 1 to 4 hours (depending on model) |
| Distinct Models Benchmarked | 45  |
| Distinct Benchmark Runs | 264 |

|     |     |
| --- | --- |
| **Group** | **Suite Examples** |
| Perception | `vision, image_variations, checkboxes` |
| Numeric and logical | `arithmetic, counting, sorting, math_scripts` |
| Tables at scale | `big_table, copy_table, code_gen_large_tables` |
| Context and retrieval | `needle_in_haystack, long_prompt, many_fields` |
| Tooling | `code_gen, web_search, document_filter, inline_citations` |
| Formats | `spreadsheets, visual_xlsx, email` |