--- title: "AI Engineering Blog" url: "/engineering-blog/" updated: 2026-09-10T22:19:21.577997+00:00 --- # AI Engineering Blog Kolena Engineering # Building extraction on models we don't control. Notes from the team running document extraction in production — benchmarks, non-determinism, migration, cost. [Read the latest post](#)[Subscribe](#) All postsBenchmarkingReliabilityModel migrationAccuracyCost [ LatestBenchmarking12 min read ## How we benchmark document extraction Public benchmarks measure preference. A rent roll has a right answer. Why we score individual extracted fields against ground truth, why we stopped running 304 test cases we always passed, and what fifteen months of weekly runs changed about how we read the score. MC Mark Chen August 26, 2026 ](#) Recent [ 12 min read How we benchmark document extraction ](#)[ 9 min read Improving extraction accuracy outside the model ](#)[ 8 min read Non-determinism in production extraction ](#)[ 11 min read Debugging systems we don't control ](#)[ 10 min read Perpetual model migration ](#) Improving extraction accuracy outside the model Prompt decomposition, document filtering, deskewing, generated code for arithmetic. Most of last year's accuracy came from the system around the model. DraftingNotify me → Non-determinism in production extraction Same document, same prompt, same model version, different answer. What varies, how much, and what we made deterministic on purpose. OutlinedNotify me → Debugging systems we don't control A provider ships a silent update under the same version string and your extraction changes. Detecting it, isolating it, deciding whether to act. OutlinedNotify me → Perpetual model migration Six model generations on the main extraction path in nine months, three migrations in flight at once. Migration as a standing process, not a project. OutlinedNotify me → Measuring the cost impact of context caching Cache hit rates, prompt ordering, and what caching actually saved per extraction once we measured end to end instead of per token. PlannedNotify me → #### Working on something adjacent? We're hiring engineers who like measurement problems more than model problems. See open roles → The series ## Six posts on running extraction in production. One email per post. Nothing else — no product announcements, no webinar invites.