Back to Blogs
How to Benchmark

How to Benchmark

Document AI has a leaderboard problem. The IDP Leaderboard is a good place to see it. A benchmark makes a claim to authority. It says: this is what better means, and here is who is better by it. That claim carries obligations, and they aren't optional ones. You have to characterise your noise before you publish an ordering, document what's in your corpus and where it came from, justify why the numbers you're combining can be combined at all, and choose documents difficult enough to separate go

Arindam Ghosh

Aug 27, 2026
12 min read

Document AI has a leaderboard problem. The IDP Leaderboard is a good place to see it.

A benchmark makes a claim to authority. It says: this is what better means, and here is who is better by it.

That claim carries obligations, and they aren't optional ones. You have to characterise your noise before you publish an ordering, document what's in your corpus and where it came from, justify why the numbers you're combining can be combined at all, and choose documents difficult enough to separate good systems from bad ones. Above all you have to ship the part that was hard, because that's usually the only part that was worth doing.

The Nanonets IDP Leaderboard misses the mark for rigor. We're using it as our example not because it's unusually bad, but because it's public, well documented, and representative of how this category now works. Ten models, sixteen datasets, 9,229 documents, six tasks, one number.

That number is where we should start.

One number, three problems

The overall score is the mean of six task scores. Each task score is the mean of its dataset scores. The announcement presents this as a procedural detail. It's the central design decision, and it breaks three separate ways.

The units don't match. KIE, OCR, VQA and long-document processing are scored with edit-distance accuracy. Classification uses exact match. Table extraction uses GriTS. Exact match is a coin flip outcome. Edit-distance accuracy is a continuous string similarity whose behaviour depends heavily on how long the target field is, which varies by an order of magnitude across these tasks. GriTS measures grid structure similarity. These quantities have different variances, different ceilings, and different meanings at the same numeric value.

Average them and you get something with no interpretation. You cannot say what a one-point gain represents, because it represents a different thing depending on which column it came from.

Dataset size is thrown away. Inside a task, datasets are averaged flat. A fifty-document set counts exactly as much as a five-thousand-document set. So the task score is dominated by whichever dataset is smallest and noisiest.

Task count becomes the weighting scheme. Because the top-level average is also flat, the weight on any underlying capability is set by how many tasks happen to touch it. Add one more OCR-flavoured task and text recognition silently doubles in importance. Nobody argues for it. Nobody has to notice. The definition of "best IDP model" changes without a single new measurement being taken.

The six tasks aren't six things

They overlap by construction. OCR is a prerequisite for key information extraction, for table extraction, for long-document processing, and for most document VQA. A model that reads characters badly fails all of them, and it fails them together.

So the aggregate isn't measuring six capabilities. It's measuring one or two repeatedly, and reporting the redundancy as coverage. A model that's strong at recognition and weak at structure gets its strength counted four times and its weakness counted once, in the table column. For production document work that's backwards. Recognition is largely solved. Structure is where the expensive failures live.

Here's the version of this that stuck with us.

You can measure a car's top speed on a straight. Then how it handles broken roads. Then how it does off-road. All legitimate measurements. But nobody averages them into a single "car score," because they answer different questions for different buyers, and the mean answers none of them.

Multi-attribute evaluation is a solved problem in every mature testing domain. The solution is to publish the vector. Document AI hasn't adopted it.

A 0.05% difference is not a finding

Gemini-2.5-Flash is reported as underperforming Gemini-2.0-Flash by 1.84% on OCR and 0.05% on classification. The second figure gets written up as a performance degradation.

There's no basis for treating that as a result. The document reports no seeds, no repeated runs, no confidence intervals, no decoding temperature, and no variance estimate of any kind. At these document counts a 0.05% difference amounts to a fraction of a single document, and it would move by more than that across repeated sampling, an API-side update inside the same version string, or a change in the order examples were fed.

This isn't one overstated comparison. It tells you variance was never characterised at all, which puts every ordering on the leaderboard in question, including the headline one.

The same gap undermines the more interesting claim, that the older gpt-4o-2024-08-06 beats the newer gpt-4o-2024-11-20. If that's true it's worth understanding, and the prior is against it, which is exactly why it needs error bars. What it rests on is single-pass results under a single prompt per task. That can't separate a real regression from decoding noise, or from the older checkpoint simply fitting that one prompt phrasing better.

Which is the deeper issue. One prompt per task, across ten heterogeneous models from four vendors, measures prompt fit at least as much as capability. Prompt sensitivity in VLM extraction is large and well known. Without a paraphrase sweep, neither the leaderboard nor anyone reading it can tell the two apart.

Scoring a refusal as a failure

o4-mini declined to transcribe a 32-row by 11-column table, noting that manual cell-by-cell extraction to JSON would be lengthy and error-prone, and suggested scripting it instead. The announcement reports this as a model shortcoming.

The task as posed is the shortcoming, and the inversion matters more than it looks.

In a real pipeline, the alternative to a model refusing work it will do badly is a model doing it badly and returning something well-formed. A fabricated table is worse than a refused one, because a refusal routes to human review and a fabrication doesn't. A benchmark that scores abstention identically to fabrication isn't neutral between those outcomes. It selects for the second. The models that climb a leaderboard built this way are the ones most willing to guess under load.

There's a standard fix, borrowed from selective prediction: let the model abstain, then report accuracy as a function of coverage. A model that declines 10% of documents and is 99% accurate on the rest is a better production component than one that's 95% accurate on everything. No current IDP leaderboard can express that difference.

The documents are easy exactly where it counts

Key information extraction carries nearly all the commercial weight in document AI, and here it runs on invoices, receipts and forms. Stable layouts, printed text, small well-defined field sets. Extracting an invoice number, a date and a total produces high scores and compressed model-to-model variance. The ranking on the task people actually buy is the least informative part of the leaderboard.

The headroom is somewhere else. Long sparse unstructured tables top out at 47%. Long-document understanding peaks at 69.08% on documents up to 21 pages. Those are real numbers and the announcement deserves credit for publishing them rather than burying them. But they're headroom on tasks most buyers aren't procuring against, and they're reported with no error analysis: no breakdown of whether table failures are recognition, structure recovery, or serialisation.

What's missing is the intersection: hard documents on the task that matters.

A prescription note where the medicine name is the field you need and the handwriting is the obstacle. A lab report photographed at an angle, mixing Devanagari and English, with a stamp sitting on top of the one value you're extracting. A claims form with a regional layout nobody standardised. A field that appears once, on page fourteen, in a table with no gridlines.

That's where extraction fails in production, and where model-to-model variance would be large and diagnostic. Building those cases needs annotation work that public datasets don't supply. Which is exactly why they're absent, and exactly why building them would have been the contribution.

A related gap: the corpus mixes public, synthetic and in-house data and never discloses the split. No per-dataset counts either, which matters directly because the averaging is unweighted. No synthetic generation pipeline named. No annotation protocol, no inter-annotator agreement. For extraction ground truth that's not a formality: field boundaries, date and amount normalisation, and how absent fields are handled are all judgement calls that move scores.

The one that mattered was deferred

The motivation section contains the best paragraph in the announcement, and we agree with it completely:

There is no leaderboard evaluating confidence prediction for language or vision-language models in any domain. Without it, a model with 98% accuracy still requires manual review of 100% of its outputs — because nothing tells you which 2% to look at.

That's correct. It's the central obstacle to actually automating document workflows. We arrived at the same conclusion independently, from the deployment side rather than the benchmarking side.

Confidence is listed as the seventh task. Results were promised shortly. They aren't there.

We don't read that as cynical. It's what happens when the schedule belongs to a launch: the six tasks with existing metrics and existing datasets ship, and the one that needs a metric designed from scratch slips. But the effect is the same. The document's own argument for its existence rests on the deliverable that doesn't exist, and everything that did ship has prior benchmarks and known metrics.

It's worth saying why that task is hard, since the announcement doesn't.

With a generative model, the only uncertainty signal you get out of the box is logprobs. That's token confidence, and because it requires no probe, no training and no access to internals, it's what every production pipeline reaches for first. In our own testing it correlates weakly with correctness and is close to useless for flagging wrong output on an OCR task.

The reason is that what counts as correct differs by task, and token probability is blind to the difference. OCR is judged on exactness: the string on the page is the answer, and one wrong character is a wrong extraction. Document VQA is judged on answer correctness, where several surface forms are equally acceptable and exactness was never the criterion. Token probability is a fluency measure under the model's own prior, which makes it a passable proxy for the second and a bad one for the first.

The failure it produces is specific and it's the one that matters. A character on the form is genuinely ambiguous, and the decoder resolves it toward the likelier word rather than the one that's printed. The model is confident, the output is wrong, and it's wrong in precisely the way a workflow needs flagged. Raw logprob will not tell you, because from the model's point of view nothing unusual happened.

There's a granularity problem stacked on top of that. Workflows make routing decisions per extracted field. Models expose probabilities per token. The mapping between the two is unsolved, and it doesn't get solved by averaging.

So a confidence benchmark can't report one number per model. It has to specify which signal, at what granularity, evaluated for which task, and it has to separate whether the signal ranks errors correctly from whether a stated 0.9 actually means right nine times in ten, because those are different properties with different fixes and a model can have one without the other.

What we think the measurement looks like

Isolate confidence scoring from extraction accuracy and evaluate it as its own object.

Build a curated corpus where ground truth is known at field level and difficulty is deliberately varied, including the ambiguous and degraded cases that produce genuine uncertainty, not the clean ones that don't. Run extraction. Then score the confidence mechanism rather than the extractor: how well does reported confidence correlate with actual missed and wrong predictions, per field, per document class?

The headline number becomes the accuracy of the confidence model itself.

An extractor at 92% with a confidence signal that reliably finds its own 8% is a better production system than one at 96% that can't. No current benchmark can say that.

This is the problem we've been working on internally, and we've made meaningful progress on uncertainty quantification for extraction: enough to be confident the out-of-the-box signal isn't the ceiling, and enough to know why building a benchmark around it is harder than building one around accuracy.

So: how to benchmark

First, credit where it's due. Nanonets built and released something, which is more than most of the people who will criticise it have done, and the difficulty of the thing they attempted is real. A benchmark that covers all of intelligent document processing has to span OCR, extraction, question answering, tables and long documents, across every industry that touches paper, with ground truth for each. Nobody has managed it. We suspect nobody will, and that the ambition itself is where the design went wrong.

What we're building instead is per-domain, because that's what an industry buyer can actually use. Take healthcare. The first job isn't running models at all, it's understanding the document types, because a discharge summary, a prescription, a lab report and a claims form fail in unrelated ways and pooling them hides all four. The second job is inside the document: getting a medicine name wrong and getting a hospital name wrong are not the same error for the organisation that has to live with the output, and a rubric that scores them identically has already thrown away the information the buyer needed. The corners and edges are where the domain gets interesting, and a general benchmark has to iron them out to make its numbers comparable. That flattening is the cost, and for anyone actually deploying, it's the whole cost.

The rest of what follows isn't technically novel, and that's the point. These methods exist and are standard in adjacent fields. Their absence reflects effort and incentive, not the state of the art.

Separate by domain first. There is no IDP score because there is no IDP buyer. Nobody procures "the best IDP model": there's no such object. There's a company with one document class, one field set, a volume, an error budget, and a downstream process that breaks in specific ways. Finance, healthcare and logistics have different distributions and different failure costs. Rank within a domain, not across them.

Start with the documents, not the models. Before anything gets evaluated, the domain has to be mapped. In healthcare that means knowing that a discharge summary, a prescription, a lab report and a claims form fail in unrelated ways, and building the suite around that structure. Pool them and every failure mode averages out of view.

Weight fields by consequence. Uniform weighting asserts every error costs the same, and that's false in every domain we've worked in. A wrong medication name and a wrong hospital name are not the same event. An emergency contact number and a dosage count are both numbers, both score identically under edit distance, and are nowhere near equivalent when wrong. The rubric has to be sensitive to that, and building one that is means keeping the corners and edges that make the domain hard rather than ironing them out to make the numbers comparable.

Choose documents for difficulty. Handwritten clinical notes. Ambiguous and non-standard layouts. Multi-page edge cases where the field appears once, deep in the document. Phone captures with skew and glare. A good evaluation corpus produces large, diagnostic variance between models. Easy documents produce neither.

Run it more than once. Multiple seeds and multiple prompt paraphrases, with confidence intervals on every reported difference and no directional claims below the noise floor. This is also the only way to separate a genuine model difference from prompt sensitivity, which is the more likely explanation whenever an older checkpoint appears to beat a newer one.

Publish the vector, not the average. Per-task and per-dataset scores as the primary artifact. If an aggregate gets published at all, weight it deliberately, say how, and never average across metrics that don't share units.

Say what's in the corpus. Per-dataset document counts, the real versus synthetic split, and how the synthetic documents were generated. Without the counts, an unweighted average can't be interpreted. Without the split, nobody can tell how much of the score reflects real documents.

Make it easy to bring your own model. If an open-source model can't be evaluated without reverse-engineering the harness, the leaderboard is a publication rather than infrastructure. Extensibility is what separates a benchmark the field can build on from one that only its author can run.

Make confidence a first-class metric. Not a seventh task, and not an afterthought. Build a curated corpus where errors are known at field level, then score the confidence signal against them: how well does it correlate with the predictions that were actually missed or wrong? The number you want is the accuracy of the confidence mechanism itself. And once you have it, abstention stops being a failure, a model that declines the documents it would get wrong is doing the job, and the benchmark should be able to say so.

Closing

We've been harsh, so let's be precise about the target. The Nanonets team built something and released it, which is more than most of the people who will criticise it have done. Code, datasets and a live leaderboard are all public, and they published numbers that don't flatter the models. The failures above are ordinary ones, the kind that come from shipping against a launch date rather than against a question.

But a benchmark is a claim about what better means, and the field will optimise toward whatever the leaderboard rewards. Right now what it rewards is a scalar that averages incommensurable things, on documents chosen for how cleanly they can be scored, from single runs, with no signal for which outputs to trust. Optimise against that and you get models that are excellent at invoices and confidently wrong on a prescription.

The gap the announcement itself named (no way to know which 2% to check) is the one that decides whether any of this can be deployed without a human reading every page. That's the problem we're working on, and it's the reason we think the next useful benchmark in this space will be narrow, domain-specific, and much less impressive-sounding than one number across ten models.

Sources

Primary

Benchmarks and metrics referenced

  • GriTS (Grid Table Similarity), the table extraction metric used — Smock et al., Microsoft Research
  • OmniDocBench — OpenDataLab
  • olmOCR-Bench — Allen Institute for AI
  • DocILE, DocVQA, ChartQA — the public datasets inside IDP Core
View All Blog Posts