The top document parsing models score 96 out of 100. No benchmark checks whether they read the invoice total correctly.
Turning a scanned page into text got much better in the last year, and there is now a leaderboard for most of it. The top scores sit above 95 and within a point or two of each other, which reads like a solved problem.
We went looking for a model that could read a scanned purchase order in Thai. The leaderboards could not answer that. Working out why turned out to be more useful than the rankings.
Where the leaderboard stands today#
OmniDocBench 1 v1.6, the version current as of August 2026:
| Model | Params | Overall |
|---|---|---|
| PaddleOCR-VL-1.6 5 | 0.9B | 96.34 |
| MinerU2.5-Pro 6 | 1.2B | 95.75 |
| GLM-OCR | 0.9B | 95.22 |
| PaddleOCR-VL-1.5 | 0.9B | 94.93 |
| Qianfan-OCR | 4B | 93.90 |
| Youtu-Parsing | 2.5B | 93.74 |
| MinerU-Pipeline | 86.47 | |
| Marker 7 | 78.44 |
Read the parameter column before the score column. The top three are 0.9B, 1.2B and 0.9B. The 4B and 2.5B models rank below all of them. The two pipeline tools at the bottom are the non-VLM approach, and the gap to them is the largest one on the board.
Everything above that gap is within about a point of everything else, which is where the rest of this post starts.
Four benchmarks carry most of the field#
OmniDocBench 1 (OpenDataLab, CVPR 2025) is the one most people quote. It is 981 PDF pages across nine document types: books, slides, financial reports, textbooks, exam papers, magazines, academic papers, handwritten notes, newspapers.
Text is scored by normalized edit distance against a reference transcript. Tables are converted to HTML and scored with tree edit distance similarity. A later version grows the set to around 1,355 pages.
olmOCR-Bench 2 (AllenAI) works differently. Instead of diffing a page against a reference, it expresses over 7,000 test cases across 1,400 documents: this header is present, this cell holds this value, this is the reading order. Each assertion passes or fails.
IDP Core 3 (Nanonets) scores key information extraction, tables and document question answering over around 2,000 production documents. It asks whether you got the invoice number, which is closer to what a business needs. It is also run by a company that sells document AI, and its own model leads the table.
ThaiOCRBench 4 (SCB 10X, IJCNLP-AACL 2025) is 2,808 human-annotated samples across 13 tasks. As far as we can tell it is the only serious Thai document benchmark that exists. It exists because a Thai bank's AI lab needed one.
The scores are quoted well past their precision#
OmniDocBench's 981 pages split across nine document types and three language buckets. That leaves roughly 35 to 100 pages per cell.
Rankings off that corpus get published to two decimals: 96.34 against 95.75 against 95.22. No confidence intervals accompany them. At that sample size a handful of pages reorders the top of the table.
It is possible to do better, and olmOCR shows how: it reports its own score as 82.4 ± 1.1. Once you have seen an error bar, the two-decimal rankings look like a choice rather than a convention.
Part of the answer key was written by the contenders#
OmniDocBench's ground truth was produced in stages: machine pre-annotation using LayoutLMv3, PaddleOCR, UniMERNet and GPT-4o, then human correction, then expert inspection. The human correction is real work and the paper describes it plainly.
Still, PaddleOCR helped produce the reference, and a PaddleOCR model leads the leaderboard. That does not make the result wrong. It means the benchmark rewards agreement with the pre-annotator's conventions, which is a different thing from rewarding accuracy.
A wrong digit costs about the same as a wrong comma#
Normalized edit distance treats every character alike.
A model that reads a total of 168,253,200 as 168,253,280 has one character wrong on a dense page. It scores near the top.
For any workflow that touches money that single character is the whole outcome. Everything else on the page can be perfect and the result is still unusable. The metric producing the leaderboard order is close to blind to the failure that matters most.
There is a related gap. A language model asked to read a smudged figure will produce a plausible digit, because producing plausible text is what it does. None of these benchmarks reward a model for marking a figure unreadable.
A model that abstains on 3% of amounts and is right on the rest is more useful than one that guesses well 97% of the time. The second one's mistakes arrive silently, and signed off.
The ranking inverts depending on the document type#
This is the part that undoes the leaderboard, and OmniDocBench reports it honestly in its own paper. The overall score is an average across nine document types, and the ordering underneath that average is not stable.
On newspapers, most VLMs land at edit distances around 0.67 to 0.90, while the pipeline tools sit between 0.17 and 0.69. The pipelines win, and not narrowly.
On handwritten notes it reverses. MinerU's pipeline scores 0.984, which is close to producing nothing usable, while general VLMs generalize considerably better.
Nougat, an expert VLM, dominates books, slides and academic literature, and the paper says it fails almost completely on newspapers and Chinese text.
So "which model is best" has no answer without naming a document type. A single overall number is an average over a document mix, and the mix is the benchmark author's, not yours.
None of those nine types is a business document#
Here is the full list of what OmniDocBench contains: books, slides, financial reports, textbooks, exam papers, magazines, academic papers, handwritten notes, newspapers.
No invoices. No purchase orders. No delivery notes, no quotations, no packing lists. "Financial reports" is the closest, and a published annual report is a typeset document produced by a design team. It has almost nothing in common with a vendor invoice that was printed, stamped, signed, photocopied, and photographed on someone's phone.
The documents that run a business are the ones being scored least. They are also the ones where the numbers on the page turn into money, which is where this post started.
Your document mix is not the benchmark's document mix#
Put those two together and the practical consequence is uncomfortable. A leaderboard ranks models on an average over nine document types, none of them yours, in two languages that may not be yours, with a ranking that demonstrably flips depending on which type you look at.
Your corpus has its own mix: a proportion of native PDFs to phone photographs, its own stamp and signature conventions, its own scanner, its own language, its own decades of layout habits. Nothing in a public score accounts for any of it.
The uncomfortable part is that this cannot be fixed by picking a better leaderboard. It can only be fixed by running the models on your own documents, which is a small piece of work that almost nobody does because the ranking already exists and looks authoritative.
A fully published benchmark gets easier every month#
ThaiOCRBench ships under CC-BY-SA 4.0 with every sample and every ground truth answer downloadable, in a single test split. Releasing it that way was a real contribution, and the licence is why anyone can use it.
It also means it can be trained on, including by accident, and no held-out portion or submission server exists that would let anyone detect it. Scores on it will climb whether or not models get better at Thai documents. This is true of every fully public benchmark and is why the older ones saturate.
The models ship faster than the benchmarks#
PaddleOCR-VL has gone through three generations. Mistral OCR has gone through three. MinerU is on its third major line. OmniDocBench added around nineteen models in a single update in March, and new entrants keep arriving: GLM-OCR, dots.ocr, DeepSeek-OCR-2, MonkeyOCR, Youtu-Parsing, HunyuanOCR, Infinity-Parser2-Pro, Surya 2. New benchmarks appear at a similar rate.
Any evaluation run once is stale within a quarter.
Parameter count stopped predicting quality#
The most useful thing we found has nothing to do with rankings.
Look again at the table at the top. The leader is 0.9B. So is the third place. A 4B model and a 2.5B model both rank below them. On olmOCR-Bench a 1B model edges out a 9B one.
An earlier version of this post said GLM-OCR led at 0.9B, which came from a secondary source. The primary leaderboard has it third on v1.6. Correcting that in a post about not trusting numbers you did not produce seems like the least we can do.
If that holds up, the economics change. A 0.9B model is a different deployment question from a 7B one: it can plausibly run on an ordinary CPU box, next to the documents, without a GPU and without sending anything anywhere.
We have not verified it ourselves yet, and this piece is an argument for not trusting numbers you did not produce.
They still separate a 78 from a 95#
That gap is real, and it is enough to cut a field of twenty candidates to six, which is exactly what we used them for. Most of these are careful work by people who documented their own limitations without being asked. OmniDocBench's paper names its weak spots itself.
The complaint is narrower. These benchmarks measure transcription similarity on clean typeset pages in English and Chinese. A document workflow lives on field accuracy over photographed business paper, often in a language none of them cover.
The answer key can come from the system that filed the document#
Every benchmark above pays people to write down what a page says.
Business documents do not need that. They are attached to a record in the system that filed them, and the purchase order already states the vendor, the date and the amount. The answer key is sitting next to the scan: free, exact for the fields that must agree, and available at a scale no annotation budget reaches.
That changes what is worth scoring. Field level correctness, weighted by what an error costs, so a wrong vendor name and a wrong total are not averaged together. Credit for abstention. Deployment feasibility as a first class result, because a model needing hardware a business does not have is not a candidate at any score.
It also introduces a problem we have not solved. A document is not obliged to agree with its record: a quotation predates the negotiation, a vendor invoice carries the vendor's own numbering, a delivery note may be partial.
So a disagreement has two possible causes, and a benchmark that conflates them would be confidently wrong in exactly the way this post is complaining about. Working out which fields are safe to score is the next thing to do.
References#
- Ouyang et al. OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations. CVPR 2025. arXiv:2412.07626 · github.com/opendatalab/OmniDocBench. Leaderboard figures read from the repository, v1.6, August 2026.
- AllenAI. olmOCR and olmOCR-Bench. github.com/allenai/olmocr. Test counts and the 82.4 ± 1.1 figure are the project's own.
- Nanonets. IDP Core benchmark. benchmarking.nanonets.com. Published by a vendor whose model appears in the results.
- SCB 10X. ThaiOCRBench: A Benchmark for Vision-Language Understanding in Thai Documents. IJCNLP-AACL 2025. arXiv:2511.04479 · dataset, CC-BY-SA 4.0.
- PaddlePaddle. PaddleOCR and PaddleOCR-VL. github.com/PaddlePaddle/PaddleOCR.
- OpenDataLab. MinerU. github.com/opendatalab/MinerU.
- Datalab. Marker. github.com/datalab-to/marker.