OmniDocBench: Limitations for Invoice Processing & OCR

Key takeaways

• OmniDocBench's Overall metric is a page-level average of text edit distance, Table TEDS and formula CDM (the formula is ((1 − text edit distance) × 100 + table TEDS + formula CDM) / 3), so it says nothing about whether your invoice's VAT total landed in the right JSON key.
• The benchmark corpus is academic: 1,355 bilingual pages drawn from books, slides, financial reports, textbooks, exam papers, magazines, academic papers, handwritten notes and newspapers. No supplier invoices.
• Rebuild the same pages as physical photographs and scores fall: optical artifacts like moiré, glare and shadows cause 5–7% relative drops, and geometric distortion (warping, skew) causes the steepest degradation.
• Field-level reality: invoice totals and vendor names hit 99%+, but line items and multi-row tax breakdowns typically land at 95–97%, and a single wrong line item fails the whole document.

What does a 94% OmniDocBench score actually measure?

It measures how closely a model's markdown or HTML reconstruction of a page matches a human-annotated reconstruction, averaged over three sub-metrics. Text and reading order are scored with edit distance, tables with Tree-Edit-Distance-based Similarity (TEDS), and formulas with Character Detection Matching (CDM). The composite is an unweighted mean of those three. So a model can score 94 while mangling every table on the page, provided its text edit distance and formula CDM are excellent.

Progress is real. PaddleOCR-VL-1.5 raised the overall score from 92.86% to 94.50%, and on the newer v1.6 protocol the top six overall scores fall within 94.49–96.19, a range of only 1.70 points, with MinerU2.5-Pro at 96.12 and GLM-OCR 0.07 points ahead. That clustering is the tell. When six systems sit inside two points, the benchmark has stopped discriminating, and the ranking you read is noise plus test-set fit.

When six systems sit inside two points, the benchmark has stopped discriminating.

Worth remembering how the corpus was built: the original pages were collected from diverse web sources, including Common Crawl and search engines. Public web PDFs. Your accounts-payable inbox is not public web PDFs.

Why don't page-level parsing metrics predict invoice field accuracy?

Because edit distance rewards getting most characters right, while an invoice pipeline is graded on whether a discrete set of keys is exactly right. Get 98% of characters on a page and you still have a corrupted IBAN. Reading order illustrates the same mismatch: read-order edit distance on OmniDocBench 1.5 ranges from 0.026 to 0.082, so even the best models struggle with layout logic. A 0.05 reading-order error is cosmetic in a textbook and fatal on a two-column invoice where a line-item description gets stitched to the wrong quantity.

Field-level benchmarks expose this directly. RealDocBench, built on regulated documents, reports 96.0% per-field and 90.9% per-question accuracy for the leading system, and notes that requiring all keys to be correct drops most systems by 5–17 points relative to their per-field score. That per-question metric is what your AP team feels: a document either clears or it goes to review.

If you are choosing an approach, the trade-offs between pipeline OCR and end-to-end models are structural, not just accuracy points: see the OCR vs VLM comparison for how each fails.

What happens when the invoice is a phone photo, not a clean export?

Scores drop, and now there's a benchmark that quantifies it. Real5-OmniDocBench performs a full-scale, one-to-one physical reconstruction of the entire OmniDocBench v1.5 (1,355 images) across five scenarios: Scanning, Warping, Screen-Photography, Illumination and Skew, producing 6,775 images with one-to-one digital/physical ground-truth mapping. Apart from the scanning subset, all images were captured by hand with mobile devices. The authors state the motivation bluntly: VLMs achieve near-perfect scores on digital benchmarks like OmniDocBench, but their performance in the unpredictable physical world remains largely unknown.

Results split by failure mode. Warping and skew induce the steepest degradation, with TEDS dropping sharply in pipeline models and only modestly in VLMs tuned for geometric invariance, while screen-photography and illumination cost 5–7% relative. Best reported configurations still land around 92.05% overall, below clean-page numbers. And the authors concede an inherent penalty: handheld capture introduces a resolution gap compared to digital originals. Your expense receipts and freight documents live in exactly that gap.

Which invoice fields actually break?

Header fields are effectively solved; the failures cluster in repeating structures and in anything requiring arithmetic consistency. Published field-tier data puts traditional OCR-only approaches at 85–95% on structured invoice fields and LLM-based extraction at 97–99%, with simple unique fields like totals and vendor names above 99% and complex fields such as line items and multi-row tax breakdowns at 95–97%. Multiply 96% per line across a 30-line invoice and straight-through processing collapses.

Format irregularity, not image quality, is often the real culprit. One invoice benchmark found low resolution had minimal impact on detection quality and multi-line descriptions did not degrade most models, but unconventional invoice structures with missing fields or sub-items caused detection failures across all models. That matches AP experience: credit notes, retention lines, multi-currency, discounts applied at footer level, and VAT stated per line rather than per rate. Vendor accuracy claims are also conditional. A vendor at 99% on clean digital invoices from large suppliers may achieve 85% on scanned invoices from long-tail suppliers with non-standard formats.

Header fields are solved; the money is lost in line items, tax breakdowns and credit notes.

The fix is rarely a better parser. It is deterministic validation rules on top of extraction (line sum equals subtotal, VAT rate applied correctly, supplier matched to a master record) plus table extraction evaluated row-by-row rather than cell-by-cell.

How should an engineering team benchmark its own invoice pipeline?

Build a 300–500 document gold set from your own inbox, stratified by supplier tail, capture channel and document variant, then score per field, per document, and per euro exposed. Public leaderboards are useful for shortlisting only. As one evaluation vendor puts it, academic benchmarks compare models but lack production workflow integration.

Practical protocol:

1. Stratify by source (native PDF, scan, photo, screenshot) and by supplier volume decile. Long-tail suppliers deserve over-representation.
2. Score exact-match per field, then per-document all-keys-correct. Report both, always.
3. Track straight-through rate as the headline. One vendor reports 85 to 92 percent production straight-through in mixed enterprise AP environments, arguing buyers should measure straight-through rate rather than extraction accuracy alone. Treat the specific figure as vendor-reported, but the metric choice is correct.
4. Calibrate confidence scores against observed error rate, then set review thresholds by financial exposure rather than a single global cut-off.
5. Re-run quarterly. Model updates shift behaviour on your tail suppliers well before they shift leaderboard positions.

Sygnet runs this kind of per-field and per-document evaluation on customer document mixes rather than on public corpora, because the two rank systems differently. For the cost side of the same question, the manual invoice processing cost breakdown shows how much a five-point straight-through difference is worth per 10,000 invoices.

FAQ

Is OmniDocBench useless for invoice work?

No, it's just the wrong instrument. It's a strong signal for document conversion quality: RAG ingestion, markdown reconstruction, formula and table structure fidelity. It supports multi-level evaluation using 19 layout categories and 15 attribute labels, which is genuinely useful for diagnosing layout failures. Use it to eliminate weak candidates, then decide on your own field-level gold set.

Which benchmark comes closest to real invoices?

Real5-OmniDocBench for capture robustness, RealDocBench for field-level extraction on regulated documents. Real5 isolates scanning, warping, screen photography, illumination variation and skew, which maps cleanly onto how invoices and receipts actually arrive. Neither contains invoices specifically, so both remain proxies. There is currently no widely adopted public invoice benchmark with open ground truth at field level.

What straight-through rate should we target?

Set the target by exception cost, not by vendor marketing. If a reviewed invoice costs three minutes and an undetected error costs a payment correction, the optimal threshold is usually well below 100% automation. Published production figures cluster in the 85 to 92 percent range for mixed AP. Design for human-in-the-loop review on the residual and measure the reviewers' throughput as part of the system.

Do better OCR scores translate into better downstream extraction?

Only up to a point. Once text edit distance is below roughly 0.05, the bottleneck moves to structure and semantics: reading order, table row grouping, and mapping values to the correct schema keys. That's why formula CDM and read-order remain the differentiators among top models while text scores converge. For invoices, spend engineering effort on schema validation and reconciliation, not on the last character-error percentage point.