OmniDocBench

OmniDocBench is a public benchmark for evaluating document parsing systems, covering tasks like layout detection, text recognition, table extraction, and formula recognition across diverse document types. It scores models on how accurately they reconstruct a document's structure and content from an image or scan, using a fixed set of annotated pages. Teams building or buying document AI use it to compare OCR engines and vision-language models on a common yardstick.

How it works

OmniDocBench draws on a curated set of document pages: academic papers, textbooks, financial reports, exam sheets, magazines, and more. Each page is manually annotated with ground truth for layout regions, reading order, text content, table structure, and mathematical formulas. A model being tested processes the same pages and its output is compared against these annotations.

Scoring uses task-specific metrics. Text recognition is judged by edit distance against the reference transcription. Table extraction is scored on structural similarity, since a table can be "mostly right" in different ways depending on merged cells or nested headers. Formula recognition uses its own matching logic because formulas have their own syntax. Layout detection is scored on how well predicted bounding boxes and region types align with the annotated ones.

Because the benchmark spans multiple document categories, it also reports per-category breakdowns. A model might do well on clean academic PDFs and poorly on handwritten forms or low-quality scans. That breakdown matters more than the aggregate score for anyone evaluating a tool against a specific use case.

OmniDocBench does not simulate business documents like invoices, insurance claims, or contracts particularly well. Its document mix leans toward academic and general publishing content, so results should be read as a signal of general parsing competence, not a guarantee of performance on your actual document flow.

Why it matters for document processing

Vendors cite OmniDocBench scores in marketing materials, and it's a reasonable starting point for narrowing a shortlist. But a high score does not mean a model will handle your invoices, KYC files, or claims forms well. The benchmark's document categories don't map cleanly onto regulated business paperwork, and its metrics reward general layout and text fidelity rather than field-level extraction accuracy on messy real-world scans.

If you're choosing between an OCR pipeline and a vision-language model for a specific workflow, treat OmniDocBench as one input among several. Run your own test set of representative documents, including bad scans and edge cases you actually see in production, before committing. See our breakdown of where the benchmark falls short for invoice processing specifically.

Related terms

• OCR vs VLM
• Document parsing
• Document layout analysis
• Table extraction
• OmniDocBench: Limitations for Invoice Processing & OCR

FAQ

Is OmniDocBench a good way to evaluate a vendor for invoice or claims processing?

Not on its own. OmniDocBench measures general document parsing quality across academic and publishing-style documents, not the specific fields, layouts, and noise patterns found in invoices, claims, or contracts. Use it to screen candidates, then validate with a test set drawn from your own documents before making a decision.

What does a high OmniDocBench score actually tell you?

It tells you the model reconstructs layout, text, tables, and formulas accurately on the benchmark's own document mix. It does not tell you how the model handles field-level extraction, poor-quality scans, or document types outside that mix, which is often where real business deployments run into trouble.