Why Table Parser Benchmarks Fail RAG Accuracy

Key takeaways

• Table structure scores and retrieval quality are measured on different things: in a benchmark of 8 local extractors, table structure (TEDS) and retrieval hit@5 disagreed, and five tools landed within 0.04 hit@5 of each other.
• TEDS assumes a single correct output, so it punishes valid alternative renderings: one semantically rich HTML table scored TEDS 0.34 while cell content accuracy on the same table was 0.68.
• Reading order, not table fidelity, usually decides where your chunk boundaries fall, and OmniDocBench reports it as a separate metric that never enters the headline Overall score.
• Evaluate parsers on your own corpus with question-level metrics (hit@k, answer correctness), not on a public leaderboard built from other people's documents.

Why does a top-ranked table parser hurt RAG accuracy?

Because the metric that crowned it (TEDS) rewards reconstructing an HTML tree, while your RAG pipeline is graded on whether a retriever finds the right 800 tokens and an LLM reads them correctly. Those two objectives diverge more often than vendors admit. A parser can produce a near-perfect <table> tree that your chunker then slices through the middle of, stripping the header row from rows 14 to 40 and leaving a chunk of bare numbers with no retrievable semantics.

The empirical evidence is blunt. In an open benchmark of 8 local PDF extractors across 27 documents, the authors measured TEDS against hand-annotated table HTML alongside retrieval hit@5 over 233 questions with a bge-large embedder and bge-reranker, and found that table structure and retrieval disagree. The ranking inversion is visible in their own numbers: Marker (no OCR) hit TEDS 0.82 on the real tables it completed but only 0.50-0.52 retrieval hit@5, while Tesseract 5.5 with far worse character error rates (0.17-0.43 CER) topped retrieval at 0.83.

A parser that wins on table trees and loses on reading order will hand your chunker perfect tables in the wrong places.

What exactly does TEDS measure, and what does it ignore?

TEDS measures tree edit distance between a predicted table and one reference HTML table, combining structure and cell content into a single similarity score. It captures structural similarity by comparing tree representations of HTML tables, and it folds cell spatial accuracy (is each cell in the right position with its relationships intact) together with cell content into one number.

What it ignores: everything outside the table box. Reading order across the page, whether a footnote marker stayed attached to its cell, whether the caption survived, whether the multi-page table got stitched or duplicated. OmniDocBench is explicit about this split. It evaluates four dimensions (text transcription by normalized edit distance, formula recognition by CDM, table reconstruction by TEDS and TEDS-S, and reading-order recovery by edit distance over block sequences), but the overall score averages only text, formula CDM and table TEDS. Reading order is reported and then excluded from the headline. If you pick a parser off that leaderboard, you have optimized for three things and ignored the one that governs chunk boundaries.

Why do benchmark rankings flip on your own documents?

Because public corpora are not your corpus, and parser performance collapses unevenly by document type. One study of 800+ documents across 7 frontier parsers found academic papers degraded every system: GPT-5.1 dropped from 84% overall edit similarity to 39% on academic pages (-45pt), LlamaParse from 78% to 38% (-40pt), and Claude Sonnet 4.5 from 78% to 34%. A 40-point swing by document class is larger than any gap between the top five parsers on an aggregate leaderboard.

Failure rate matters more than peak quality too. In the 27-document extractor benchmark, MinerU failed 9 of 27 documents and Docling 8, while PyMuPDF, pdfplumber and Tesseract failed none. A parser scoring TEDS 0.90 on the 70% of pages it completes and returning nothing on the rest gives you a silently incomplete index, which is the worst failure mode in RAG: the system answers confidently from whatever did get indexed. Our own notes on evaluating extraction accuracy start from coverage for exactly this reason.

Which metric should you actually optimize?

Optimize question-level metrics on your documents: retrieval hit@k, then answer correctness, with parse-level scores as diagnostics only. The right experimental design varies one component at a time. The FinanceBench-derived work on financial QA does this explicitly: they evaluate end-to-end RAG pipelines varying parsers, chunking strategies and LLMs on both FinanceBench and a table-derived QA set, measuring the impact of each component choice on retrieval accuracy and answer correctness.

Two practical substitutions help. First, replace single-reference scoring with semantic scoring: one 2026 comparison reports that LLM-based semantic evaluation of tables reached Pearson r=0.93 with human judgment versus r=0.68 for TEDS alone. Second, use fact-checking benchmarks rather than string distance where you can. olmOCR-Bench verifies unambiguous machine-checkable facts about each page, like whether a specific sentence appears exactly in the output, because edit distance may penalize valid variations or miss critical semantic errors.

How do parser choices propagate into chunking errors?

Structure determines chunk boundaries, so a structural error becomes a retrieval error one step later. A parser can ace text extraction at 80% and fail structure recovery at 40%, and that gap matters for RAG pipelines where document structure determines chunk boundaries and retrieval quality. The reading-order metric is the leading indicator here: one 2026 comparison argues directly that reading order is the metric that governs whether chunk boundaries fall where they should.

A second, quieter failure: layout models that classify a chart as a picture. The 27-document benchmark found that layout-analysis tools classify charts as pictures and throw their text away, with Docling recovering 3% of chart facts while EasyOCR recovered 84% and Tesseract 65%. Nothing in a TEDS score will warn you that a third of your figures silently vanished. If your questions touch chart annotations, axis labels or callouts, a high-TEDS layout pipeline can be the worst choice available.

Nothing in a table-structure score will tell you that a third of your figures were classified as pictures and discarded.

When is a high TEDS score the right thing to chase?

When your queries resolve inside dense numeric tables and your chunker is table-aware. Financial statements, regulatory filings and lab result grids genuinely reward structural fidelity, and the spread between parsers on those documents is large: on the opendataloader-bench corpus, Docling 2.110.0 scores TEDS 0.93 against the standard Nutrient CLI's 0.74, and the hybrid configuration lifts table TEDS from 0.489 to 0.928. That is not noise.

The condition is that the table arrives at the LLM intact, with headers repeated per chunk and the caption attached. If you serialize each table as a standalone unit with its header row and a short natural-language summary prepended, a 0.93 TEDS parser beats a 0.74 one. If you naively split by token count, the structural advantage evaporates. Sygnet works on document extraction pipelines where this distinction shows up constantly, and the RAG document ingestion checklist covers the serialization step. The parser-model question itself (layout pipeline versus vision-language model) is handled separately in OCR vs VLM.

FAQ

Is TEDS a bad metric?

No, it is a precise metric for a narrow task: reconstructing one table against one reference tree. The problem is treating it as a proxy for RAG quality. CER and TEDS assume one correct output, so they penalize the multiple valid representations that generative and VLM parsers produce, which can reorder rankings. Report it alongside reading order, coverage and retrieval, and it stays useful.

How many documents do I need for a meaningful internal benchmark?

Fewer than you think, if they are stratified. The 8-extractor study worked with 27 documents spanning 1 to 487 pages and 233 questions, which was enough to separate parsers by failure rate and chart handling. Aim for 20-40 documents covering every layout family you actually ingest (scans, native PDFs, multi-column, dense tables) plus 10-15 questions per family with known answers.

Which number should I report to stakeholders?

Answer correctness on your own question set, with retrieval hit@k as the diagnostic underneath it. Parse-level scores explain why a pipeline failed, they do not predict whether it will. If the business case needs cost attached, the ROI calculator pairs volume and error rate with downstream review time.