Evaluating Table Extraction: Metrics & Benchmarks

Key takeaways

• A 90% TEDS score on a public benchmark tells you nothing about your error rate on a double-header invoice: the only measure that counts is a test set drawn from your own PDFs.
• Structure metrics (TEDS-S, GriTS-Top) and content metrics (TEDS, GriTS-Con) measure two different things, and a parser can excel at one while failing at the other.
• On RD-TableBench, the best published average similarity was 90.2%, which means roughly one cell in ten remains disputable on complex tables: your RAG (retrieval-augmented generation) pipeline needs to be designed with that in mind.
• The errors that break RAG are not character-level typos but row/column misalignments: an amount attached to the wrong header produces a confidently wrong answer.

Why do multi-column invoices break a RAG pipeline?

Because RAG operates on linearized text, and a table with multi-level headers loses its meaning as soon as it's flattened. A typical supplier invoice stacks a "Quantity" column, two price columns (pre-tax / discounted), a VAT rate per line, and sometimes a subtotal per cost center. If the parser merges two columns or shifts a span by one row, the indexed chunk still contains the right figures, but attached to the wrong labels. The embedding still looks plausible, retrieval still works, and the model confidently returns the wrong amount.

This is exactly the type of table that recent benchmarks target. OmniDocBench's tables include nested structures, multi-level headers, and merged cells that put pressure on every system's structural prediction. On the real-world data side, RD-TableBench covers scans, handwriting, language detection, and merged cells. The operational takeaway: your worst documents aren't edge cases, they're the category the literature treats as the reference hard case.

A misaligned chunk doesn't cause an error, it causes a wrong answer that sails through human review.

Which metrics should be used to evaluate a parser on tables?

Three families, measured separately: structure, content, and business-field accuracy. TEDS (Tree Edit Distance based Similarity) compares HTML tree representations of tables, while normalized edit distance measures the transformation cost from one string to another. In OmniDocBench, TEDS combined with NED evaluates both structural accuracy and table content accuracy, while the TEDS-S variant isolates structure alone.

TEDS has a known flaw: it conflates formatting with structure. Hence GriTS. GriTS (grid table similarity) evaluates a predicted table directly in its natural matrix form, unifying cell topology, cell location, and cell content within a single framework. A third approach uses sequence alignment: RD-TableBench uses Needleman-Wunsch scoring, which gives proportional credit to partially correct tables instead of zeroing them out.

None of these metrics tell you whether the total including tax is correct. So add a fourth layer: field-level accuracy on the values your system actually consumes.

Are public benchmarks enough to choose a parser?

No, but they frame your expectations and quickly eliminate weak candidates. The published gaps are wide: on RD-TableBench, table similarity ranged from 90.2% for Reducto to 82.7% for Azure, 80.9% for Textract, 76.0% for GPT-4o, and 60.2% for Unstructured. Thirty points between the top and bottom performer, on a task where every vendor claims to offer "table extraction."

A second caution concerns annotation provenance. PubTabNet and FinTabNet offer large volumes but come from a homogeneous corpus with labels generated programmatically from file metadata, whereas RD-TableBench relies on 1,000 complex table images manually annotated by PhD-level labelers. A third caution concerns contamination: only part of the evaluation framework is published publicly, to avoid leakage into training data. An excellent public score may therefore reflect partial exposure to the test set. We cover these blind spots in more detail in our analysis of OmniDocBench's limitations for invoices.

How do you build a representative internal test set?

Aim for 150 to 300 annotated pages, selected by difficulty rather than at random. Random sampling over-represents simple invoices, the ones every parser handles correctly, and drowns out the signal. Instead, stratify by difficulty attributes, exactly as the literature does: OmniDocBench annotates 5 page attributes, 3 text attributes, and 6 table attributes, which lets you read scores by table type rather than as a single average.

For an invoice corpus, keep at least these strata: two-level headers, vertically merged cells, tables spanning two pages, scans at 200 dpi or lower, skewed photographed documents, multi-currency invoices, invoices with negative discount lines, and structured formats such as Factur-X (the French hybrid PDF/XML e-invoicing format), where the PDF layer sometimes contradicts the XML. Annotate in HTML with rowspan / colspan attributes, plus a dictionary of expected business fields. Reference annotations can include both LaTeX and HTML for tables, but for internal use HTML alone is sufficient and compares directly using TEDS.

A test set of 200 well-chosen pages is worth more than 10,000 pages picked at random from the incoming flow.

How do you test the parser in a RAG context, not just in isolation?

By measuring the end result: the rate of correct answers on questions that require cross-referencing rows and columns. A TEDS score is a lab metric. What matters is whether "what is the VAT amount on the January service line?" returns the right value after chunking, embedding, and generation.

Minimal protocol: generate 300 to 500 single-answer, verifiable questions from your annotations (one cell = one expected answer), then compare two or three linearization strategies while keeping the parser constant. The choice of serialization changes results as much as the choice of model: raw HTML, Markdown, or explicit triplets (row, header, value). Then measure the abstention rate. A system that refuses to answer when the structure is doubtful is worth more than a system that is 92% confident, and this trade-off is governed by arithmetic validation rules: sum of line items = pre-tax total, VAT recalculated by rate, consistent total including tax. These checks catch column misalignments that no similarity metric will.

What non-accuracy criteria should factor into the decision?

Throughput, cost per page, variance, and failure mode. Variance matters as much as the average: a system that performs excellently on the 60% of documents it doesn't fail on reads very differently in a benchmark that separates the failure rate from the quality score. So demand a report broken down by stratum, not a single overall figure.

Cost should be read per page, not per token, and VLM (vision-language model) approaches remain more expensive and slower than classic OCR pipelines at volume, a trade-off we describe in our comparison of OCR versus VLM. Add the question of where processing happens: for supplier invoices containing personal data, data residency and non-retention are eligibility criteria, not optional extras. Sygnet evaluates parsers on all three dimensions jointly (accuracy by stratum, cost per page, processing scope) because a 3-point TEDS gain paid for with five times the latency never makes it into production.

FAQ

What TEDS score should you target before going into production?

There is no universal threshold, because TEDS depends on the corpus. One useful reference point: Nemotron-Parse achieves a TEDS of 86.2 on RD-TableBench and 82.68 on OmniDocBench 1.0 in English. On your own invoices, set the threshold based on the cost of an error: if a misaligned line triggers a payment rejection, aim instead for field-level accuracy above 99% on amounts, with human review on the rest.

TEDS or GriTS: which should you choose?

Use TEDS to benchmark your results against the literature, and GriTS to diagnose. GriTS separates cell topology, location, and content within a single framework, which lets you identify whether the parser is getting the grid wrong or the text wrong. TEDS, by contrast, aggregates everything and conflates formatting with structure. In practice, many teams report TEDS and TEDS-S side by side, then move to GriTS when they need to choose between two closely matched models.

How many pages need to be annotated for a reliable test?

Plan for 150 to 300 stratified pages, roughly 2,000 to 5,000 reference cells. That's enough to detect a 3-to-5-point gap between two parsers on the difficult strata. For a sense of scale on the other end, public benchmarks operate at a much larger scale: GriTS results are averaged over 44,381 tables from the PubTables-1M test set. You don't need that volume for a purchasing decision, but you do need the stratification.