Table Extraction Metrics: GriTS, TEDS, and Semantic Evaluation
By Sygnet Research, checked before publication
Key takeaways
- GriTS compares tables as 2D matrices and TEDS compares them as HTML trees, so both score where a cell sits and what string it holds, never whether the header above a column is the right header for that column.
- In a human validation study of over 1,500 quality judgments, LLM-based semantic evaluation correlated with human judgment at Pearson r=0.93, versus r=0.68 for TEDS and r=0.70 for GriTS.
- TEDS penalises a missing row more than a missing column, because
trnodes sit higher in the DOM thantdnodes: dropping a column from a 5-row table costs 5 edits, dropping a row costs 6. - If your downstream task is key-value reading (invoice line items, lab panels, financial statements), add a header-binding check and a per-field exact-match test on top of GriTS/TEDS, because a 0.95 GriTS table can still invert two column headers.
What exactly do GriTS and TEDS measure?
They measure structural and string agreement between a predicted table and a ground truth table, nothing about the meaning of the association between a header and its column. GriTS evaluates a predicted table directly in its natural form as a matrix, generalising the NP-hard 2D largest common substructure problem to a "2D most similar substructures" problem solved with a polynomial-time heuristic. A particular GriTS variant is defined by a choice of grid cell property plus a similarity function: topology, content, and location. For content, cell text is compared with normalised longest common subsequence.
TEDS takes the tree route. Tables are represented as HTML trees with thead and tbody children, tr rows, td leaves carrying colspan, rowspan and content attributes, and similarity is computed with the Pawlik–Augsten tree-edit distance at unit insertion and deletion cost. Both metrics ask: does cell (i, j) exist, and does its string roughly match? Neither asks whether column 3's values belong under the label sitting in row 0, column 3.
Why do inverted column headers slip through?
Because swapping two header labels is a tiny string-level perturbation that leaves the grid geometry untouched. Take a 12-column financial table where "Q1 2025" and "Q2 2025" land above the wrong columns. Grid topology: identical. Cell locations: identical. Only two content cells differ, each still partially similar under LCS or Levenshtein. On a 12x20 table that is 2 cells out of 240, so content-based scores stay above 0.99 while every number in the table is now attributed to the wrong period.
Two swapped labels out of 240 cells costs less than one percent of your metric and one hundred percent of your data's correctness.
TEDS, GriTS and SCORE compare structure and content purely syntactically, so they cannot distinguish discrepancies that alter the semantic content of a table from those that are merely representational. The asymmetry cuts both ways: harmless reformatting gets punished, and header inversion, which destroys semantics, gets almost nothing. Parsers routinely flatten multi-level headers or resolve rowspans because formats like Markdown have no colspan/rowspan mechanism, producing repeated values or empty padding cells that drag scores down for no real loss of information.
Isn't header detection a separate task?
Yes, and that separation is precisely why the popular metrics are blind to it. The table extraction literature splits the problem into three parts. Functional analysis recognises the keys and values of a table, and is distinct from table structure recognition, which the PubTables-1M authors treat from three perspectives: cell topology, cell content and cell location. GriTS was built to isolate TSR from functional analysis, partly as a criticism of earlier metrics. The authors note that metrics from Zhong et al. and Li et al. do not isolate TSR from functional analysis, mixing aspects of both, which motivated new metrics preserving a table's true topological structure.
That design choice is defensible for model research and misleading for production. A clean TSR score tells you the grid was reconstructed; it says nothing about whether your extraction pipeline will put the right number in the right field. TEDS at least carries a thead/tbody split, which partially encodes header role, but it still treats header labels as interchangeable strings inside that region.
What do these metrics actually catch and miss?
| Failure mode | GriTS | TEDS | Field-level exact match |
|---|---|---|---|
| Missing row or column | Caught, symmetrically (GriTS-Con returns a nearly identical value whichever half of the rows or columns is removed) | Caught, but asymmetrically (missing rows incur a higher penalty than missing columns) | Caught if the field is in scope |
| Multi-hop cell shift | Caught | Caught (90% first-row shift gives TEDS 34.9% vs adjacency-relation F1 80.3%) | Caught |
| OCR noise in cell text | Partial credit via LCS | Partial credit via Levenshtein | Caught, hard fail |
| Two column headers swapped | Barely registers | Barely registers | Caught |
| Markdown header flattening | Robust to markup variations | Over-penalised | Not applicable |
| Semantic equivalence (units, number formats) | Missed | Missed | Missed unless normalised |
How do you actually test for header orientation?
Evaluate the key-value binding directly, not the grid: for each ground truth (header, row label, value) triple, check that the parser's output reproduces the same triple. This turns table extraction into a field-level test and makes header inversion a hard failure instead of a rounding error. Three practical layers:
1. Triple-match accuracy. Serialise both tables to (column header, row header, cell value) tuples and compute exact-match precision and recall. A swapped header breaks every tuple in two columns at once, which is the signal you want.
2. Adversarial perturbation suite. Build it the way the metric papers built theirs: controlled corruptions of ground truth. Evaluating a metric on modified or corrupted ground truth is how you expose its sensitivity or insensitivity to a given property. Permute header labels, transpose header blocks, drop a header row, and confirm your score drops accordingly. If GriTS moves by 0.004 on a header swap, you have documented the blind spot.
3. Semantic judging where tuples are ambiguous. LLM-as-a-judge reached r=0.93 against human judgment in the benchmark cited above, against 0.68 and 0.70 for TEDS and GriTS, which makes it a reasonable tiebreaker for unit changes and reformatted multi-level headers. Keep it as a second pass, not the primary gate: it is slower and nondeterministic.
Sygnet publishes a checklist for evaluating extraction accuracy that covers field-level test design, and a RAG document ingestion checklist for teams whose tables end up in a retrieval index rather than a database.
Why does this matter more for RAG and VLM pipelines?
Because header-to-value binding is the only thing a retrieval chunk carries, and a vision-language model will happily produce a fluent, well-formed table with the labels in the wrong order. Benchmarks like OmniDocBench (1,355 pages) and READoc (3,576 documents) include explicit table evaluation but rely on TEDS and edit distance, capturing structural similarity without assessing semantic equivalence. So a model can top a public leaderboard and still invert headers on your documents, because the leaderboard metric has no term for it.
It gets worse when tables are flattened for embedding. Once a table becomes Markdown or a sentence per row, the header association is baked in permanently: there is no downstream step that can repair it, and a retrieval system has no way to detect the inversion. Compare also the unit-test style approach used by some parsers, where olmOCR-Bench includes 1,020 table-specific unit tests across 1,402 PDFs focused on cell-level pass/fail verification. Binary cell assertions are coarse, but at least a well-written assertion can encode "this value lives under this header".
For anyone building rather than buying, the practical conclusion is that your eval harness needs a layer the research metrics do not provide. The trade-offs are laid out in Sygnet's build vs buy analysis for IDP.
FAQ
Is GriTS better than TEDS for production table extraction?
GriTS is better behaved structurally. It operates on the 2D grid rather than a tree, treats rows and columns symmetrically, and is robust to markup variations, though content comparison remains string-based. TEDS penalises missing rows more than missing columns and over-reacts to Markdown-style header flattening. Neither is sufficient alone: both will score a header-inverted table above 0.95 on a large grid.
How large is the blind spot numerically?
It scales inversely with table size. Swapping two header labels changes 2 content cells. On a 10x20 table that is roughly 1% of cells, and because LCS gives partial credit for shared characters (common in labels like "Q1 2025" and "Q2 2025"), the actual drop is usually well under 1%. Meanwhile 100% of the values in both columns are now mislabelled.
What should we report alongside GriTS?
Report three numbers: GriTS-Con F-score for structure, field-level exact-match accuracy on the keys your business logic reads, and a header-binding pass rate from an adversarial permutation suite. The first tells you whether the grid parsed, the second whether the data is usable, the third whether the second number is trustworthy on documents whose headers you have not seen.
One email when we publish something worth your time.