# Evaluating Financial Reconciliation Pipelines: A Guide

> Learn how to effectively evaluate financial reconciliation pipelines, focusing on reconciliation verdicts, tolerance bands, and specific metrics for each stage.

Published: 2026-10-25

Canonical: https://sygnet.ai/blog/evaluating-financial-reconciliation-pipelines-a-guide

---

## Key takeaways

- Field-level accuracy is the wrong headline metric here: the thing you ship is a **reconciliation verdict** per figure (match, explained variance, unexplained gap), so your eval set must be labelled at the level of triplets (CIM figure, liasse fiscale figure, audited figure), not single cells.
- Benchmarks say extraction is the easy half. On FinSheet-Bench, accuracy collapses from roughly 89% on simple lookups to around 20% on complex aggregations across all models, which is exactly the shape of a reconciliation task.
- Build tolerance bands into the ground truth: a €3k delta between a liasse fiscale and audited accounts is rounding, a €300k delta is a finding. A binary exact-match score can't tell those apart and will punish a correct pipeline.
- Numbers matter commercially: GF Data found sellers arriving with a sell-side QoE traded at 7.4x TEV/EBITDA on average versus 7.0x for those who did not, so false reassurance from a reconciliation engine has a price tag.

## Why does a reconciliation pipeline need a different eval than a normal extraction pipeline?

Because the output is a comparison, not a value, and comparisons fail in ways that single-document extraction metrics never surface. A parser can read "EBITDA 12 480" correctly from the CIM and "Résultat d'exploitation 11 920" correctly from the liasse fiscale, score 100% on field accuracy, and still produce a garbage verdict because the two labels are not the same concept. French liasses fiscales (form 2052/2053 for the income statement, 2050/2051 for the balance sheet) use statutory French GAAP line items; audited consolidated accounts may be IFRS; the CIM shows adjusted, pro-forma, management EBITDA. Three different definitions of "the same figure".

So the unit of evaluation becomes the mapped triplet plus the adjustment that bridges them. Quality of Earnings is roughly 30% of buy-side financial due diligence effort and covers EBITDA reconciliation, one-time item identification and pro-forma run-rate adjustments, with an Adjusted EBITDA bridge as the deliverable. Your pipeline is effectively drafting that bridge. Score the bridge.

> A parser that reads every cell perfectly and maps "résultat d'exploitation" onto "adjusted EBITDA" has not made an extraction error, it has made a reconciliation error, and only one of those shows up in field accuracy.

## What should the ground-truth set actually contain?

A reconciliation ground truth is a table of triplets with a verdict and a tolerance, built from closed deals where a human team already did the work. Minimum viable set: 3 to 5 historical deals, 12 to 24 months of periods each, and roughly 150 to 400 reconciled line items per deal. Anything smaller and your confidence intervals on a 2% error rate are useless.

Each row should carry: the canonical concept (revenue, EBITDA, net debt, NWC, capex), the source values with page and bounding-box provenance for all three documents, the document identity (CIM v3, liasse 2052 FY2023, audited FY2023 signed), the expected delta, the explanation category (GAAP difference, scope/perimeter difference, non-recurring add-back, reclassification, genuine discrepancy), and the tolerance in both absolute euros and basis points of revenue. Label the hard cases deliberately: restated comparatives, change of closing date, a subsidiary consolidated mid-year, intercompany eliminations. If your eval set contains only clean years, you are measuring nothing. The generic version of this work is laid out in our [extraction accuracy evaluation checklist](https://sygnet.ai/checklists/extraction-accuracy-evaluation).

## Which metrics do you report, and at what threshold?

Report four, separately, because they fail independently: localisation, value fidelity, concept mapping, and verdict quality. Collapsing them into one "accuracy" number hides which stage broke.

| Stage | Metric | Suggested gate | Failure it catches |
|---|---|---|---|
| Table structure | TEDS / Needleman-Wunsch cell alignment | ≥ 95% on your own doc mix | Merged cells, multi-level headers in liasse PDFs |
| Value fidelity | Exact numeric match after normalisation (thousands, parentheses-negatives, French decimal comma) | ≥ 99.5% | Sign flips, unit errors, OCR digit swaps |
| Concept mapping | Macro-F1 over the canonical chart of accounts | ≥ 0.95 on top 30 concepts | Mapping résultat d'exploitation to EBITDA |
| Verdict | Recall on true discrepancies, at fixed false-positive rate | Recall ≥ 0.98 | The only error a buyer actually pays for |

Calibrate the gates against published baselines rather than vendor claims. On FinTabNet, DocLD reports 82.1% average table accuracy, 4.1 points above IBM's GTE and 17.1 points above Microsoft's TATR, while a spatial-transformer approach reports a 98.09% TEDS score on FinTabNet. The gap is mostly metric definition, which is the point: ask any vendor which scorer and which split, or the number means nothing. For the parser-layer tradeoffs behind those numbers, see [OCR vs VLM](https://sygnet.ai/ocr-vs-vlm).

## How do you score "explained variance" without rewarding hallucinated explanations?

Score explanations with a two-key test: the category must be right, and the cited evidence must exist at the claimed location. An LLM that invents "non-recurring restructuring charge" for a €412k delta and cites note 7 of the audited accounts is right 40% of the time by luck and wrong in a way that passes human skim-reading.

Operationally: require every explanation to emit a span pointer (document ID, page, cell or paragraph offset). Then compute explanation precision only over explanations whose pointer resolves to text containing the figure. Everything unpointed scores zero, regardless of plausibility. In our experience this single rule cuts apparent performance by 10 to 20 points on first run and is the most useful signal you will get.

Keep the arithmetic out of the model. FinSheet-Bench's authors conclude that reliable financial spreadsheet extraction will likely require architectures that separate document understanding from deterministic computation. Let the model locate and classify; let a deterministic engine sum the bridge and test it against the tolerance. Then your eval has two clean halves instead of one fuzzy one.

## What does a realistic error budget look like on a mid-market deal?

Work backwards from reviewer hours, not from model benchmarks. A mid-market data room holds 5,000 to 50,000+ pages, and deal teams routinely face 30,000 to 100,000+ documents. If your pipeline surfaces 400 reconciliation items and runs a 5% false-positive rate, that is 20 bogus findings for an analyst to chase, at maybe 25 minutes each: roughly a day burned. Acceptable. At a 2% false-negative rate on 400 items, you have missed 8 real gaps, and one of them might be the working-capital adjustment that moves the price.

That asymmetry should be explicit in your scoring function. Weight recall on discrepancies 5 to 10x over precision, and report precision at fixed recall rather than F1. The downside is documented: an ACG survey cited by ChatFin reports that 38% of US acquirers found material financial issues post-close that were present in the data room but not identified during diligence. Also measure throughput honestly against deal timing, given average diligence now runs 203 days, up from 124 days pre-2013. If you want unit economics rather than percentages, [cost per correctly extracted field](https://sygnet.ai/blog/cost-per-correctly-extracted-field-the-true-parser-metric) is the better frame.

> Weight recall on discrepancies five to ten times over precision, because a false positive costs an analyst twenty minutes and a false negative costs the buyer a price adjustment.

## How do you test drift and document-version chaos?

Treat document versioning as a first-class test dimension, because in a live data room it is the main source of silent error. CIMs get reissued, liasses get amended, audited accounts arrive in draft and then signed, and a pipeline that silently mixes CIM v2 with audited FY2024 final will reconcile two numbers that were never meant to agree.

Three tests worth automating. First, a version-confusion test: inject a stale CIM alongside the current one and assert the pipeline either picks the newest by declared date or refuses. Second, a perimeter test: feed statutory accounts for the parent only against consolidated audited accounts and assert the pipeline flags scope mismatch rather than reporting a €2m "discrepancy". Third, a regression suite pinned to model versions, rerun on every prompt or model upgrade, since mapping behaviour shifts between model releases even when table extraction does not.

Access control belongs in the same test plan: data rooms run on granular per-bidder permissions, structured Q&A workflows and detailed audit trails, and an extraction layer that bypasses them is a finding of its own. Sygnet publishes its [security and compliance posture](https://sygnet.ai/security) for exactly this review step, and documents the deal-side workflow under [M&A and due diligence](https://sygnet.ai/solutions/private-equity-due-diligence).

## FAQ

### How large does the labelled set need to be for a credible accuracy claim?

For a claim like "99% value fidelity, ±1 point", you need on the order of 1,000 to 2,000 labelled numeric fields, which 3 to 5 deals usually supply. For verdict-level recall on discrepancies, the binding constraint is the number of true discrepancies, not the number of fields: aim for at least 50 confirmed real gaps across the set, otherwise your recall estimate swings wildly with one or two labelling disagreements.

### Can one model handle both the extraction and the reconciliation?

Technically yes, but the evidence argues against it for production. FinSheet-Bench found accuracy dropping from above ~90% on simple lookups to just above 30% on complex calculations for the top three models, with the authors noting that none of the ten configurations tested produced error rates acceptable for unsupervised production use. Split the job: model for locate-and-classify, deterministic code for arithmetic and tolerance tests.

### What tolerance should trigger a flag between liasse fiscale and audited accounts?

Use a two-part rule rather than a single euro threshold: flag anything above max(€25k, 10 bps of revenue) for a mid-market target, and flag any delta regardless of size on a short list of sensitive lines (related-party balances, provisions, revenue recognition cut-off, deferred revenue). Tune both from your historical deals, and record the tuned values in the eval config so the score is reproducible across model versions.

### Does this change if the target files abroad or reports under IFRS?

Yes, and it mostly changes the mapping layer. Add a jurisdiction field to every document record, maintain one canonical chart of accounts with per-jurisdiction mapping tables, and score concept mapping per jurisdiction. A pipeline at macro-F1 0.96 on French liasses can land near 0.80 on Belgian or German filings, and an aggregate score will hide that entirely.