# Invoice vs. Claims Document Extraction: Why Models Fail

> Explore why AI models trained on invoices struggle with complex claims documents, highlighting key differences and solutions for improved extraction accuracy.

Published: 2026-10-12

Canonical: https://sygnet.ai/blog/invoice-vs-claims-document-extraction-why-models-fail

---

## Key takeaways

- An invoice model is a template matcher in disguise: on Google's VRDU benchmark, micro-F1 for unseen templates fell 13 to 17 percentage points below mixed-template performance at 200 training documents, and expert reports have no templates at all.
- Receipt benchmarks flatter everyone. Top scores sit at 96-97% on CORD and SROIE but only around 45% on VRDU's Ad-buy Forms with 200 training samples.
- The hard part of a claims file is not OCR, it is hierarchy and reasoning: VRDU reports a 60 to 70 point gap between hierarchical entities and flat entities across all training-set sizes.
- Build your schema around evidence spans and document-level reconciliation, not around 8 flat fields, and evaluate per field type rather than with a single average.

## Why does an invoice-trained extraction model fail on an expert report?

Because invoice extraction is a **bounded key-value task** on a near-fixed layout, while an expert report (loss adjuster report, repair estimate, medical assessment) is a multi-page narrative where the answer is a sentence, a conclusion, or a number buried in prose. The model has learned spatial priors ("total is bottom-right, near a currency symbol") that do not exist in a 14-page report. And it has learned a schema of 8 to 20 flat fields when a claims file needs causality, dates of occurrence, exclusions, and liability shares.

Public benchmarks show how narrow that training distribution is. SROIE, the benchmark most extraction papers cite, is 1,000 scanned receipts with four key fields: company, date, address and total. CORD is receipt images from shops and restaurants with 30 entities grouped into Menu, Void menu, Subtotal and Total. A model tuned on that distribution has never been asked to decide whether a described damage is covered.

> A benchmark built on 400 restaurant receipts cannot predict how a model behaves on a 14-page loss adjuster report.

The failure is rarely loud. The model returns valid JSON, plausible numbers, and confidence scores that look fine. That is precisely why claims teams find the errors three weeks later, in reserve calculations.

## How large is the drop when layouts change?

Measured drops range from roughly 9 F1 points to more than 25, depending on how much of the original benchmark score came from memorised layouts. One study resampled the FUNSD and SROIE splits to remove information redundancy between train and test; average F1 fell from 94.34 to 85.60 and the best score fell from 96.55 to 89.38. Text-only models suffered most: BERT, AlBERT and RoBERTa dropped 10.5 F1 points on average versus 7.5 for multi-modal models.

Template generalization tells the same story more bluntly. In VRDU's Unseen Template Learning task, train and test documents come from disjoint sets of templates, which is the hardest setting since new templates may look substantially different. Newer did not mean more robust: LayoutLMv3 degraded most on unseen templates, from 89.77 to 62.58 micro-F1, trailing LayoutLMv2 at every training-set size. If a 27-point drop happens between two variants of the same registration form, assume a larger one between an invoice and an expert report.

## What actually differs between an invoice and a claims document?

The difference is structural, not cosmetic: field cardinality, page count, evidence location, and the amount of inference required between the text and the target value.

| Dimension | Invoice / receipt | Expert report, claims file |
|---|---|---|
| Length | 1-2 pages | 5-40 pages, often several documents |
| Layout variability | Dozens of templates, strong positional priors | No template; free prose, photos, annexes |
| Target fields | 4-20 flat fields (SROIE: 4 entities) | 30+ fields, nested, conditional, repeated |
| Answer form | Verbatim string on the page | Often inferred across pages |
| Error detection | Arithmetic checks (total = sum of lines) | No internal checksum |
| Benchmark proxy | CORD, SROIE (96-97%) | VRDU Ad-buy, DUDE (much lower) |

Hierarchy is the specific killer. Line items, repeated damages, multiple injured parties: these are nested entities, and VRDU notes that no existing models take the hierarchical entity type into consideration, with a grouping heuristic standing in instead. A claims file is almost entirely nested entities.

## Are multi-page, multi-domain benchmarks any better as a proxy?

Yes, and they are sobering. DUDE was built as a large-scale multi-paged, multi-domain, multi-industry DocVQA benchmark, and its authors report that zero-shot and fine-tuned performance of state-of-the-art models lags far behind human baselines, partly because of the need for more holistic modeling of language, vision and richly structured layouts. The dataset deliberately includes tables, charts, figures, lists, checkboxes and stamps, which is roughly the content inventory of a real claims folder.

DUDE also names the right framing for this problem: multi-domain long-tailed recognition, with label imbalance, divergent label distributions across domains, and possible train-test domain shift. That is exactly what a claims portfolio looks like. Auto glass claims arrive by the thousand; subsidence and construction defect reports arrive in dozens, with the most expensive consequences. Any model trained on your high-volume tail will underperform on the documents that actually move the reserve.

## What should engineering teams do instead of fine-tuning harder?

Change the task definition before changing the model. Three moves pay off more than another round of fine-tuning.

**1. Split the pipeline into parse, locate, decide.** Parsing quality can be measured separately from reasoning. LandingAI showed the usefulness of this separation by scoring parse fidelity rather than a VLM: 5,286 correct out of 5,331 DocVQA questions (99.16%) answered by an LLM using only the parsed output, with no image access during the QA step. Whether your parser is OCR-based or a vision-language model changes the failure modes you inherit, which is the trade-off covered in [OCR vs VLM for document extraction](https://sygnet.ai/ocr-vs-vlm).

**2. Require evidence spans for every field.** A field without a page, a bounding box or a quoted sentence cannot be adjudicated by a human reviewer in under a minute, so it will not be reviewed. Grounding also makes silent drift visible.

**3. Score per field type, not in aggregate.** Dates, amounts, free-text conclusions and nested tables fail differently. A single 94% average hides a 45% score on line items, which is the VRDU lesson in one sentence. Sygnet publishes [a checklist for evaluating extraction accuracy](https://sygnet.ai/checklists/extraction-accuracy-evaluation) that walks through field-level scoring and sampling plans.

> Measure your own documents on your own fields, because no public benchmark contains a loss adjuster report.

## How many documents do you need in your own evaluation set?

Plan for 150 to 300 labelled documents per document type, stratified by claim line, not a flat random sample. VRDU used 300 documents in the testing set for each task, and its few-shot splits were sized at <cite name="" index="23-11,23-12">10, 50, 100 and 200 training instances precisely to mitigate the large variance from sampling different training docs</cite>. That variance is the reason small internal pilots produce wildly different accuracy numbers from one week to the next.

Arithmetic matters as much as sampling. Per-field accuracy compounds: at 98% per field across eight fields, roughly 85% of documents come through clean, which is arithmetic rather than a measurement. Push that to a 30-field claims schema and per-field accuracy has to be far higher to get an acceptable document-level pass rate. Decide upfront which fields are blocking (reserve amount, date of loss, liability share) and which can tolerate review, then set thresholds per field. For claims-specific field schemas, Sygnet documents its approach to [insurance claims processing](https://sygnet.ai/solutions/insurance-claims) and to [medical report extraction](https://sygnet.ai/documents/medical-report), which shares the same narrative-document difficulty.

Also note that vendor-published numbers rarely help: one review found that none of the three major cloud invoice models publish an accuracy figure in their documentation. Your own stratified test set is the only number you can defend to an actuary.

## FAQ

### Can a general-purpose LLM replace a fine-tuned invoice model on claims documents?
Often yes for narrative fields, with caveats. Prompt-based extraction without fine-tuning already reaches an F1 of 80.9 on CORD and 83.9 on SROIE, below specialised models on receipts but far more portable across unseen layouts. For claims files, that portability matters more than the last few points on a receipt benchmark. Keep grounding and per-field thresholds regardless.

### Is fine-tuning on expert reports worth it?
Only after you have a labelled evaluation set and know which fields fail. Fine-tuning helps on recurring sub-formats (one insurer's standard estimate form, one network's repair quotes) and helps little on free prose. VRDU's few-shot results show a 15+ point gap between 10 and 50 training samples, so that gap persists across all tasks and all three models tested: the first 50 examples buy the most.

### Should we build or buy for claims extraction?
Build if your document mix is narrow, stable, and your fields are flat. Buy if you face long-tail document types with nested fields, since that is where in-house models decay fastest and where maintenance cost, not initial accuracy, dominates. The trade-offs are laid out in [build vs buy for IDP](https://sygnet.ai/build-vs-buy-idp).