LLM hallucination

LLM hallucination is the tendency of a large language model to produce an answer that sounds plausible but has no basis in the source document. In document extraction, this usually means the model returns a value (a date, an amount, a clause) that was never written on the page, filled in from pattern memory rather than read from the text. The output looks clean and confident, which is exactly what makes it dangerous in a business workflow.

How it works

A language model generates text one token at a time, choosing the most statistically likely continuation given everything it has seen so far. It has no built-in concept of "I don't know." When a document is blurry, a field is missing, or a layout confuses the model's reading order, it still has to produce something. So it draws on patterns learned during training and fills the gap with a value that fits the expected shape: a plausible invoice number, a reasonable-looking total, a contract date that matches the format of the rest of the page.

This differs from a simple OCR error, where a character is misread but the source location is real. A hallucination has no anchor at all. The model may report a VAT amount that doesn't exist anywhere in the file, or misattribute a clause to the wrong party because that pairing appeared often in training data.

Extraction systems built on Vision-Language Model architectures are not immune to this, since the language-generation component still sits downstream of visual reading. The risk grows with longer documents, ambiguous layouts, and fields that require arithmetic (totals, subtotals, tax calculations) rather than direct copy. It also grows when a schema asks for a field that simply is not present in the document: a well-behaved system should say "not found," but a poorly constrained model will often invent one anyway.

Why it matters for document processing

Hallucinated fields break trust in automation precisely where automation is supposed to save time. An invoice total that's slightly wrong, a policy number pulled from thin air, a contract renewal date that doesn't match the PDF: these errors pass silent checks and land in downstream systems (ERP, claims, KYC records) as if they were verified fact. Unlike a system crash, nobody notices until reconciliation fails or an auditor asks a question.

The fix is architectural, not just prompt engineering. Extraction should be grounded: every field tied back to a bounding box on the source page, every value cross-checked against document layout, and every low-confidence score routed to a human reviewer instead of auto-approved. This is why straight-through processing targets should always be paired with visible confidence thresholds, and why regulated workflows like KYC onboarding or insurance claims need human-in-the-loop review on anything the system can't point to on the page.

Related terms

• Confidence score
• Human-in-the-loop (HITL)
• Bounding box
• Vision-Language Model (VLM)
• Zero-shot extraction

FAQ

Can hallucination be eliminated entirely?

No, not with current model architectures. It can be reduced sharply through grounding techniques (linking every extracted value to a specific location in the source document), schema validation, and confidence thresholds that route uncertain fields to human review instead of accepting them automatically. Treat hallucination as a risk to manage continuously, not a bug to patch once.

How is hallucination different from an OCR misread?

An OCR misread has a real anchor: the character exists on the page but was read incorrectly, often due to poor scan quality. A hallucination has no anchor at all. The model produces a value that isn't written anywhere in the document, generated from learned patterns rather than the actual page content. Grounded systems can flag the second case; OCR alone cannot.