Contract Clause Extraction: Accuracy, Recall, and Risks

Key takeaways

• On the CUAD benchmark, the best model tested by the authors reached only 44.0% precision at 80% recall, meaning a tool tuned to catch 4 out of 5 clauses buries every true hit under roughly one false positive (CUAD paper).
• Change-of-control is empirically one of the hardest clause types to extract, it ranked among the six worst-performing categories for a DeBERTa baseline (Bhattacharya et al.).
• For revenue-risk audits the correct target is recall ≥ 95% on consent-triggering clauses, not overall accuracy: a 90%-recall tool on a 600-contract data room silently drops roughly 12 of 120 clauses.
• Classification is not extraction: frontier LLMs score ≥88% balanced accuracy on short CUAD clause questions but drop to 47.8% on MAUD merger-agreement questions (LegalBench).

How many missed contracts does it take for a change-of-control clause to cost you the deal?

One. A single unflagged consent requirement in a top-10 customer contract can force a post-signing renegotiation, a price adjustment, or an indemnity claim, which is why practitioners describe the scenario bluntly: a single overlooked change-of-control provision buried in a vendor agreement can stall a $200M acquisition. The relevant benchmark question is therefore not "is the tool accurate?" but "what is its recall on the specific clause families that trigger consent, and how much revenue sits behind each missed contract?"

Make the arithmetic explicit before you buy anything. Take a 600-contract data room where 20% of agreements carry an assignment or change-of-control trigger, 120 clauses at risk. At 99% recall you miss 1. At 95% you miss 6. At 90% you miss 12. At the 80% recall level used as a reporting standard in the CUAD literature, you miss 24. Now weight each miss by annual contract value. If the average flagged contract carries €400k of ARR and a 30% probability the counterparty uses the trigger to terminate or reprice, 12 misses represent roughly €1.4m of expected revenue exposure, often more than the entire diligence budget.

What does the published research actually say about clause extraction accuracy?

Published benchmarks show clause extraction is far from solved, which is why vendor "99% accuracy" claims deserve interrogation. CUAD comprises over 510 commercial legal contracts annotated with 13,000+ expert labels across 41 categories relevant to contract review. On it, the authors report Area Under the Precision-Recall curve, Precision at 80% Recall and Precision at 90% Recall, with DeBERTa-xlarge achieving the best 44.0% Precision @ 80% Recall, substantially better than BERT-base at 8.2%. RoBERTa-base scored 42.6% AUPR versus 48.2% for RoBERTa-large.

Two findings matter for M&A buyers. First, precision and recall trade off directly: as recall increases, precision decreases, so Precision @ 90% Recall is lower than Precision @ 80% Recall. A high-recall configuration is supposed to generate noise, that is the cost of not missing clauses. Second, model size is not the lever people assume. ALBERT-xxlarge-v2 has more than 20 times the parameters of its smallest version yet performs only around 3% better.

Why is change-of-control specifically harder than other clauses?

Because change-of-control is defined by economic effect rather than by a fixed phrase, and it frequently hides inside an anti-assignment paragraph rather than under its own heading. Empirically, termination- and change-of-control-related clause types are among the toughest to identify, with three of four such types appearing in the top six worst-performing clause types for a DeBERTa benchmark.

The drafting reality explains it. Some anti-assignment provisions include prohibitions against change of control without prior written consent, for example deeming any change of control resulting from an amalgamation, reorganisation, arrangement or business sale to be an assignment, so a share purchase requires prior written consent. There is no "Change of Control" header to anchor on. Thresholds also vary: common triggers include a change in ownership of more than 50% of the equity of the contracting party (the most common), any merger, consolidation or sale of substantially all assets, or a change in beneficial ownership exceeding a threshold amount. A retrieval pipeline that scores semantic similarity to a canonical clause will rank the plain-vanilla version highly and demote the deemed-assignment variant, the exact one that kills stock deals. Retrieval quality here is a function of chunking and embeddings choices, not just model size.

How should you benchmark an extraction tool before a live data room?

Run a gold-set bake-off on 80–150 of the target's own contracts, annotated by your own counsel, and score recall per clause family rather than one global accuracy figure. Vendor demos are almost always run on clean, digital-native agreements; real data rooms contain scans, amendments and side letters, and text recognition has to handle scanned legacy contracts and mixed PDF bundles before extraction even begins.

Score five things: (1) recall on assignment/change-of-control/termination clauses; (2) precision at that recall level; (3) whether every extraction returns a page-and-span citation, a jump-to-source link back to the corresponding section of the original contract is the minimum bar for auditability; (4) behaviour on amendments that supersede the base clause; (5) calibration, i.e. whether the confidence score actually predicts error. Also test the failure mode that legal reviewers fear most: a fluent, plausible clause summary with no textual basis, the classic LLM hallucination pattern.

Where does human review still have to sit in the workflow?

On every flagged clause and on a sampled slice of the unflagged set, automation should compress reading time, not replace the consent decision. The economics are well documented: AI flags change-of-control provisions, assignment restrictions and termination rights across entire agreement sets simultaneously, and a senior lawyer who previously spent two days reviewing supply agreements now reviews the flagged list in two hours. Practitioners report first-pass timelines shrinking from 2–3 weeks of associate time to 24–48 hours with AI plus attorney review.

But interpretation stays human, because materiality is commercial, not textual: a flagged change-of-control clause may be standard in the target's sector and immaterial to the deal structure. And detection alone does not settle the legal question, anti-assignment and change-of-control provisions are fairly ubiquitous in commercial contracts, but when consent is actually triggered depends on the proposed transaction's structure. A human-in-the-loop design with routed exceptions is the only configuration that converts high recall into a defensible consent schedule. Sygnet builds document-extraction pipelines of this type, including contract clause analysis with span-level source citations.

Which numbers should go into the audit report itself?

Report clause-level coverage, not tool accuracy. A usable revenue-risk deliverable states: number of contracts ingested, number classified as material by ARR, number containing assignment or change-of-control language, number requiring affirmative consent, ARR behind each, and the measured recall of the extraction pass on your gold set. Anti-assignment prevalence is high enough that the consent schedule usually drives closing mechanics, one M&A survey cited by practitioners found 83% of target company material contracts contained anti-assignment provisions requiring consent gathering as part of the closing process, though prevalence varies sharply by sector and contract vintage, so measure it on the actual portfolio.

State residual risk numerically. If you ran 95% measured recall over 600 contracts, write that approximately 6 triggering clauses are statistically expected to remain undetected, and name the mitigation, a seller representation plus a specific indemnity. That framing survives an audit committee; "we used AI" does not.

FAQ

What recall should I require from an AI contract review tool for M&A diligence?

Require ≥95% recall on the consent-triggering clause families, measured on your own annotated sample, with 100% human review of everything flagged. Accept the resulting false-positive load: CUAD results show precision decreases as recall increases, so a quiet, high-precision tool is usually a tool that is missing clauses. Below 90% recall, treat output as triage only.

Can a general-purpose LLM handle change-of-control extraction without fine-tuning?

Partly. Averaged across the 38 CUAD clause tasks, GPT-4, GPT-3.5 and Claude-1 all reached balanced accuracy of at least 88%, but performance degrades on tasks with longer text sequences or multi-class classification. Those tasks judge a clause already isolated for the model. Finding the clause in a 7,000-word scanned agreement, plus its amendments, is a retrieval and parsing problem, which is why pipeline design matters as much as model choice.

Is a stock sale safer than an asset sale for contract continuity?

Not automatically. Asset sales generally require individual contract assignment, while stock sales preserve contracts by default but can trigger change-of-control language. The audit therefore has to score both clause families separately, because for businesses with heavy contract counts, the structure choice often comes down to which set of clauses is less burdensome.

How do I know whether to build this pipeline or buy one?

Compare your annotation capacity against vendor cost, since data, not model size, is the binding constraint: CUAD's authors note data is a large bottleneck for contract review in this regime. If you run fewer than a handful of deals a year, a bought tool with span-level citations wins; at portfolio scale, the build vs buy trade-off shifts toward an internal gold set you can re-benchmark each deal.