How to check an AI summary of a contract
An AI can summarize a forty-page contract in seconds, and the summary reads well whether it's right or not. Here is a method to check it against the original without reading everything again.
Where AI summaries go wrong
Language models write fluent text, and a wrong amount reads exactly like a right one. The mistakes that matter most in a contract summary tend to be of four kinds.
- Wrong figures. An amount, a percentage or a cap that differs from the contract, or a number that appears in the contract but refers to something else.
- Dates and durations. “Five years” when the contract only gives a start date and an end date, or a notice period that changed between drafts.
- Missing conditions. A payment or a penalty that the contract only applies in some cases, presented as if it always applied.
- Terms that aren't there. Automatic renewal, exclusivity or a guarantee that the summary mentions and the contract never says.
A method that works
- Make sure the contract has text. If it's a scan, recognize the text first, or you won't be able to search it.
- Copy the summary and split it into short claims, one fact per line.
- Mark the claims that contain a figure, a date, a period or a condition. Those are the ones to check first.
- For each one, search the contract for the number and for the key word. Numbers are often written both ways, like “sixty (60) days”.
- Read the whole clause, not just the line. Look for “provided that”, “except”, “unless” and “up to”.
- Mark each claim as supported, contradicted or not found. A claim with nothing behind it is not a fact.
aval Verifica is in development
Leave your email and we'll let you know when you can try it.
An example
A summary says: “The agreement runs for five years. Either party can terminate with 60 days' notice. Liability is capped at $5,000,000. It renews automatically every year.” Checked against the contract, it looks like this.
| Claim in the summary | What the contract says | Result |
|---|---|---|
| Runs for five years | Clause 3: from April 1, 2024 to March 31, 2029. | Matches, worked out from the two dates |
| 60 days' notice | Clause 12.2: sixty (60) days' prior written notice. | Matches |
| Liability capped at $5,000,000 | Clause 14: capped at the fees paid in the preceding twelve months. | Contradicts |
| Renews automatically | No clause mentions renewal. | Not found |
Two of the four claims hold. The other two are the kind of error that changes a decision.
Asking the AI to cite clauses
It helps to ask the model to quote the clause behind each point. It doesn't replace checking: models sometimes quote text that isn't in the document, or quote the right clause and still summarize it wrong. Search for each quote in the original before trusting it.
aval Verifica
aval Verifica does this check automatically. You paste the contract and the summary, and it looks up every figure, date and term, cites the clause, and tells you which claims match, which contradict the contract and which aren't there. It's in development; you can leave your email to try it when it opens.
Neither this method nor Verifica replaces reading the contract or legal advice. They tell you where to look.
Questions
Are AI summaries of contracts reliable?
They are useful to get an overview, but they can get figures, dates and conditions wrong, and sometimes mention terms the contract doesn't contain. Check every figure, date and term against the original before relying on it.
What should I check first?
Amounts, caps and percentages; dates and durations; and any condition words such as “provided that”, “except” or “up to”.
Is checking a summary the same as legal advice?
No. It tells you whether the summary matches the text. Whether a clause is fair or enforceable is a question for a lawyer.