How to Catch AI Hallucinations in ChatGPT and Claude

Key takeaways

Most hallucinations in document work are not wild inventions. They are real-looking numbers, quotes and clauses that are slightly wrong.
Asking for sources does not fix it. A citation is generated too, and it can point to the wrong page or quote words the document never uses.
Check one thing first: does the quoted wording exist on the page? If it does not, the answer is unsupported, however reasonable it sounds.
Separate "contradicted" from "not in the document". One is an error to correct, the other is a gap to fill from another source.

A hallucination is an answer that sounds right and is not supported by anything. In open chat that might be a made-up book or a fake statistic. When you attach a contract, a 10-K or a policy to ChatGPT or Claude, it is subtler: a figure from the wrong year, a clause the agreement does not contain, a quotation that is almost word for word but not quite.

These errors arrive in the same fluent, confident tone as the correct answers, which is what makes them dangerous. This guide covers what they look like, why asking for citations is not enough, and a short test that catches them before they reach a memo, a model or a client.

What a hallucination looks like

When the assistant has the document in front of it, hallucinations fall into a handful of patterns:

Kind	Example	Why it happens
Invented figure	An EBITDA margin for a company that never discloses EBITDA	The model fills a gap instead of saying the figure is missing
Wrong period or unit	Last year's revenue reported as this year's, or millions read as thousands	Columns and unit notes get separated when a table becomes text
Wrong scope	A segment figure presented as the company total	The same metric appears several times close together
Near-quote	"The tenant shall maintain" quoted as "the landlord shall maintain"	Quotations are generated from memory of the text, not copied from it
Stronger wording	"May terminate" summarised as "will terminate"	Summaries smooth out the qualifiers that matter
Invented clause	A change-of-control right the agreement does not contain	Common contract terms are assumed to be present
Wrong location	A real passage cited to page 14 when it is on page 41	Page references are produced the same way as the rest of the answer

Notice that most of these would pass a quick read. They are plausible, specific and close to something in the document. That is why reading the answer again does not catch them; only going back to the source does.

Why asking for sources is not enough

The natural fix is to ask the assistant to cite its sources. It helps, because a cited answer can be checked. It does not make the answer correct:

A citation is part of the answer. The model writes the page number and the quotation the same way it writes everything else, so they can be wrong in the same ways.
A file is not a passage. "Source: credit-agreement.pdf" tells you where to search, not whether the answer is there.
Page numbers are ambiguous. The model may give the printed page number, which is often not the page of the PDF file you open.
A real quote can still be misused. The sentence exists, but it describes a different period, a different entity or an exception to the rule.

Lawyers have learned this the hard way: courts have sanctioned filings that cited cases and quotations an AI tool invented. Our guide to cite-checking AI-drafted briefs covers that case in detail.

Which answers to check first

You rarely have time to check every sentence. Start where an error would cost the most, and where hallucinations are most likely:

Numbers that drive a decision: prices, rates, totals, dates and deadlines.
Anything calculated: growth rates, ratios, sums and date arithmetic, where one wrong input changes the result.
Negative answers: "the contract has no termination right" is easy to say and hard to prove.
Answers drawn from tables and footnotes, where structure is lost and qualifiers hide.
Answers that confirm what you hoped. They get the least scrutiny, so give them the most.

A four-check test for any answer

This works the same in ChatGPT and Claude, and for any PDF:

Does the quote exist? Search the PDF for a distinctive part of the quoted wording. If you cannot find it, stop: the answer is unsupported. Allow for line breaks, hyphenation and footnote markers before you conclude it is missing.
Is it the right place? Read the table heading, the column and the sentence before. Confirm the period, the entity and the unit.
Does it say what the answer says? Compare the claim with the quote word for word. Watch for "may" becoming "will", or an exception becoming the rule.
Does the arithmetic hold? Recompute anything derived from inputs you have found yourself, not from inputs the model restated.

Then note the page, the passage and your decision. If someone asks later how you know, the answer is already written down.

Prompts that make the test faster

"For each answer, quote the exact sentence or table row you relied on, word for word, with the PDF page number."
"If the document does not say, answer 'not in the document'. Do not estimate or assume."
"For any figure you calculate, show the formula and quote each input."

Unsupported is not the same as wrong

When an answer fails the test, it matters how it fails. There are three outcomes worth keeping apart:

SupportedThe passage exists and says what the answer says.
ContradictedThe document says something different. This is an error to correct before anyone relies on it.
Not in the documentThe document does not address it. This is not a failure; it is a gap to fill from another source or to raise as a question.

Treating "not in the document" as a valid answer, and telling the assistant so, is one of the most effective ways to reduce hallucinations. Much invention happens because a model is asked for something the document does not contain and tries to be helpful.

Checking answers inside ChatGPT and Claude

The four checks are quick for one answer and slow for fifty. CiteLLM for ChatGPT and CiteLLM for Claude do the finding inside the chat you already use. Attach a PDF and ask. Each answer opens in a viewer at the right page with its passage highlighted, and a quotation that cannot be found in the PDF is never highlighted.

Each answer is marked supported, calculated, contradicted or not in the document.
Calculated figures show the formula, with every input highlighted where it appears.
Up to 20 PDFs and 100 answers per PDF fit in one review.
You approve or flag each answer, and can download a highlighted PDF or an evidence report with your decisions.

In ChatGPT, connect CiteLLM from the app directory and choose it when you write your message. You can try it without signing in. In Claude, add CiteLLM from the directory, and mention CiteLLM in your request so Claude uses it for the PDF. The ChatGPT and Claude setup guides have the details, and What to ask has example requests by profession.

CiteLLM confirms where each quotation appears. Whether the passage supports the answer is the assistant's assessment, and the final judgment is yours, which is why the review records your decision on each answer.

Frequently asked questions

Can you stop ChatGPT or Claude from hallucinating?

Not completely. Attaching the source document, asking for verbatim quotations and allowing "not in the document" as an answer all reduce hallucinations, but no prompt removes them. An answer that matters still needs to be checked against the page it relies on.

Is Claude less likely to hallucinate than ChatGPT?

Error rates vary by model version, document and question, and both assistants can quote a passage that is not in the file. The safe assumption is that either one can be wrong, and the checking method is the same for both.

Does a citation mean the answer is right?

No. A citation is generated text like the rest of the answer. It can name the right file and the wrong page, or quote wording that does not appear anywhere. Open the page and find the quoted text before relying on it.

Does this work with scanned PDFs?

Only if the PDF has a text layer. Scanned pages need text recognition (OCR) first, otherwise there is no text to quote or search.