How to check ChatGPT's answers against your PDF
ChatGPT can misquote, round or invent figures from a PDF. A practical method to check every answer against its source page, plus prompts that help.
Key takeaways
- A reference to a file is not a reference to a passage. Check that the exact wording exists on the page the answer relies on.
- Ask for quotations, not summaries. A quote can be found and checked; a paraphrase cannot.
- Recompute every number the model derived, using inputs you can see in the source.
- Ask the model to say "not in the document" when it cannot find something, and treat that as a valid answer.
ChatGPT is good at reading a long PDF and answering questions about it. It is also capable of returning a figure from the wrong year, rounding a number that should not be rounded, or stating something the document never says, all in the same confident tone. If the answer is going into a memo, a model or a client email, someone has to check it.
This guide shows a method that works for any PDF, from a 10-K to a lease, and the prompts that make the checking fast.
Why ChatGPT gets PDF answers wrong
Most errors are not random. They come from a few predictable causes:
- Long documents are read in pieces. The model may not see the one paragraph that qualifies the number it reports.
- Tables lose their structure. When a table becomes text, a value can drift into the wrong row or column, especially across page breaks.
- The same metric appears more than once. Revenue for a segment and revenue for the company, or this year and last year, sit close together.
- Units are easy to drop. "In millions" usually appears once, at the top of a table.
- The model fills gaps. Asked for a figure the document does not disclose, it may produce a plausible one instead of saying so.
Citations help only when they point to a passage you can see. "Source: annual-report.pdf" tells you which file to search, not whether the answer is in it.
A five-step method for checking an answer
- Ask for the exact wording. Request the sentence or table row each answer relies on, quoted word for word, with the page.
- Find every quote in the PDF. Search for a distinctive part of the quotation. If it is not there, the answer is not supported, however reasonable it sounds. Watch for hyphenation, line breaks and footnote markers, which can make a real quote hard to find.
- Check the context around the quote. Is it the right period, the right entity, the right unit? Read the table heading and the sentence before.
- Recompute anything derived. Growth rates, ratios and totals should be recalculated from inputs you have found in the document.
- Record what you checked. Note the page, the passage and who checked it. It takes seconds now and saves the whole exercise later.
Prompts that make checking easier
- "Answer each question with the exact sentence or table row you used, quoted verbatim, and the PDF page number."
- "If the document does not state something, answer 'not in the document'. Do not estimate."
- "For any figure you calculate, show the formula and quote each input."
Mistakes to look for
| Error | What it looks like | How to catch it |
|---|---|---|
| Wrong period | Last year's figure reported as this year's | Read the column heading of the quoted table |
| Wrong unit | $416 million instead of $416 billion | Find the "in millions" or "in thousands" note |
| Wrong scope | A segment figure presented as the company total | Check the row label and the table title |
| Paraphrase drift | "may terminate" summarised as "will terminate" | Compare the quote with the claim word for word |
| Invented figure | A number for a metric the document never discloses | The quote cannot be found anywhere |
| Wrong document | An answer drawn from an older draft or another file | Confirm the file name on the cited page |
Checking answers inside ChatGPT with CiteLLM
Doing the five steps by hand works, but it is slow for twenty questions or a 200-page document. CiteLLM for ChatGPT does the finding for you. Attach the PDF and ask your questions. Each answer comes back with the passage it relies on highlighted on the original page, and a quotation that cannot be found in the PDF is never highlighted.
- Each answer is marked supported, calculated, contradicted or not in the document.
- Calculated figures show the formula, with every input highlighted where it appears.
- You approve or flag each answer, and can download an evidence report with your decisions for the file.
A highlight shows where a passage is, not that a conclusion is right. The judgment stays with you, which is why the review records your decision on each answer.
Frequently asked questions
Can ChatGPT cite page numbers from a PDF?
It can give page numbers, but they are not always right, and they may refer to the printed page number rather than the position in the PDF file. Always open the page and find the quoted text before relying on it.
Does this work with scanned PDFs?
Only if the PDF has a text layer. Scanned pages need text recognition (OCR) first, otherwise there is no text to quote or search.
Is it safe to upload confidential PDFs?
That depends on your ChatGPT plan and your firm's policy. CiteLLM deletes PDFs 30 minutes after upload by default and never uses documents to train models; see the privacy policy.