Free Agent Skill · released
source-check: a free Agent Skill that checks AI answers against the source document
When an AI model summarizes a long report, the mistakes that matter are quiet ones. It takes a number from the right table but the wrong column, calls preliminary figures final, or adds a forecast the document never made. source-check is an open-source Agent Skill that checks an answer against the document it came from. Every claim gets a verdict, a page number and a quote from the document, and a script confirms each quote is really on that page before you see the table.
Short answer
npx skills add TianyiDataScience/source-check
- Works in Claude Code, Codex, Cursor and other agents that read
SKILL.mdskills. - Ask "Check this summary against Q3-report.pdf", or "Is everything in your last answer actually in the contract?"
- You get a table: each claim marked ✅ supported, ❌ contradicted, ⚠️ not in source or ❔ can't check, with the page and the quote, then a list of what to fix.
- Free and open source (MIT) on GitHub. No account and no API key of its own.
What a check looks like
The repository comes with a short fictional water utility report and an AI-style summary of it with planted mistakes. Five of the eleven rows the skill returned:
| Claim in the summary | Verdict | Page | Evidence |
|---|---|---|---|
| “demand reached 42.9 million gallons in September, seasonally adjusted” | ❌ Contradicted | 2 | “Before seasonal adjustment, September demand was 42.9 million gallons.” Seasonally adjusted, it was 41.2. |
| “up 3.5% from August” | ✅ Supported | 2 | 41.2 / 39.8 = +3.5%, for the adjusted figures |
| “The Board raised the base rate to $4.25 per 1,000 gallons” | ❌ Contradicted | 2 | “The Board left the base rate at $4.15 per 1,000 gallons.” |
| “The Authority expects demand to ease in October” | ⚠️ Not in source | 2 | “The Authority does not publish demand forecasts in this report.” |
| “September figures are final” | ❌ Contradicted | 3 | “Figures for September are preliminary.” |
The full check has all eleven claims, the arithmetic behind each derived number and the list of fixes. Ask the agent for a corrected answer and it changes only the flagged sentences, with the page next to each change.
On a real 39-page report
We gave three models the BLS Employment Situation report for September 2026 and the same prompt: summarize it in 10 bullet points for a reader who won't open it, with the key numbers. Then source-check checked every answer against the PDF.
| Model | Claims | Supported | Contradicted | Not in source |
|---|---|---|---|---|
| Claude Sonnet 5.5 | 51 | 51 | 0 | 0 |
| GPT-6.1 Sol | 47 | 46 | 0 | 1 |
| Gemini 3.1 Pro Preview | 28 | 27 | 0 | 1 |
Every number was right. The two flags are sentences that describe the report in words it doesn't use:
- GPT-6.1 Sol wrote that the household survey found 406,000 more employed people, "though the report characterized the change as small." The 406,000 is in Summary table A on page 4, but the text never describes it. The nearest words, "changed little" on page 2, are about the participation rate and the employment-population ratio.
- Gemini titled a bullet "Healthcare Leads Job Gains". Health care did add 17,000 jobs, but the report doesn't rank industries and never says it led.
The answers, the three check tables and how the run was done are in the example folder. We also asked the same three models three questions about this report on our test page, and checked those answers the same way.
How it works
- A script turns the document into text with a marker before each page, so every quote can carry the page number a PDF viewer shows.
- The agent splits the answer into claims of one number, date, name or comparison each, and keeps qualifiers such as "seasonally adjusted" or "preliminary" with them.
- A search script lists every page where a number or phrase appears. It matches 40367 to 40,367 but never to 140,367, so long documents don't have to be read in full.
- Each claim gets one verdict. Changes, ratios and margins are recomputed from the document's own figures, with the arithmetic in the note.
- The agent writes its quotes to a file, and a script looks for each one on the cited page. A quote on another page, a table cell from another row or a quote that isn't there has to be fixed before the table goes out.
- The agent reports the totals, the table and what to fix.
The rules the agent follows are in SKILL.md, written in the Agent Skills format.
Install
With the skills CLI:
npx skills add TianyiDataScience/source-check
From the Claude Code plugin marketplace:
/plugin marketplace add TianyiDataScience/source-check /plugin install source-check@source-check
By hand, for Claude Code:
git clone https://github.com/TianyiDataScience/source-check.git cp -r source-check/skills/source-check ~/.claude/skills/
For Codex, copy the same folder to ~/.codex/skills/. The scripts need Python 3.8 or later. For PDFs they use poppler's pdftotext or the pypdf package, and on a Mac with neither, the PDFKit framework built into macOS. Text, Markdown, HTML and .docx files need nothing else.
Limits
- It checks the answer against the document, not against the world. If the document is wrong, a supported verdict is wrong too.
- Charts and images are checked only when the agent can see the page image. Scanned PDFs need OCR first.
- The checking agent can still misjudge a claim. Every verdict cites a page and a quote, so you can check the checker.
FAQ
Which AI agents can use source-check?
Any agent that reads SKILL.md skills, including Claude Code, Codex and Cursor. Install it with npx skills add TianyiDataScience/source-check, or copy the skills/source-check folder into your agent’s skills folder.
Does source-check search the web?
No. The only evidence is the document you give it. A claim that is true in the world but missing from the document is marked not in source, which is what you need to know before you repeat it.
Does it need an API key or an account?
No. It runs inside the agent you already use, with that agent’s model. Its scripts need Python 3.8 or later, plus pdftotext or pypdf for PDFs. On a Mac with neither, it uses the PDFKit framework built into macOS.
Can it check scanned PDFs or charts?
Not on their own. The extraction step lists pages with no text layer, and claims that depend on them get can’t check unless the agent can read the page image. Run OCR on scanned documents first.
How long does a check take?
A few minutes for a short answer and a short document. Three summaries of a 39-page jobs report, 126 claims in all, took about 25 minutes together. On long documents the agent searches first and reads only the pages it needs.
Is source-check free?
Yes. It is open source under the MIT license, and the code, tests and examples are on GitHub.
Who made it
We make Creatos, a desktop AI canvas where one PDF, video or web page sits next to several models' answers. On our model tests every answer is checked against the page it came from. source-check is that checking step, written down so any agent can do it. Issues and pull requests are welcome on GitHub, especially documents where it gets a verdict wrong.