AI chatbots are getting smarter, but can you trust them to fact-check your work? We put ChatGPT, Claude, and Gemini through a series of fact-checking tests to see how they handle verifying claims, catching errors, and distinguishing truth from fiction.
The results were more nuanced than you might expect.
What We Tested
We gave each model the same set of tasks:
- Verifying historical dates and events
- Checking statistics and numerical claims
- Identifying false information in mixed true/false paragraphs
- Confirming current company leadership and organizational details
- Evaluating scientific claims with citations
We used the latest versions available: ChatGPT-4o, Claude 3.5 Sonnet, and Gemini Advanced. Each test included both straightforward facts and deliberately tricky misinformation designed to catch models that over-rely on pattern matching.
Where AI Fact-Checking Works Well
All three models excelled at catching obvious factual errors. When we presented a paragraph claiming the Eiffel Tower was built in 1899 (it opened in 1889), or that water boils at 90°C at sea level, every model caught it immediately and provided the correct information.
They were also surprisingly good at identifying numerical inconsistencies within documents. When we gave them a report where the percentages didn’t add up to 100%, or where totals contradicted itemized figures, all three flagged the errors without prompting.
Claude performed particularly well on scientific claims, often providing context about why a statement might be misleading even if technically true. For example, when presented with a correlation study claiming causation, Claude explained the distinction clearly.
Where AI Gets It Wrong
The trouble starts with recent events and current information. When we asked models to verify claims about company announcements from the past six months, accuracy dropped significantly. ChatGPT with search capability performed better here, but still produced confident-sounding answers that were outdated or incomplete.
All three models struggled with claims that sounded plausible but were completely fabricated. When we included a fake statistic attributed to a real organization (“according to the CDC, 42% of adults experience seasonal allergies”), the models often accepted it at face value unless we specifically asked them to verify the source.
More concerning: each model occasionally “hallucinated” citations when asked to verify information. They’d respond with confidence, sometimes inventing plausible-sounding study names or publication details that didn’t exist.
When to Trust AI Fact-Checking (and When Not To)
AI can be a useful first pass for fact-checking, but it shouldn’t be your only line of defense. Here’s what we learned:
Good use cases:
- Catching obvious errors in dates, names, and well-established historical facts
- Spotting mathematical inconsistencies in data and reports
- Identifying logical contradictions within a single document
- Getting a quick sense of whether a claim is plausible enough to investigate further
Don’t rely on AI for:
- Verifying very recent events or current statistics
- Checking claims about specific studies or research without going to the original source
- Anything where being wrong has serious consequences (legal documents, medical information, financial data)
- Attributing quotes or checking citations—always verify these manually
The most effective approach we found: use AI to flag potential issues, then verify anything important through primary sources. Think of it as a smart assistant that can spot red flags, not as a replacement for actual research.
One practical tip: when fact-checking with AI, ask the model to express its confidence level and explain its reasoning. Responses like “I’m not certain, but…” or “This should be verified against…” are more trustworthy than absolute certainty.
Want to stay sharp on what AI can and can’t do? Subscribe to the One Two Three AI newsletter and get one practical AI insight delivered daily—no hype, just what actually works.