What Is Hallucination?
Written byFlavio Amiel Founder, Roborank
A hallucination is a fluent, confident statement produced by a large language model that is factually wrong, unsupported, or entirely fabricated, yet presented as if it were true. The model generates plausible text from statistical patterns rather than retrieving verified facts, so invented citations, dates, quotes, and case names can appear indistinguishable from real ones.
- Hallucinations arise because a language model predicts the next likely token from learned patterns, not from a database of checked facts — so fluency is no guarantee of accuracy.
- Anchoring an answer to a retrieved source reduces hallucination but never fully eliminates it; a model can still misread or over-generalize from a genuine source.
- In Mata v. Avianca (2023), a lawyer filed a brief full of fake cases invented by ChatGPT and was fined $5,000 by a federal judge on June 22, 2023.
- Benchmarks such as Google DeepMind’s FACTS Grounding exist specifically to measure how often a model stays anchored to provided facts rather than hallucinating.
How Hallucination Works
A large language model does not store facts the way a database does. It stores statistical relationships between words, learned from vast training text, and generates an answer by repeatedly predicting the most probable next token. That process is astonishingly good at producing fluent, well-formed prose. It has no built-in mechanism for checking whether the fluent prose is true. When the training data supports a claim strongly, the prediction tends to be correct; when it doesn’t, the model still produces a confident-sounding answer, because “I don’t know” is rarely the highest-probability continuation.
This is why hallucinations are hardest to catch. The invented material is not garbled — it is grammatical, specific, and formatted exactly like a real fact. A fabricated legal citation carries a plausible case name, volume number, and court. A made-up statistic arrives with a decimal point and a year. The surface signals a careful reader uses to gauge reliability are precisely the signals the model is best at imitating.
Retrieval-augmented generation is the main defense. By fetching real documents and asking the model to answer from them, the system can anchor claims to a grounding source rather than to memory alone. This lowers the hallucination rate substantially, but it does not reach zero: the model can still misread the retrieved text, blend two sources incorrectly, or extend a real fact past what the source actually says. Measuring that residual failure is the whole point of factuality benchmarks.
Google DeepMind’s FACTS Grounding benchmark, published December 17, 2024, exists precisely to quantify this. It hands a model a source document and a question, then scores whether the answer stays fully grounded in that document with no hallucinations, using a panel of frontier models as judges to reduce bias. The fact that even leading systems score in the low-to-mid eighties on such a task — not near-perfect — is the plainest evidence that hallucination is a property of how these models work, not a bug awaiting a quick patch.
Types of Hallucination
Practitioners usually separate hallucinations by where the answer diverges from reality:
- Factual hallucination — a claim about the world that is simply untrue, such as a wrong date, a nonexistent product feature, or an invented statistic.
- Fabricated attribution — a citation, quote, study, or source that does not exist, or a real source misquoted to say something it never said.
- Ungrounded extrapolation — the model is given a real source but states more than the source supports, filling gaps with plausible-sounding but unverified detail.
The first two are dangerous because they look authoritative; the third is dangerous because it hides inside an otherwise well-sourced answer.
Example of Hallucination
The most thoroughly documented hallucination case is Mata v. Avianca, Inc., decided in the U.S. District Court for the Southern District of New York in 2023. A lawyer representing Roberto Mata used ChatGPT to help draft a legal brief. The tool produced a motion citing several supporting decisions — cases with confident names, docket numbers, and internal quotations. None of them existed.
When opposing counsel and the court could not locate the cited cases, the lawyer asked ChatGPT to confirm they were real. The model doubled down: it insisted the cases were genuine and “can be found in reputable legal databases such as LexisNexis and Westlaw.” They could not, because the model had invented them wholesale, then invented reassurance about them.
On June 22, 2023, Judge P. Kevin Castel sanctioned the attorneys and imposed a $5,000 fine, and the case has since become the landmark reference courts cite when AI-fabricated law appears in filings. It is a clean illustration of every property that makes hallucination hazardous: the output was fluent, specifically formatted, internally consistent, and completely false — and the model expressed no less confidence about the fake cases than it would have about real ones.
The lesson generalizes well beyond courtrooms. An engine asked about your company, your pricing, or your track record will answer with the same fluent confidence whether or not it can retrieve real facts. The defense is not to hope the model behaves; it is to make the true version the easiest thing for it to find and quote.
The thing people get wrong is imagining hallucinations as random static — an occasional glitch that better models will iron out. In practice they cluster, and they cluster exactly where verifiable evidence is thin. Ask an engine for a specific statistic, a court citation, or a product spec that isn’t cleanly documented anywhere it can retrieve, and it will happily manufacture one, because the model’s job is to produce a plausible answer, not to admit a gap. That reframes hallucination from a model problem into a supply problem you can act on. If the facts about your brand, your data, and your claims exist as plain, checkable sentences an AI crawler can reach, the engine anchors to them; if they don’t, it fills the vacuum with an invention — and that invention is what your customers read.
Frequently Asked Questions
What is an AI hallucination?
Why do AI models hallucinate?
Can AI hallucinations be stopped?
How do hallucinations affect my brand in AI search?
The Bottom Line
A hallucination is the gap between how certain an AI sounds and how true it actually is: text generated to be plausible, not verified, so a fabricated fact wears the same confident voice as a real one. The fix is rarely a better model on its own — it is supplying clean, retrievable, checkable evidence so the engine has something true to anchor to instead of an empty space to fill.
Sources
Rank & Cash — the weekly SEO breakdown
One practical teardown a week on ranking in search and getting cited by AI. No fluff.