Hallucination

A hallucination is when an AI system produces something false and presents it as fact. Invented statistics, invented case citations, invented product features, invented people. The term rather undersells it, because it suggests a rare malfunction rather than a permanent property of how these systems work.

Why it happens

Large language models predict plausible continuations of text. They are not looking anything up. Given a question, the system produces what a good answer would probably look like, one piece at a time, based on patterns.

Most of the time a plausible answer is also a correct one, which is why these tools are useful. When it is not, nothing in the machinery notices. There is no internal flag distinguishing a fact it has seen a million times from a name it has just assembled because the sentence needed one.

That is the part worth understanding: the wrong answer arrives in exactly the same confident tone as the right one. There is no tell. No hedging, no hesitation, no change in register.

Why asking it to be accurate does not work

A common instinct is to add "only use real sources" or "be accurate" to the instruction. This helps marginally and does not solve the problem, because the system has no reliable way to distinguish what it knows from what it is constructing. Asking it to try harder is asking for something it cannot do.

Retrieval helps considerably more. If the system is given your actual documents and told to answer from those, and to cite which one, the failure mode shifts from invention to retrieving the wrong document, which is at least visible.

What actually controls it

A person who knows the subject checks the output. There is no substitute, and the rule worth setting is tied to consequence rather than volume: anything that leaves the building, goes to a client, or informs a decision gets checked.

Source citation. A system that shows where an answer came from lets someone verify in seconds. One that does not gives you no way to tell.

Narrow scope. A tool constrained to your own material fails far less often than one answering from general knowledge.

The question to ask a supplier

"What does your system do when it does not know?"

Good answers involve saying so, escalating to a person, or returning nothing. Answers involving confidence should worry you. Any supplier claiming their system does not hallucinate is either misinformed or hoping you are.