In plain English
AI chat tools are built on large language models, which write by predicting which words are likely to come next. Most of the time that produces useful, accurate text. But the model is aiming for an answer that reads well, not checking facts, so when it doesn’t know something it can fill the gap with something that sounds right.
Hallucinations are more likely when you ask about things the tool hasn’t been given, such as your own clients or recent events, or when you want precise details like dates, references, sums or quotes. They’re less likely when the tool answers from a document you’ve supplied, which is why answering from your own files (retrieval-augmented generation) helps. Less likely isn’t never.
The practical rule: treat AI output as a first draft from a quick but occasionally careless assistant.
An example from a small business
Imagine a hypothetical surveying practice using an AI tool to summarise a 40-page tender document. The summary is clear and well organised, but it gives a completion date that appears nowhere in the tender. The tool has filled a gap with a plausible guess. Because the practice’s rule is that every date and figure in a summary is checked against the source before it’s used, a junior surveyor catches it in a couple of minutes.
What to watch out for
- Check names, numbers, dates and references. These are the details AI most often gets wrong, and the ones that cause real trouble.
- Ask for sources, then open them. Tools that show where an answer came from, as Copilot does with links to the files or web pages it used, make checking much quicker. A source can still be misread, so click through.
- Don’t treat a chatbot as the authority. Legal, tax and safety questions need a proper source, not a fluent answer.
- Build the check into automations. Anything the AI isn’t sure about should go to a person before it’s sent or entered (see human in the loop).
Related terms
- Large language model (LLM): the technology that hallucinates, and why.
- Retrieval-augmented generation (RAG): answering from your documents to cut errors.
- Human in the loop: the person who catches what slips through.
Where AlphaOps can help
Every automation we build is tested on your own documents first, and anything unclear goes to a person, so errors get caught rather than entered. See document processing for an example, or book a free automation audit.