Why does AI sometimes make things up, and how do I catch it in time?

A couple of months ago a developer on the team spent three hours looking for a JavaScript library method that, according to ChatGPT, existed. It didn’t. It sounded so plausible — something like Array.prototype.groupByAsync — that nobody checked it until the build broke. That’s a hallucination: the AI doesn’t say “I don’t know,” it makes up an answer with the same confidence as if it were true.

And it’s not a rare bug that only happens to a mediocre model. It happens to all of them, including the good ones. A language model doesn’t “know” things the way a database does: it predicts the most likely next word based on what it saw during training, and when the topic is very specific or poorly documented, that prediction can sound perfect on the outside and be empty on the inside.

What we’ve learned at the agency, more than distrusting AI as a whole, is distrusting it selectively. When I ask a model for a verifiable fact — a library version, the exact name of a method, a figure — I treat it as a hypothesis, not a fact. If I can check it in the official documentation in 30 seconds, I check it. If I can’t, it doesn’t go into the client’s code. Full stop.

There are patterns that help catch it before it’s too late. The more niche or recent the topic, the more likely the invention: a library released two months ago is fertile ground for the model to fill gaps with something plausible. If you ask the model for the exact source of what it just claimed and the answer is vague, or changes between attempts, that’s a bad sign. And when it generates code, you run it, always. A broken import gets caught in two seconds by running the project; eyeballed in a quick review, it slips through without anyone noticing.

With non-technical clients this is trickier, because many assume that if the answer “comes from AI” and sounds coherent, it’s correct. I explain to them that a customer service chatbot that makes up a price or a returns policy isn’t a cosmetic glitch, it’s a real legal and trust problem. That’s why when we build something like that we almost always tie it to the company’s real data (with document retrieval or heavily restricted prompts) instead of letting the model “remember” the information from training.

What I still haven’t fully figured out is how to explain this risk without the client ending up distrusting the whole tool. Saying “it can make things up” sounds like it doesn’t work, and that’s not it: it works very well for almost everything, except for the data it can’t afford to get wrong. That nuance is the hardest part to get across in a sales meeting, and I still haven’t found the sentence that sums it up without sounding like a legal disclaimer.