Why does an AI give me different answers to the exact same question?

A client wrote to me a few weeks ago, pretty annoyed: he’d asked ChatGPT to summarize the same document twice, with the exact same prompt, copied and pasted, and got two different summaries back. He thought the tool was broken. It isn’t. It works like that by design, and it took me a while to accept that too, the first time we built an integration with a model for a client and the results in testing just wouldn’t come out the same twice.

A normal program is deterministic: same input, same output, every time. A spreadsheet doesn’t have good days and bad days. A language model doesn’t work that way underneath. When it generates text, at every single step it computes a list of candidate words with a probability attached to each one, then picks one from among the most likely, not always the single most likely word, but one sampled at random from within that range. That’s controlled by a setting almost every API exposes called “temperature”: higher means more variety (and more risk of wandering off); lower, or set to zero on some models, means answers much closer to each other, though almost never a hundred percent identical. In one line: an AI gives different answers to the same question because it doesn’t always pick the single most likely word — it samples among several candidates based on that randomness setting, so some variation isn’t a bug, it’s part of the design.

Why build it this way, when it makes the tool feel unreliable? Because the same mechanism that makes two summaries come out different is what lets the model avoid sounding the same every time, lets it finish a sentence naturally instead of repeating the statistically flattest word combination over and over. Without that randomness a chatbot would sound like a machine, not like something imitating a real conversation.

The trouble shows up when someone, us included the first time, uses a model for a task that actually needs to be deterministic. We built an automation for a client that read invoices out of PDFs and pulled the amount, date and supplier using an LLM. It worked fine in testing. In production, every so often the exact same invoice format would come back with a field misread that had come out perfect the time before. It wasn’t a bug in our code, it was the model being a model. We dropped the temperature to the minimum and added a validation layer with fixed rules, the amount has to be a valid number, the date has to actually exist, and the problem got a lot better. It didn’t disappear entirely. With these tools, “almost always right” is not the same as “always right”, and you need to design knowing which one you actually need for each piece of the process.

For normal use, asking something, getting ideas, having it help draft an email, this variability doesn’t matter at all, it even helps. The problem is when someone builds an automated business process assuming an LLM behaves like a spreadsheet formula. It doesn’t. The sooner you have that straight while designing the automation, the fewer surprises with weird data you’ll run into six months after switching it on and walking away.