Why does the AI chatbot on my website take so long to answer?

A client who runs a furniture shop wrote to us pretty annoyed because the chatbot we’d just installed “took forever” to answer. We grabbed a phone and timed it: 4.2 seconds from the moment the user hit enter to the first word showing up. To us, after months of staring at logs for this stuff, that didn’t sound like much. To someone used to WhatsApp, where the reply shows up almost instantly, it is.

Why an AI chatbot takes so long to answer has a fairly concrete explanation: it depends on the size of the model you’re using, how much context you’re sending with each request, and whether it has to call an external tool (a database, a catalog, another API) before it can reply. A large model, carrying a long conversation history, with an inventory lookup thrown in, can easily take more than three seconds even when everything is working exactly as it should.

Model size matters more than people expect. When we ran the same flow on GPT-4 and on a smaller model built for simple tasks, the difference went from 4 seconds down to 900 milliseconds for questions like “do you have this table in oak?”. The large model was still needed for the odd cases, the ones mixing several conditions or requiring a bit of actual reasoning. For everything else, we were paying extra for precision nobody needed at that point.

What the client actually cared about, though, wasn’t the exact number — it was how it felt. And there’s a trick that changes the perception without touching the real latency: showing the reply word by word as it arrives (what’s usually called streaming), instead of waiting for the full text and dumping it on screen all at once. Psychologically, seeing that something is “already happening” is tolerated far better than a blank screen with a spinner.

There are other things that actually cut the real time, not just the feeling of it. Trimming the system prompt down to what’s strictly needed helps, since every extra token gets processed before anything gets generated. Caching results from external calls that don’t change by the minute — prices, rough stock levels, shipping policies — also removes the slowest part, which is almost never the model itself but the round trip to some other system; it’s the same logic we go into when explaining why you sometimes don’t see changes on a website, just applied to data instead of pages. And using a fast model as a first filter, one that only escalates to the big model when the question actually calls for it, is basically what we did for the furniture shop: same inbox, two speeds depending on what’s needed.

The furniture client is still running that mixed setup months later. What we haven’t fully figured out, and we won’t dress it up, is exactly where to draw the line between “this is simple, send it down the fast path” and “this needs the big model”: every time that rule gets tweaked, some new question shows up that doesn’t quite fit in either drawer.