What is a chatbot's system prompt, and why does it change its behavior so much?

A client called us a couple of months ago, pretty annoyed: the customer-service chatbot we’d built for them had started recommending competitors’ products. It wasn’t a model glitch. It was the system prompt, badly written, that never told the bot what to do when it didn’t know the answer.

A chatbot’s system prompt is the block of instructions sent to the model before every conversation, invisible to whoever’s typing, and it sets the bot’s role, tone and limits; change a handful of sentences there and the same model can go from sounding like a generic bot to acting like the assistant for one specific brand.

In practice it’s a text file, sometimes two paragraphs, sometimes four pages. We put things in it like “you’re this store’s assistant, you only talk about products in this catalog” or “if you don’t know the shipping cost, say so, don’t make up a number.” That’s where 80% of the real work is when we build an agent or chatbot for a client: not in picking the model, in tuning that text against weird conversations a normal user would never type in a demo.

What nobody tells you starting out is that the same prompt behaves differently depending on the model. We had one that worked great on GPT-4 and turned evasive on Claude with the exact same instructions, word for word. Had to rewrite half the instruction just because of that provider switch.

And there’s a limit we sometimes forget to flag for the client: a system prompt isn’t security. A patient user can get the bot to ignore its instructions. Stopping that takes more than a nicely written paragraph up front: filters outside the prompt, backend validation, limits that don’t rely on the model “remembering” to behave.