What's a token in AI, and why does it decide what I pay every month?

The first time a client asked me “wait, why are you charging me per word?” I had to stop the meeting and explain that it’s not words. It’s tokens, and the difference matters more than it sounds like once the API bill lands at the end of the month.

A token is a chunk of text, almost never a whole word. “Cat” is usually one token. “Extraordinarily” might split into three or four. Models from OpenAI, Claude or Gemini don’t read letters or loose words: they read these fragments, and every API call gets billed on how many go in (the prompt you send) and how many come out (what the model replies). Spanish, as it happens, tends to burn more tokens than English to say the same thing — we found that out the hard way on a hospitality client’s project, when the monthly API spend came in almost 40% over what we’d estimated using English test copy.

That has consequences that shape how you design an AI feature, not just what it costs:

• A long system prompt (the fixed instructions you send the model on every request) gets billed on every single call, even if the user only typed two words.
• If a chatbot keeps the full conversation history to fake “memory,” every new message drags the whole thread along with it, so the cost per turn climbs as the chat goes on.
• The more capable models don’t just charge more per token — left unconstrained, they also tend to write longer replies, so the cost stacks twice over.

When we budget an AI integration for a client, the first question isn’t “how much will AI cost” in the abstract anymore. It’s estimating tokens per interaction. We take real examples of what the user is going to type, run them through each provider’s tokenizer (they all publish one, free to use) and multiply by the expected monthly volume. It’s a tedious step. It’s also the one that stops surprises.

There’s a detail almost nobody asks about that more people should: a model’s context limit is also measured in tokens, not “messages” or “pages.” If a client wants the AI to “read the whole product catalog” before answering, that catalog has to fit, in tokens, inside that window. If it doesn’t, you either chunk it up or build a vector store that pulls in only what’s relevant per query — and at that point it’s a different project with a different budget, not a box you tick in the first meeting.

What still bugs me is that almost no provider shows you a live token count while you’re typing the prompt, as if they’d rather you not think about it until the charge hits the card.