What's the difference between using AI "in the cloud" and running one myself on my own servers?

A few months back a client in the legal sector asked us for something very specific: they wanted an assistant to help their team search through thousands of pages of case files, but they flat-out refused to let any of those documents leave their server. Not to OpenAI, not to Anthropic, not to anyone. That’s exactly where the difference between “cloud” AI and self-hosted open-source AI comes in.

When you talk to ChatGPT or Claude, your questions travel to a server run by OpenAI or Anthropic, get processed there, and the answer comes back to you. Convenient, fast, and you never have to think about infrastructure. The catch is that for some businesses, sending sensitive information to a third party simply isn’t an option — whether because of a contract, industry regulation, or the client just not trusting it.

That’s where open-source (or “open-weight”, the more precise term) models come in: Meta’s Llama, Mistral, Qwen. Models you can download and run on your own hardware, so not a single word of what you ask ever leaves your network. With tools like Ollama or vLLM, setting one up isn’t nearly as complicated as it sounds — a couple of days of work, not weeks.

What does cost real money is the hardware. For the law firm’s project we ended up renting a GPU machine from a cloud provider (buying your own graphics cards is something we almost always advise against), and even then the monthly bill ran about four times what using the Anthropic API would have cost for the same volume of queries. And honestly, the quality of the answers was worse: Llama does a decent job, but it’s not in the same league as Claude or GPT-4 on complex reasoning over long text.

So the real question isn’t “open source, yes or no”. It’s “which matters more to me, confidentiality or quality”. For most of our clients the answer is clear: using a serious provider’s API, with a solid data processing agreement in place, ends up cheaper, faster to set up, and gives better results. The law firm was one of the rare cases with no way around it: their own lawyers had confidentiality obligations that ruled out any middleman, however good the contract on paper.

There’s a middle ground people don’t talk about as much: some providers (Azure OpenAI, Amazon Bedrock) offer the same closed models but with contractual guarantees that your data isn’t used for training and isn’t shared with anyone else. For a good number of clients worried about privacy, that solves the problem without standing up your own servers or giving up quality. We’ve yet to explain this to someone without watching their shoulders drop a little.

What I honestly don’t know is what happens in two years, once open models close the quality gap with the closed ones. Maybe “self-hosted or not” stops being a question of sacrificing quality and turns into a purely operational cost calculation. Or maybe the big models just stay one step ahead forever. We’ll see.