Can I train an AI on my company's data?

A client who resells secondhand industrial machinery asked us a couple of months ago if we could “train an AI” on fifteen years of emails and quotes sitting on a server. He wanted an assistant that could answer his sales reps’ technical questions using that history. I told him something like that was doable, but “train” was the wrong word, and that if anyone offers to “train your own model” for two thousand euros, he should be suspicious.

Training a model from scratch, or even fine-tuning an existing one, takes a lot more data than a typical small business has lying around, and the result usually isn’t any better than the cheaper option: giving the model access to your documents at the moment someone asks a question, instead of trying to get it to memorize them. That’s called RAG (retrieval-augmented generation), and it’s what almost any agency actually builds when they talk about “AI with your company’s data,” even if they don’t always call it that.

The practical difference is this: with fine-tuning you’re modifying the model’s weights, which means GPU time, training runs, and having to retrain every time your data changes. With RAG you store your documents, chunked up, in a vector database, and when someone asks something, the system pulls the most relevant chunks and hands them to the model as context before it answers. Adding a new quote is literally just uploading a file. No retraining, no GPU bill.

For the machinery client we built exactly that: their emails and quotes chunked and dropped into a vector database (we used pgvector since they already ran Postgres, no need to bolt on a new tool), plus an internal assistant a rep can ask “what did we quote client X for that forklift model back in 2024?” and get the relevant document back with the answer already summarized. It cost a fraction of what fine-tuning would have, and we had it in production in two weeks.

There is one case where fine-tuning actually makes sense: when you need the model to always answer in a very specific format or tone and RAG can’t force that on its own (say, generating code in a particular internal style based on thousands of existing examples). But that’s rare in the kind of work we do for small agency clients, and when it has come up it’s almost always been for a very specific content-generation case, not “answer questions about my data.”

What bugs me most about this whole topic is how many proposals I’ve seen floating around offering to “train your own custom AI” when what’s actually being built is a RAG setup with a handful of lines of code wrapped around it. It’s not exactly a lie, but it inflates the price and makes people think they need far more infrastructure than they actually do. If someone pitches it that way, just ask directly: is this fine-tuning or RAG? The answer tells you pretty quickly whether the person selling it to you knows what they’re doing.