What is multimodal AI, and how is it different from a text-only chatbot?
A client of ours who sells lamps and home decor online got in touch a few weeks ago. She wanted to automate part of the returns process: when a buyer sends a photo saying “it arrived broken,” she wanted the system to look at that image and decide whether to approve the return, instead of someone on her team opening every email and checking the picture by hand.
That’s exactly what multimodal AI does: it understands text and images (sometimes audio or video too) in the same conversation, without needing a separate system for each type of content. You send a photo along with a question, and it answers based on what it actually sees, the way you’d describe a picture to someone over the phone.
Until a couple of years ago, building something like this meant stitching pieces together: a vision model to classify the image, OCR if there was text in the photo, and then a separate language model to draft a response using whatever the other two handed it. Three systems, three places where something could break. With a multimodal model like Claude or GPT-4o, all of that collapses into a single call: the image sits in the same context window as the text, and the model reasons over both at once.
We had a working prototype for the client in an afternoon, literally. We fed it the photo plus a prompt with her return policy’s conditions, and it comes back with whether there’s visible damage, what kind, and a short note for the team. It handles obvious breakage well (a shattered piece of glass is easy to catch) and struggles more with subtler stuff: a small scratch on a matte finish gets flagged some of the time and missed the rest. We still keep a human checking before anything gets approved automatically. The AI cuts down the work of eyeballing every photo one by one. It doesn’t replace the final call.
What almost nobody asks upfront, and should, is what happens to those photos once they leave your server. They go to a third-party provider’s API, same as with any language model, so if a customer uploads something with visible personal details (an address on a label, a face in the background) it needs the same care as any other data you hand over to an AI. In this case we told the client upfront that return photos go through that process, and put it in writing in her policy, not as a footnote.
We haven’t tried feeding audio into the same flow yet, though the same client has already asked whether something similar could work for the voice notes people sometimes send explaining what happened. Technically, sure, it could. Whether it’s worth building for the volume of returns she gets is a question I don’t have a clean answer to.