How we find out something broke in production before the client does
One Saturday morning, a client who runs an online grocery store wrote to us saying their checkout had been broken since Friday night. We hadn’t spotted it. He had, staring at the orders panel and wondering why nothing had come in for fourteen hours. That was over a year ago and it still makes me a little uncomfortable to tell, but it’s the reason almost no project leaves our hands anymore without something watching whether it’s still alive.
What had happened was dumb: a payment gateway change started returning a different code than our backend expected, and the error got swallowed silently inside a try/catch that someone (me, to be precise) had written months earlier “just in case” and never looked at again. No clear logs, no alert, nothing. The checkout just showed a blank screen and the user left.
After that we started adding Sentry to every project with its own backend, not just the big ones. It costs almost nothing — the free tier covers a small client fine — and it catches unhandled exceptions before an annoyed user does. For simpler sites without a real backend, we run a self-hosted Uptime Kuma on the same VPS where we already had other things running: it pings the important URLs every minute and warns us on Telegram if something stops responding or gets too slow.
The part we don’t tell as often is that the first version of this was a disaster of excess, not neglect. I configured Sentry to notify me of every exception, ungrouped, no threshold. Within a week I had over two hundred emails piled up from a minor error in a third-party library that didn’t affect anything real. I stopped reading them. And of course, the day something actually mattered, it got buried in the usual noise.
We had to learn to tell apart “this breaks something the user notices” from “this is a warning that can wait until Monday.” Critical alerts now go to a Telegram channel we all actually check, and everything else sits archived in the dashboard for a weekly review. Considerably less elegant than any observability talk you’ve sat through at a conference, but it works.
What we still haven’t figured out is on-call. There are three of us and none of us wants to be the one checking their phone at eleven on a Sunday night for a client paying three hundred euros a month in maintenance. Right now the informal rule is “whoever sees it first tells the group,” which holds up as long as the team stays small and nobody’s on holiday at the same time. I don’t know what we’ll do the day those two stop lining up.