The Day an Expired SSL Certificate Took Down a Client's Site Before We Noticed
A client who sells second-hand farm machinery called me at nine on a Tuesday morning. His site was showing “Your connection is not private” in Chrome’s usual red warning, the kind that scares anyone who doesn’t know what it means. He thought he’d been hacked. Nobody had touched anything: the SSL certificate had expired at 3:17am that same night, and we found out from him, not from our own monitoring.
An SSL certificate expires when the automatic renewal process fails quietly and nobody catches it in time: Let’s Encrypt only issues certificates valid for 90 days, so if the certbot cron job doesn’t run cleanly, the browser starts blocking the site the moment it expires, with no warning visible to whoever manages the server.
In our case the cron was running. That was the real problem, actually, because it gave us a false sense that everything was fine for weeks. When we moved that server to the new VPS back in August, we changed the system user running the services, and certbot lost permission to write the renewed certificate to the path nginx expected. The command exited with a non-zero code every single night, dumping the error into /var/log/letsencrypt/renewal.log, a file that, let’s be honest, nobody checks until after something’s already broken.
And here’s the part that bugs me most: we’ve spent years telling clients the browser’s padlock isn’t cosmetic, that Google ranks unencrypted sites lower, that a contact form without HTTPS is careless. Having it happen to us was more of a hit to the ego than a serious technical problem. Once we found it, the fix took fifteen minutes: correct the permissions, force a renewal with certbot renew --force-renewal, reload nginx. The embarrassment lasted a lot longer.
A cron that “runs” isn’t enough
The real failure wasn’t the wrong permission, that kind of thing happens. The failure was having no alert for when renewal didn’t actually succeed, just trusting that the cron would run and leaving it at that. A cron that runs but fails is indistinguishable, from the outside, from one that doesn’t exist at all, and in both cases the result is the same: you find out when someone stares at the red screen.
Since then, every certificate on that server pings an external heartbeat monitoring service right after it renews successfully. If that ping doesn’t arrive within 25 hours (we left some margin over the daily cron’s 24), an alert goes to the team’s Telegram group, not to an inbox nobody opens on a weekend. It takes about five minutes to set up per domain and would have saved us this whole story. We currently run fourteen client domains with their own certificates on that server, and going through them one by one to check if the cron was still alive, when this happened, ate an entire afternoon we don’t want to lose again.
What we still haven’t solved
What we haven’t solved, and I won’t pretend otherwise, is what happens the day the monitoring service itself fails instead of the certificate. Adding an alert that fires when something breaks fixes today’s problem, but pushes the same risk one level up: now we depend on that heartbeat working, and on someone occasionally checking that it’s still active. After the security audit we ran on our own server, we already knew that blindly trusting “something automated” to keep working forever is the cheapest way to get a nasty surprise. This was just the latest reminder.