Browsers show a full-page security warning, mobile apps fail to connect, and integrations start rejecting connections. A certificate expired. This is among the most preventable outages in existence - the expiry date was known months ahead - and it keeps happening because organisations automate renewal and then never verify that the automation still works.
01Automated renewal fails quietly
Renewal typically fails for mundane reasons: the validation challenge path stopped being reachable after a configuration change, a firewall rule now blocks the validating service, DNS credentials expired, or the renewal timer was disabled during unrelated maintenance and never re-enabled.
The renewal process usually logs the failure and exits without notifying anyone. Nothing appears broken because the existing certificate remains valid for weeks, so the failure is discovered at expiry rather than at the point it started failing.
| Cause | Detected by |
|---|---|
| Validation path no longer reachable | Dry-run renewal |
| Firewall blocking the validator | Dry-run renewal |
| DNS API credentials expired | Dry-run renewal |
| Renewal timer disabled | Checking the timer is active |
| Certificate renewed but not reloaded | Monitoring the served certificate |
| Rate limit from repeated attempts | Renewal logs |
02Monitor what is served, not what is on disk
A renewed certificate sitting in a directory does nothing until the service reloads. A surprisingly common failure is successful renewal followed by no reload, so the server continues presenting the expired certificate while every file on disk looks correct.
Monitoring must therefore connect over the network and inspect the certificate actually presented. That check catches both renewal failure and reload failure, which file-based checks cannot distinguish.
03Alert with enough runway to act
A single alert at seven days is insufficient if it fires during a holiday or lands with someone on leave. Escalating alerts at thirty, fourteen and seven days give progressively louder signals and ensure the problem is visible while there is still comfortable time to fix it.
Route these to a channel a team monitors rather than an individual's inbox. Certificate alerts sent to a departed employee are a recurring theme in expiry postmortems.
| Certificate | Failure when it expires |
|---|---|
| Main public domain | Full site outage with browser warning |
| API subdomain | Mobile apps and integrations fail |
| Internal service certificates | Service-to-service calls fail |
| Client certificates for partners | Partner integration stops |
| Code signing certificate | Installers rejected by the OS |
| Mail server certificate | Mail clients refuse to connect |
04Test renewal, do not assume it
Most tooling offers a dry-run mode that exercises the full renewal path without issuing a certificate. Running that on a schedule and alerting on failure detects a broken renewal within a day of it breaking, rather than at expiry.
This turns a silent, delayed failure into a prompt notification, which is the entire difference between routine maintenance and an outage. It costs almost nothing to set up.
05Shorter lifetimes make this worse if unautomated
Certificate lifetimes continue to shorten across the industry. Any process that depends on someone remembering, or on a calendar reminder, becomes progressively less viable as renewals grow more frequent. Automation is no longer optional at these intervals.
The complete solution has three parts, and all three are necessary: automated renewal, automated reload of the services that present the certificate, and independent monitoring of what is actually being served. Any two without the third eventually produces this outage.
Topics
Marcus Hale
Security Lead · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer