The restore was attempted and it did not work. The file was truncated, or the tooling version could not read it, or it restored successfully and was missing a fortnight of data. Every one of these is common, and every one is detectable in advance by performing a restore before you need one.
01A backup is unverified until restored
Backup jobs report success when they finish writing a file. That says nothing about whether the file is complete, readable, or contains what you assume. Silent partial failures are routine - a job that ran while a volume was full, or a dump interrupted and never validated.
The only meaningful verification is restoring to a separate environment and confirming the application starts against it and the data is present. Anything short of that is an assumption, and it is tested for the first time during an incident.
| Failure | Detected by |
|---|---|
| Truncated or corrupt file | Attempting a restore |
| Version mismatch in tooling | Restoring with production tooling |
| Missing extensions or roles | Starting the application against it |
| Encryption key unavailable | Restoring in an isolated environment |
| Backup excludes a critical table | Comparing row counts |
| Restore exceeds acceptable downtime | Timing the restore |
| Object storage not backed up at all | Inventorying what is covered |
02State your objectives as numbers
Recovery time objective is how long restoration may take. Recovery point objective is how much data you can afford to lose. Both must be explicit numbers agreed with the business, because they determine the architecture and its cost.
An RPO of one hour means backups at least hourly, or continuous replication. An RTO of thirty minutes on a large database rules out restoring from a dump entirely, because copying and replaying will take longer than that regardless of preparation.
03Databases are not the only thing to back up
Uploaded files in object storage, secrets, infrastructure configuration, TLS certificates and DNS records are all needed to reconstitute a service. A restored database referencing files that no longer exist gives you a working application displaying broken links throughout.
Inventory everything required to rebuild from nothing, and confirm each item has a backup with an owner. The gaps are usually in the components that were configured once by hand and never captured anywhere.
| Step | Confirms |
|---|---|
| Restore to a clean environment | The backup is readable |
| Start the application against it | Schema and extensions are complete |
| Compare row counts to production | Nothing was silently excluded |
| Check the newest record's timestamp | Actual recovery point |
| Verify uploaded files resolve | Object storage is covered |
| Record total elapsed time | Actual recovery time |
04Keep copies somewhere separate
Backups in the same account and region as production share its failure modes. An account compromise, an accidental deletion with sufficient permissions, or a regional outage can remove production and its backups together.
Keep at least one copy in a separate account with restricted permissions, ideally with object lock or immutability so it cannot be deleted within its retention period. This is specifically what makes ransomware and insider-deletion scenarios survivable.
05Automate the drill so it keeps happening
A restore tested once decays as schemas evolve, dependencies change and tooling is upgraded. A scheduled job that restores the latest backup to a scratch environment, runs verification queries and reports the outcome keeps the guarantee current rather than historical.
Alert when that job fails, and treat the failure with the urgency of a production incident. A broken restore process is a production problem that has not surfaced yet, and the interval between it breaking and you needing it is entirely a matter of luck.
Topics
Lena Voss
Lead Architect · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer