Disaster recovery that has actually been tested: RTO, RPO and the restore nobody tried

01RTO and RPO are business decisions with engineering price tags

Recovery Time Objective is how long you can be down. Recovery Point Objective is how much data you can lose. They are frequently stated as near-zero because nobody wants to be the person who agreed to losing an hour of orders, and then the architecture is built to a budget that cannot deliver it.

Force the conversation the other way round. Present what each tier costs and let the business choose. An hour of RPO is cheap; five minutes needs continuous replication; zero needs synchronous writes across sites and a latency penalty on every transaction. When the numbers are visible, most systems turn out not to need the expensive tier - and the two or three that do get funded properly.

02The restore is the part that fails

Backup jobs report on whether bytes were written. They cannot tell you the dump was consistent, the encryption key is still available, the target engine version can read the format, or that a restore fits inside your RTO. Each of those has taken teams down after a backup dashboard showed green for months.

Restore time is dominated by things nobody measures until the day: transferring a multi-terabyte snapshot across regions, replaying write-ahead logs to a point in time, and rebuilding indexes afterwards. A database that takes fifteen minutes to snapshot can take four hours to restore and warm, and the RTO in the runbook was written from the snapshot number.

03Test restores on a schedule, into a real environment

Automate a restore into an isolated environment at least monthly, and assert on the result rather than on the process completing. Check row counts against expectations, run a handful of representative queries, verify the newest record is within your RPO, and record the wall-clock time from decision to serving traffic.

That wall-clock number is your real RTO. Publish it and let it be uncomfortable. The first time most teams do this, the measured figure is three to five times the documented one, and the reasons are mundane - the runbook assumed someone knew a credential, or the restore needed a parameter group that only existed in production.

• Restore on a schedule, automatically, into a fresh environment.
• Assert on data content and freshness, not just exit codes.
• Time the full sequence including DNS, cache warm and smoke tests.
• Rotate who runs it, so recovery is not held in one person's memory.

04The failure modes backups do not cover

Most DR planning assumes infrastructure loss. The more common real events are logical: a bad migration truncates a table, a bug writes nulls over a column for three days, or a compromised credential deletes data deliberately. A replica faithfully reproduces every one of those in milliseconds.

That is why replication is not backup. You need point-in-time recovery far enough back to cover the time to notice a slow-moving corruption, and you need at least one copy that the production credentials cannot delete. Immutable retention on the backup store is the control that survives an attacker with your admin keys, and it is cheap relative to what it protects.

• Keep point-in-time recovery covering longer than your detection lag.
• Store one copy in a separate account with immutable retention.
• Verify backups cannot be deleted by production identities.
• Cover the non-database state too - object storage, secrets, queue contents.

05Dependencies decide your real recovery order

Services fail over in dependency order, and most plans document each service in isolation. If your application recovers in ten minutes but its identity provider, DNS, secrets store and container registry are in the failed region, your recovery time is the slowest of those, not the fastest.

Map the dependency graph including the third parties, then test the order. The recurring discoveries are that the deployment pipeline itself lives in the region that is down, that the secrets needed to bootstrap are stored behind the service being restored, and that certificates were issued by an automation that cannot run. Circular bootstrap dependencies are the most common reason a technically sound plan takes a day to execute.

06Make the exercise routine and the plan short

Long DR documents go stale within a quarter because nothing forces them to stay true. Short runbooks that are executed monthly stay accurate because they break loudly when they are wrong. Prefer a one-page sequence of commands over a thirty-page policy document.

Run a game day twice a year where the plan is followed by someone who did not write it, with the author unavailable. What that surfaces is not technical gaps but knowledge gaps - the steps that were obvious to the author and are unrunnable by anyone else. Fix those, and your recovery stops depending on which people happen to be reachable.