Cloud and DevOps
Disaster recovery
Disaster recovery is the plan and infrastructure for restoring a service after a major outage, set by how fast it must recover and how much data it may lose.
Teams write a disaster recovery plan for the failures that redundancy inside one region cannot absorb: a region outage, a deleted production database, ransomware. The plan sets two numbers up front, how long you can be down and how much data you can lose, then buys just enough infrastructure to meet them.
Why does disaster recovery matter?
- Regions fail. Rarely, but every major provider has lost a region for hours, and high availability across zones does not help when the whole region is gone.
- People make mistakes. The most common disaster is an engineer or a script deleting production data. Replication copies the mistake; only a backup from before it helps.
What are RTO and RPO?
- Recovery time objective (RTO). The longest the service may be down, from the moment of the disaster to the moment it serves customers again.
- Recovery point objective (RPO). The most data, measured as time, that may be lost. It is the gap between the last good copy and the disaster.
A worked example. An online store sets an RTO of four hours and an RPO of one hour. Backups therefore run at least hourly, and the restore procedure, including rebuilding the servers and switching DNS, must finish in under four hours. If a disaster strikes at 14:35 and the last backup was at 14:00, the store loses 35 minutes of orders, inside the RPO, and must be serving again by 18:35. A pilot light meets this; backup and restore from cold storage probably does not, and active-active is more than the business needs.
What are the disaster recovery strategies?
The diagram at the top of this page shows the four standard strategies as a strip, ordered by RTO and cost. The primary region on top serves users today, and each column is one possible shape for the second region, from backups only to a full running copy.
| Strategy | What waits in the DR region | RTO | RPO | Cost |
|---|---|---|---|---|
| Backup and restore | Backups in storage, nothing running | Hours to a day | Hours | Lowest |
| Pilot light | A live database replica; core services ready but switched off | Tens of minutes | Minutes | Low |
| Warm standby | A scaled-down copy of the whole stack, running | Minutes | Seconds to minutes | High |
| Active-active | A full copy serving live traffic alongside the primary | Near zero | Near zero | Highest |
- Backup and restore. Data is copied to storage in another region, and recovery means provisioning everything from scratch and restoring it. Cheap, slow and the only option that also protects against a corrupted database, because backups are point-in-time copies.
- Pilot light. The database replicates continuously to the second region, but the application tier is defined and not running. Recovery means starting the app servers.
- Warm standby. Everything runs in the second region at reduced size. Recovery means scaling up and switching traffic.
- Active-active. Both regions serve traffic all the time with data replicated between them. The hardest to build, because writes in two places need conflict handling.
What does a disaster recovery plan include?
- Scope. Which systems are covered, in what order, with an RTO, RPO and strategy for each tier.
- Backups. What is backed up, how often, where it is kept and how it is protected from the same disaster, including from an attacker with production credentials.
- Infrastructure definition. The second region defined in infrastructure as code, so rebuilding is a CI/CD pipeline run rather than a weekend.
- Runbooks. Steps for declaring a disaster, failing over, verifying and failing back, with named roles.
- Diagrams. An architecture diagram of both regions and the replication between them, the first thing the on-call engineer opens.
How do you build a disaster recovery plan?
- Classify systems by business impact and assign an RTO and RPO to each tier.
- Pick a strategy per tier that meets the objectives at the lowest cost.
- Define the second region as code and provision at least the data layer.
- Set up backups and replication to meet the RPO and verify they restore.
- Write the runbook and have someone else follow it.
- Test a full failover, measure the real RTO and RPO and fix the gaps.
- Plan the failback. Returning to the primary region is its own procedure and is often forgotten.
What are common disaster recovery mistakes?
- Backups that were never restored. Restore is the only test that counts.
- Backups in the same blast radius. Same account and credentials as production, so a compromised key takes both.
How does disaster recovery relate to other practices?
A backup is a copy of the data; disaster recovery is the whole plan to run the service again from it. High availability handles failures inside one region; disaster recovery handles the region itself. Multi-cloud and hybrid cloud designs sometimes use a second provider or an on-premises site as the DR region, at the cost of two sets of tooling. All of it rests on the regions and zones of a cloud computing architecture.
What tools help with disaster recovery?
- Managed backup and replication services. Cross-region snapshots, continuous database replication and storage replication from any major cloud provider.
- Infrastructure as code and DNS tools. The tools that build the primary region rebuild the second, plus a DNS service or global load balancer to switch traffic.
- AI diagramming tools. Tools like Eraser turn a description of your regions, replication and failover paths into an architecture diagram, or draw the current state from your infrastructure code with a Terraform diagram generator, so the runbook has a picture that matches reality.
For more on this kind of diagram, see architecture diagrams.
Frequently asked questions
-
What is the difference between RTO and RPO?
The recovery time objective (RTO) is the maximum acceptable downtime: how long from the disaster until the service is back. The recovery point objective (RPO) is the maximum acceptable data loss, measured as time: how old the last surviving copy of the data may be. A four-hour RTO and one-hour RPO means back within four hours having lost at most an hour of writes.
-
What is the difference between disaster recovery and backup?
A backup is a copy of the data. Disaster recovery is the whole plan to get the service running again: where the backup is, how the infrastructure is rebuilt, who does what, how DNS is switched and how it is tested. A backup you have never restored is not a disaster recovery plan; it is a hope.
-
What is the difference between disaster recovery and high availability?
High availability handles small, frequent failures inside one region automatically in seconds, using redundant components and health checks. Disaster recovery handles the loss of a whole region, usually with a second region and a runbook, and takes minutes to hours. High availability keeps you up on a normal day; disaster recovery gets you back after a bad one.
-
How often should you test a disaster recovery plan?
At least once a year for a full failover, and more often for the pieces: restore a backup monthly, run the runbook as a tabletop exercise quarterly and fail over a non-critical service every few months. Every test finds something: an expired credential, a missing DNS record, a step only one person knew.
