Skip to content
digitapulia.com

Backups are not recovery: running a recovery drill and measuring the real time

Why a restore job that finishes is not a business back at work, which clock a recovery drill reads, and how to run one that measures the real time.

· 27 min read

Two old stopwatches side by side in a wooden mount behind glass, from an airport's old control tower, with the empty socket of a third beside them.
Photo: Wolfmann · CC BY-SA 4.0

Ask how long it would take to have the ERP working again if its server failed tomorrow, and the answer may well come as a number: a few hours, half a day. Behind that number there is often a single measurement of a single step: how long a restore job ran the one time someone tried it, or how long the backup vendor says a restore takes.

That number is not the recovery time. The recovery time starts when the outage starts, and ends when the business can work again: when someone can enter an order, issue an invoice, dispatch a lorry. Between those two moments there is much more than a restore job. Someone has to find the backups and get the permission to use them, rebuild what the job does not rebuild, and check what came back. Some of those steps may never have been tried.

A recovery drill is how you find out the real number before you need it. This piece is about running one so that it measures the right thing: the clocks that are easy to read and misleading, the two targets the result is compared with, where the time goes after the restore job ends, how to prepare a drill that touches neither production nor the outside world, how to record the day, and how to read the result. The platform details come from Microsoft’s documentation for Azure Backup and Azure Site Recovery, listed at the end. The method is the same wherever your systems run: in your own server room, in another cloud, or in several places at once.

The clocks that are not the recovery time

During a recovery, several things show a time, and each of them measures something real. None of them measures what the business needs to know.

The restore job

Azure Backup creates a job for every restore and shows its progress: an estimated time of restore, which reaches zero when the restore operation finishes, the percentage done, and the bytes transferred. That is the time taken to bring the data back. It starts when someone launches the restore, which is already some way into the outage, and it ends when the restore operation finishes, before anyone has connected to the machine, checked it or used it.

The failover job

Azure Site Recovery tracks a test failover on its Jobs tab, and its documentation lists the steps that run: a prerequisites check, the processing of the data, and the creation of the virtual machine. Its documentation adds that in some scenarios, such as physical servers and some VMware configurations, the failover needs an extra intermediate step that usually takes around 8 to 10 minutes. Here too, the job ends when the machine exists, not when someone can work on it.

The mean time to recover

Microsoft’s Well-Architected Framework defines the mean time to recover, MTTR, as the time taken to restore a component after a failure is detected. It is a useful figure for the technical team, and by its definition it concerns one component and starts at detection.

The clock a drill reads

A drill reads a different clock: from the moment the outage is declared to the moment a person from the business says the system works for them. That is the time to compare with your recovery time objective. The job timers are worth recording as well, because they show how much of the total was the platform’s work and how much was everything else, but they are lines in the log, not the result.

RecoverypointOutagedeclaredIt works forthe businessThe job timerReachingthe backupsThe jobitselfAfter the job,the checkData lost that day,and its worst caseAgainst the RPOThe recovery timethe drill measuresAgainst the RTO
The job timer measures one stretch. The drill reads the clock from the declared outage to the business check.

The two targets the result is compared with

A measured time is good or bad only against a target. The Well-Architected Framework defines the two that matter here. The recovery time objective, RTO, is the maximum acceptable time that an application can be unavailable after an incident. The recovery point objective, RPO, is the maximum acceptable duration of data loss during an incident.

The same guide says who sets them: business stakeholders define broad requirements, technical experts assess the current state of the workload, and both must agree on realistic targets. It also notes that Microsoft publishes RTO and RPO guarantees only for some products. The targets are therefore a decision of the business, checked against what the systems can do, and not a figure copied from a vendor’s page.

If there are no targets yet, a drill still measures the time and the loss as they are today, and management can then decide whether to accept them. But it is worth asking for them first, even rough ones, because they change the drill: a recovery that must take hours is not prepared like one that may take days, and a business that accepts losing a day of work asks different questions of its backups from one that accepts an hour.

Where the time goes after the job ends

When the restore job shows zero, the recovery is not over. Microsoft’s own documentation lists what is still to be done, and each item is a step the drill times.

After an Azure Backup restore

Azure Backup’s page on restoring virtual machines lists a set of post-restore steps. The extensions present when the backup was configured come back installed, but not enabled. When the restore creates a new virtual machine, that machine is a new resource, and the role assignments scoped to the original machine do not carry over, and Microsoft says to reassign them as needed. In a drill, only roles over resources inside the test environment are given; the time the others would take is noted. A machine that had a static IP address comes back with a dynamic one, to avoid a conflict. A restored machine has no availability set. On a Linux distribution that uses cloud-init, such as Ubuntu, the password is blocked after the restore, for security reasons, and has to be reset. And a machine with a special network configuration, such as load balancing or several network cards, is restored as disks, and the machine is created again from them; in a drill it gets no public address and no external load balancer.

Each of these steps takes time that no job timer shows, and each needs someone who knows it has to be done.

After a Site Recovery failover

Site Recovery’s page on test failovers warns that if a Windows update is pending on the machine when the failover is triggered, the update might start after the failover, and nobody can log on until it completes. For application testing, it asks for a copy of the production Active Directory in the test environment, and for the DNS of the test network to be updated with the address of the DNS machine. Its page on recovery plans lists tasks that often follow a failover, inside the recovered machines, such as changing a database connection string or the web server’s configuration.

What your applications add

Then come the steps that no platform documents, because they belong to your applications: the order in which services start, the scheduled jobs, the files kept outside the database, the printer queues, the connections to other systems. Site Recovery’s page on recovery plans takes a three-tier application as its example: the SQL Server database starts first, then the middleware, then the web front end, so that each tier finds the one it needs already running and users do not connect before everything is ready. The same page warns that manual steps make the process prone to error, and that the person running the recovery might not know all of the application’s intricacies.

AFTER AN AZURE BACKUP RESTOREExtensions: enabled againRoles: only inside the testStatic IP address: set againAvailability set: nonecloud-init Linux: password resetSpecial networks: rebuiltAFTER A SITE RECOVERY FAILOVERA pending Windows updateActive Directory and DNSDatabase connection stringsWeb server configurationWHAT YOUR APPLICATIONS ADDStart-up order: database firstScheduled jobs, files elsewhereConnections to other systemsAutomatic sends: switched offNo job timer counts any of these. The drill times each one.
After the job: the steps from Microsoft’s documentation and from your own applications that no progress bar shows.

Choosing what fails

A drill simulates one outage, chosen in advance, and the choice decides what is measured.

A server that no longer starts

This measures the restore of a whole machine, and everything listed above that follows it.

A database deleted by mistake

This measures something else. The server is still there, the data is not, and the question is how far back the last good copy goes. Azure Backup’s documentation for SQL Server databases on Azure virtual machines shows the choice a drill must get right. A database can be restored to an alternate location, keeping the original source database, or to its original location, which overwrites it. In a drill it is always the alternate location: a server inside the isolated environment, which the documentation requires to be registered to the same vault as the source. That server cannot be fully isolated. Microsoft’s page on backing up SQL Server on Azure virtual machines says it needs connectivity to the Azure Backup service, Azure Storage and Microsoft Entra ID for every operation, restores included. Private endpoints, which keep that traffic off the public internet, do not include Entra ID, and can be created only on a new vault with no item registered to it; the service tags the same page offers for network security groups open the whole service, not one account. So the restore server gets one explicit outbound path, built and tested before the day, that lets through only those three services, each of them whole. It is the one exception to the isolation checklist below, and it is closed as soon as the restore ends, before the recovered application starts. That application, the one the business check will use, is brought up inside the isolated environment too, pointed at the restored copy; no connection string outside that environment is changed. With transaction log backups, a database can go back to a chosen moment, to the second, but only if it uses the full or bulk-logged recovery model; otherwise, the choice is between the recovery points of the full and differential backups. Which of the two applies to your database is one of the first things a drill finds out, and it decides how much data a real deletion would cost.

The loss of everything in one place

A server room, a building, a cloud region. This measures whether the copies are elsewhere, and whether elsewhere can run what is needed. For Azure virtual machines, Azure Backup’s cross-region restore, an option enabled separately on a geo-redundant vault, brings them back in the secondary region, the region paired with the primary one. Its documentation states that in the secondary region the RPO can be up to 36 hours in the worst case: a figure to set beside your own target before the drill, not after.

Start with one

For a first drill, one system and one outage are enough: the system the day’s work depends on most, and the outage that would hurt it most.

Preparing a drill that measures, and touches nothing

The preparation has two aims that pull in different directions. The drill must not touch production or the outside world, so the isolation is checked in advance, every time. And the drill must measure what the team really knows, so the backups, who has access and the steps are written down as the team believes them to be, without verifying or correcting them first: finding out is the drill’s job. If someone already knows that something is broken, it is the first entry in the report.

An environment with no way out

The recovered systems run in an environment of their own, with no connection to the systems at work, or to customers, suppliers or the tax authority. A recovered ERP that can reach the outside can send real invoices, orders or emails, or clash with the original.

Site Recovery’s page on test failovers recommends a network isolated from the production network, and notes that by default an Azure virtual network is isolated from other networks. Isolated from other networks does not mean cut off from the internet. Azure’s page on default outbound access explains that a virtual machine deployed with no explicitly defined outbound method is automatically assigned an outbound public IP address, and that subnets are private by default only in networks created in the portal or, from API versions released after 31 March 2026, in new networks created by other means; existing networks are not changed. A private subnet is not sealed off either: the same page notes that machines in it can still reach Azure Storage accounts in the same region, where a recovered system that holds production keys could write, and recommends network security groups to control what leaves. So before anything starts, each subnet of the test network is checked: private, with no NAT gateway, no public load balancer, no load balancer with outbound rules or with a backend pool configured by IP address, no route to a firewall that leads out, no peering or VPN to another network, no public IP address on any machine, and a network security group that blocks outbound traffic except what the restore itself needs. A machine already in the network is stopped and deallocated after its subnet is made private, because the change reaches it only then. For Site Recovery between Azure regions, each replicated machine’s own test failover settings are read too, because they can be set in advance: its test network and, for each network card, resources such as a public IP address, a load balancer or a network security group, or a network card created beforehand, which brings all its own settings with it, a public IP address too if it has one. Whatever would lead outside the isolated network is removed or replaced before the day, in the test failover settings alone: the same screen holds the settings a real failover uses, and those are left as they are. Then, as each recovered system starts and before anyone uses it, its automatic sends are switched off: e-invoices, emails, orders to suppliers.

People and copies still need a way in. The copies arrive by a route that runs only from where they are kept to the test environment. The test failover page says that connecting to a failed-over machine by remote desktop needs a public IP address, or a site-to-site connection; a drill uses neither. People reach the machines from inside the isolated network, through a single jump host and a way in to it that is chosen and checked before the day, and that leads nowhere else.

If on the day anything fails because of the isolation, the jump host included, it is written down as a blocked step, and the person leading the drill decides whether to go on or stop: no address, route or rule is added to make it work. The default outbound access page itself warns that some services, such as Windows activation and Windows Update, do not work in a private subnet with no explicit way out; that too is noted, not fixed on the spot.

The steps a drill never runs

A few actions belong to a real recovery, and never happen in a drill:

  • Azure Backup’s «Replace existing» option, which replaces the disks of the existing machine.
  • Attaching a restored disk to an existing machine.
  • A full Site Recovery failover, whose shutdown step attempts to turn off the source machines.
  • A recovery plan launched whole, even as a test failover, whose scripts may create a public IP address or point at a production database.
  • A cross-region restore launched before its target network is checked.

The first three are why a drill uses the test failover and a restore to a new machine, whose network, and for a restore its subnet, are chosen by hand as the isolated ones: a restore selects the first subnet by default. The fourth is there because Microsoft’s documentation states that a runbook is invoked when the recovery plan runs and is told whether the failover is a test or a production one, so a test of the plan runs its runbooks too; one of Microsoft’s own examples of a plan task is creating a public IP address on the recovered machine. The last is there because, according to Azure Backup’s documentation, Azure Backup configures the virtual network settings in the secondary region automatically, and a cross-region restore job, once triggered, cannot be cancelled: the network it proposes is checked before the job starts.

If there is a written recovery procedure, the drill follows it as written, without improving it, except for the steps that act outside the isolated environment: shutting down or replacing the original, switching the production DNS, telling customers or suppliers, sending anything to the tax authority. Those are not carried out; the time at which they would be is noted. Nor is a recovery plan run as a whole, also because a plan’s test failover does not let you choose each machine’s recovery point: its machines are failed over as the drill log below describes.

People with one job each

Four roles are enough. Someone leads the drill, opens and closes it and decides whether to stop it. The technical team recovers the systems, as it would in a real outage. Someone keeps the time. And someone from the business, who uses the system every day, says when it works. Microsoft’s guidance on disaster recovery plans asks for owners for declaring a disaster, closing an incident, running the operations tasks, testing and validating, and managing communication, among others; a drill needs fewer, each with a name.

A date and a limit

The drill has a date in the calendar, with the people who take part, and a time limit set in advance. If the limit is reached before the system works, the drill stops there. That is a result too, and can be the most useful one.

The drill log: recording the day

The person who keeps the time writes one line for each step as it happens: when it started, when it ended, who did it and what it waited on. The last column is the one that explains the result. A recovery is made of work and of waiting: waiting for a password, a permission, a person, an update, a copy still on its way.

STEPSTARTENDWHOWAITED ONOutage declaredhh:mmThe leadReaching the backupshh:mmhh:mmTechnical teamA permissionRestore jobhh:mmhh:mmThe platformAfter the jobhh:mmhh:mmTechnical teamAn updateBusiness checkhh:mmhh:mmOrder deskRecovery time: from the declared outage to the business check, against the RTO.Data lost, and its worst case: from the recovery point, against the RPO.
The drill log, as an example: one line per step, written as it happens. The last column explains the result.

The declared outage

The clock starts when the person leading the drill declares the chosen outage, at a stated time. From that moment, whatever the outage takes away is treated as gone: no one looks at it to settle a doubt, and nothing is copied from it. If something is needed that exists only there, it is written down as missing. And no backup taken after the declared time is used, nor any recovery point later than it: production keeps running, and its later backups and replication do not exist for the drill. With Site Recovery, the timekeeper also notes at that moment each machine’s latest processed recovery point and its time stamp, which the vault shows on the machine’s page. Site Recovery’s documentation says that when the source region goes down, log processing stops, and the failover must use the latest processed recovery point: in that outage, the one just noted. If the chosen outage takes the primary region, the timekeeper also notes each backed-up machine’s newest recovery point already in the secondary region and its time, which Azure PowerShell lists without starting a restore (Get-AzRecoveryServicesBackupRecoveryPoint with -UseSecondaryRegion); with none, its restore is a blocked step. The restore uses that point even if a newer one, taken before the declared time, arrives later: after a real loss of the primary region, none would.

Reaching the backups

The first lines cover what nobody has practised: finding the copies, getting into the place where they are kept, holding the permission to restore them. Azure Backup’s documentation, for example, reminds you that restoring a virtual machine requires the correct Azure role-based access control permissions.

The jobs, with the platform’s own times

Each restore or failover job gets its line, with the times the portal shows written next to the timekeeper’s. For a Site Recovery test failover, the line also says which recovery point was chosen. The documentation explains the trade-off: «Latest processed» gives a low RTO, because no time is spent processing data, while «Latest» first processes all the data sent to the service and gives the lowest RPO. In a drill, though, only a point from before the declared time is valid: «Latest» creates its point when the test is launched, after the declared time, and the latest processed one can be later too. So the point is chosen with «Custom», which works only for a single machine, not for a recovery plan: the last one before the declared time or, if the chosen outage would stop Site Recovery processing that machine’s data, as the loss of the source region would and a deleted database would not, the one noted at the declared time. Each machine gets its own test failover: the machines of one group of the plan are launched together, and a group only once those of the group before it are running. The log notes that the plan and its scripts did not run. Between Azure regions, points older than two hours may be thinned out according to the retention period set, and with a retention of zero days only the latest point can be used. The line names each point used and how long before the declared time it lies. If the outage would stop processing a machine’s data and thinning has removed its noted point, the newest earlier one is used, and the gap between the two is the drill’s own, which the line notes separately. A machine with no point before the declared time, or in a multi-VM consistency group, where it cannot fail over alone, is a blocked step; in an outage that would stop its processing, the log notes the block as the drill’s own when the consistency group or zero days’ retention caused it.

Everything after the jobs

Then one line for each step of the earlier sections: the extensions, the role assignments, the addresses, the passwords, the directory and the DNS, the start-up order, the connection strings, the automatic sends switched off.

The business check

The clock stops when the person from the business has done what they do every day: found a customer, opened an order, entered a test transaction that stays inside the isolated environment. That time, counted from the declared outage, is the recovery time to compare with the RTO.

The data

With the clock stopped, the timekeeper writes down the time of each recovery point the systems came back from. The data lost on the day runs from each of those points to the declared outage or, where the outage would stop Site Recovery processing the machine’s data or take the primary region, from the point noted at the declared time; what the original recorded in that span would have to be entered again. That loss is compared with the RPO, and so is the worst case, which decides whether the target can be met at all: a failure just before the next recovery point becomes usable, that is, the interval between recovery points plus the time a new one takes to be ready. Microsoft’s 36 hours for a cross-region restore is that sum: with the standard policy, a primary-region RPO of up to 24 hours, and up to 12 hours for the copy to reach the secondary region. So the timekeeper also notes the interval between the recovery points the backup or replication creates, and how long each new one takes to be ready. Anything missing altogether, such as attached documents, settings or files kept elsewhere, gets a line of its own.

Closing down

At the end, the test environment is taken apart. Site Recovery has a cleanup action for test failovers, which deletes the machines created for the test, and its documentation warns that changes made to those machines are lost: anything worth keeping is noted first. The same page asks you to record and save your observations. After an Azure Backup restore, the restored machine and its disks stay until someone deletes them, and the documentation asks for the VHD files the restore leaves in the staging storage account to be deleted. To those add the template the restore wrote there, the disks of a restore that failed, which Azure Backup still restores in the resource group, and the test server’s registration in the vault. All of them hold real data: they are protected like the originals until they are deleted, and nothing is left behind.

Reading the result

The report is short, and written in the days after the drill, while everyone still remembers what happened. It gives the time of each step and the total, compared with the RTO; the data lost on the day and the worst case, compared with the RPO, and, separately, any gap the log notes as the drill’s own, and the list of what was missing altogether; and each step that got stuck, with its cause and the time it cost, and, separately and with nothing to correct, the drill’s own blocks.

The split between work and waiting is the most useful part of it. Work gets shorter with practice and automation. Waiting goes away only when something changes: a password kept where at least two people can reach it even when the systems are down, a permission moved to an account that is still in use, a step written down. Microsoft’s guidance agrees: protect every disaster recovery asset, plans, credentials, certificates and scripts included, keep them accessible during an outage, and keep offline or printed copies for the worst case.

Some durations a drill outside production does not measure. Microsoft’s guidance names DNS propagation among the processes outside your control whose delays should be validated when evaluating recovery times. We add moving the users onto the recovered systems, the time between the start of a real outage and the moment someone declares it and, with Site Recovery, two this drill leaves out by its own choices: the recovery plan’s own run, and the processing «Latest» would add if chosen in a real outage that did not stop processing. Each that applies gets a line of its own in the report, marked as not measured, so that nobody reads the drill’s total as the length of a real outage.

The same guidance says what to do with the numbers: test and verify the RTO and RPO both overall and component by component, separately from full drills; use the metrics from testing as the minimum time needed for each recovery step when scheduling the next drills; and treat test results as inputs to improve. Its example: if a new operator hesitates, review that procedure to make sure it is clearly written.

An example, from start to finish

A logistics company in Chile, imaginary like the examples on our pages, has customers in Argentina and Peru. The systems that receive customer orders and plan the routes of its lorries run in the cloud, and the provider makes automatic backups. Management takes it for granted that a deleted database can be recovered quickly, but no one knows how recent the latest backup is, or who has permission to restore it.

The drill simulates the orders database being deleted by mistake. The targets come first: management says how long order-taking can stop, and how many hours of orders it is prepared to enter again. The isolated environment is a separate network whose only way out is the one the restore needs, closed once the database is back, and it is checked subnet by subnet. The database is restored there, never over the original, and a copy of the order application is brought up beside it. The person leading the drill, two people from the technical team, a timekeeper and one person from the order desk take part.

On the day, the drill log shows where the time went. The longest lines are at the start: the permission to restore belongs to an account no one uses any more, and much of the morning goes on reaching it. The clock stops when the orders can be seen again and the person from the order desk has opened a few and entered a test order, which stays inside the isolated environment. The interval between the recovery points brings the second finding: backups were made less often than the team thought, so the worst-case data loss is larger than the target.

The report gives the times, step by step, and the corrections in order. First the permission, because without it there is no recovery at all; then the backup frequency, because it decides the data loss; then the steps the team did from memory, written down. And a date to run the drill again after the corrections, to check that they work.

After the first drill

A drill tells you how the recovery stands on the day it is run. A new system, a person who leaves or a backup someone reconfigures can change that the following month.

Microsoft’s guidance describes three kinds of practice. Tabletop exercises, in which the team walks through the plan without touching any system, help it practise its roles and train new members. Dry runs outside production, like the drill in this piece, test the recovery without touching the day’s work. And production-level drills are, in the guide’s words, the only way to verify that a plan meets its RTO and RPO targets in real conditions, with a risk the guide names: unexpected and potentially severe failures, which is why it asks for recovery procedures to be tested outside production first.

For Site Recovery, the documentation on recovery plans recommends a test failover for each application every quarter, because applications and their dependencies change frequently. And the guidance asks for the disaster recovery plan to be treated as a living document, reviewed regularly with every team concerned, ideally every six months.

A reasonable rhythm for a company that is starting combines these: a drill outside production on a fixed date, every quarter for an application protected by Site Recovery, and another after each important change to the systems or the team; and a review of the disaster recovery plan every six months, with a tabletop exercise for anyone who has joined since.

Where to start this week

None of these steps needs anything decided first:

  • For the system the day’s work depends on most, write down where its backups are and how often they run.
  • Ask management for the two targets, even rough ones.
  • Ask who knows the backup password and who can restore, and write the answer down.
  • Find the recovery procedure, if there is one, and note its date.
  • Choose the outage to simulate and a date, with the people who would do the work.

How we do it

This is the work of our recovery drill. First we choose with you which systems are part of the drill and which outage is simulated. We read the recovery procedure, if there is one, and see where the backups are and who has access to them. On the day of the drill we recover the systems together with your technical team in a separate, isolated environment, as if the originals were gone. We measure the time until each system works again, and note which data is missing and every step that got stuck; then we write the report with the measured times and the corrections, in order.

The drill does not stop your operations: it takes the time of the people who take part, and an isolated place to recover the systems. The scope and the price are written down before we start, and they do not change afterwards.

If you want to know how long your recovery really takes, start with a free one-hour video call: tell us which systems you have and where their backups are, and we will tell you what we would simulate first.

The sources we cite

What we do about it: Recovery drill

All resources