How do you test a restore? RTO and RPO in practice

Two acronyms that decide your night

RPO, recovery point objective: how much data may you lose at most? A nightly backup gives an RPO of 24 hours: a disaster at 4 pm means redoing the whole working day. For a bookkeeping system that is annoying, for a web shop it is unaffordable.

RTO, recovery time objective: how long may it take before everything runs again? Not the moment the backup is "downloaded", but the moment your people can work again.

Both numbers are often in a policy document. Far less often they have been measured. And an estimated RTO is a guess.

Why backups fail in practice at restore time

• The backup ran, but the path was empty or a database was skipped.
• The repository looked healthy on the source server, but is unreadable on a fresh machine because the local cache masked everything.
• The dump is there, but nobody ever tried to load it into a newer database version.
• The key sits on the server that has just been encrypted.
• Restoring 600 GB over a connection with egress fees and 100 Mbit takes three days, and costs money nobody budgeted.

Each of those problems only surfaces when you really try. That is why you test.

Four levels of testing

Not every test needs to be complete. Zekura uses four levels, each with its own frequency, and that model works without Zekura too:

Level	What	Question answered	How often
L1 Integrity	read back every object of the snapshot on a separate machine, with an empty cache, check all checksums	Is everything there, and readable from scratch?	after every backup
L2 Sample	restore a sample of files and compare by sha256 with the original	Does what comes back match what was there?	daily
L3 Application	load the database into a disposable instance and count; start the application; run a health check	Does it work, not just are the bytes right?	weekly
L4 Boot	restore a full VM and boot it until the guest agent answers, and time it	How long does it really take until it runs?	monthly

The empty cache in L1 is not a detail. A restore always happens on a machine that does not know the data yet. Testing from the source proves nothing about that.

How to really measure RTO

Measure the whole journey: from the moment you decide to recover to the moment the application answers. Note per system:

• the duration of the restore (bytes per second says more than you think about your network and your storage provider);
• the time to start a database and check it;
• the time to restore access (DNS, certificates, passwords that were in the backup).

Zekura records the measured duration of every restore and every trial restore and puts it in the evidence chain. The overview always shows the last proven restore with its time: "9 seconds" or "11 minutes 42 seconds" is a fact an auditor can check, "about an hour" is not.

How to shrink RPO without a full backup every hour

For databases there is point-in-time recovery: a weekly base backup plus a continuous stream of changes (the write-ahead log for PostgreSQL, the log backups for SQL Server). The interval at which that log leaves is your RPO: every five minutes gives at most five minutes of loss. And you can go back to any point in time, including 2:06 pm, just before someone emptied the wrong table.

Test results that count as evidence

A test that gets an "ok" in a spreadsheet is of little value to an auditor. What counts:

1. Who or what performed the test, and on which machine (not the source server).
2. When, with a timestamp that cannot have been adjusted afterwards.
3. What exactly was checked: number of objects, number of files, row counts.
4. The result, including the failures. A ledger that only contains successes is suspicious.
5. The link: every test refers to the backup it applied to.

At Zekura every verification goes signed into the evidence chain, with the evidence attached. Failed verifications too; the ledger hides nothing.

A test plan for a small business

• Daily, automatic: integrity of every backup, sample of files.
• Weekly, automatic: fully restore and check one database and one application.
• Monthly, automatic: boot one VM from backup.
• Quarterly, with people: a guided recovery exercise of one scenario (ransomware, fire, employee mistake), with a report. Zekura's SLA Business includes such an exercise every quarter.
• Yearly: test the key. Without the key every other test is pointless.

Frequently asked questions

Isn't a trial restore dangerous for production?

Only if you restore over production. Always test to a separate machine or a disposable instance. Zekura never restores over the original unless you explicitly ask for it.

Does testing cost a lot of storage traffic?

At providers that charge for outbound traffic it does, and that is exactly why so little testing happens there. Choose storage without egress fees; that is why Zekura uses Cubixion.

What is a good RTO for a small business?

You answer that per system with your people: what does an hour of downtime of the web shop, the bookkeeping, the mail cost? Then measure whether you achieve it. Often the actual RTO turns out to be a multiple of the estimate.