A backup job runs every night. The dashboard is green. Last successful backup: 02:00.
Now answer one question: How long would it take to rebuild your application somewhere else and prove its data is usable?
If nobody has done it, you don't know.
I've already written about Kubernetes backup architecture and disaster recovery. This post is about the missing operational step: the restore rehearsal.
Design the drill around a real failure
"Can we download a backup file?" is a useful check, but it is not a recovery test. Pick an explicit scenario: the namespace was deleted, the storage volume is gone, or an entire cluster needs rebuilding.
Before you start, define:
- RPO: maximum acceptable loss of data, in time.
- RTO: maximum acceptable time to restore useful service.
- Recovery scope: database, volumes, manifests, secrets, DNS, ingress, identities, and dependencies.
- Success condition: a working customer transaction against the recovered service, not merely a Running pod.
Keep the rehearsal isolated from production, and make sure it cannot write to production databases or send real customer notifications.
A minimal restore exercise
- Choose one representative stateful service and a known backup timestamp.
- Provision an isolated recovery environment with separate credentials and storage.
- Recreate infrastructure and manifests from the documented source of truth.
- Restore application data using a database-consistent mechanism where needed; a raw volume snapshot may not be sufficient.
- Restore necessary secrets and external dependencies through the approved secure process.
- Run application checks: schema compatibility, record counts, login, representative writes, and reads.
- Record elapsed recovery time, missing steps, and what required tribal knowledge.
The actual recovery sequence will differ for managed databases, self-managed PostgreSQL, object storage, and etcd. Don't force all of them into one backup mechanism.
What a failed drill teaches you
Maybe a secret existed only in the old cluster. Maybe the application version in Git cannot read the restored database schema. Maybe your backups live in the same account whose credentials you're assuming are compromised. Maybe the recovery procedure depends on the person who is currently on holiday.
These are good failures to discover in a drill.
Turn the drill into an operating practice
Put a recovery owner on the calendar, retain evidence of the test, and track the measured recovery time against the business target. Repeat after major storage, database, or deployment changes—not just after incidents.
Discussion: When did your team last complete a full restore, and was it anywhere close to your stated RTO?
If recovery has never been tested, a Kubernetes Cluster Audit can identify the gaps before a scoped recovery exercise. You can also request a discovery call.