A credential vault holds the keys to everything else, so its disaster recovery plan has to work when everything else is down. If the platform is gone and the only copy of the credentials was inside it, you do not have a recovery plan. You have a rebuild project with no passwords.
This article covers what to back up, why a backup needs its own protection, how to restore safely, and what to rehearse.
What can go wrong
- The database is lost or corrupted. Hardware failure, ransomware, an operator error.
- The site is lost. A fire, a cloud region outage, a network failure that outlasts your tolerance.
- A bad change. An upgrade or migration damages the vault, and you need to return to the last good state.
- The people leave. The one person who knew the master secret is unavailable.
- The backup is stolen. A backup is a complete copy of every credential, so it is as sensitive as the vault.
A plan has to cover all five.
Principles for backing up a vault
- Seal the backup. A backup must be encrypted so that holding the file is not enough to read it. If the file is stolen, the thieves should have nothing usable.
- Verify it when it is created. A backup you have never opened is a hope, not a backup. Check that the file can be read before reporting success.
- Get it off the box. A backup on the machine it protects fails along with the machine. Copy it to separate storage, ideally in another location.
- Back up on a schedule that matches the loss you can accept: daily for production, plus before every upgrade.
- Back up the rest of the platform too. The vault database is not enough. You also need the service databases, configuration and TLS material, the secrets that unlock the platform, and session recordings.
- Back up from a replica where you have one, so backups do not load the primary database.
Two secrets, two different failures
Sealed vault backups depend on two separate secrets, and operators routinely confuse them:
| Secret | What it protects | If you lose it |
|---|---|---|
| Backup passphrase | The backup file itself | The file cannot be opened by anyone, but the live vault keeps running |
| Vault master secret | The credential values inside the vault | Every credential, in every backup and in the live database, is permanently unreadable |
Store them separately, with different people. A backup and its passphrase in the same drawer are one theft away from every privileged credential you own.
The second row is the important one. Treat the vault master secret as a recovery secret in its own right: split it across custodians, store the pieces offline, and test that the recovery works. Any design where "nobody can recover it" is a risk, and any design where "one person can" is a different risk.
Restoring safely
A restore replaces the vault. Do it with the same care as a surgery:
- Decide the path. If the platform is running and you know the master secret, restore in place. If the platform is gone and you know the master secret, recover offline, then rebuild. If the master secret is lost, the credentials cannot be recovered, and the only path is to rebuild the vault and rotate every credential at its target.
- Verify the secret before touching anything. A good restore checks the master secret against the backup before writing a single row, so a wrong password changes nothing.
- Make it atomic. The restore should run as one transaction, so a failure midway leaves the existing vault exactly as it was.
- Require a deliberate confirmation. Asking the operator to type a phrase such as "REPLACE THE VAULT" prevents accidents.
- Restrict who can do it. Only senior security roles, with an audit record.
- After the restore, check. Service health, login, a sample privileged session, and a sample credential checkout.
Offline recovery: when the platform is gone
The hardest case is when nothing is running. You need a way to read the backup without the platform:
- A small standalone tool that can verify a backup, list what it contains (names only, no values), retrieve one credential, or recover everything.
- Operations that need only the backup passphrase (verify, list) should be separated from those that also need the master secret (reveal a credential).
- "Recover everything" should be deliberately awkward to run, since it exposes every secret at once.
Keep a copy of the tool, and its build instructions, with the offline copies of the backups.
What a vault backup does not cover
Be explicit, because this is where plans fail:
- Session recordings. Large, and usually stored separately. They need their own backup and retention.
- Other service databases. Identity, policy, audit and resource data live in their own databases.
- Configuration and TLS material. Needed to bring the platform back up.
- Encryption keys used elsewhere in the platform. Without them, other encrypted data cannot be read.
A complete plan lists each one, who owns it, and where its backup lives.
RPO and RTO in plain terms
- RPO (recovery point objective) is how much data you can afford to lose, measured in time. Daily backups mean up to a day of changes can be lost.
- RTO (recovery time objective) is how long you can be without the service.
Replicated databases reduce RPO toward zero and shorten RTO considerably, but they do not replace backups: a replica copies a bad change or a deletion just as faithfully as a good one.
Quarterly drill
A plan nobody has rehearsed will fail on the day. Once a quarter:
- Restore to a clean environment from the latest off-box backup, not from the primary.
- Do an offline retrieval of at least one real credential. Verification alone passes even when the master secret is wrong; only a retrieval proves both secrets work.
- Time it. Compare against your RTO.
- Record what broke and fix the runbook.
- Rotate any credential that appeared on screen or in a transcript during the drill.
- Rehearse the worst case: who does what when the master secret holder is unavailable.
How ZenXPAM supports this
ZenXPAM creates sealed, passphrase-encrypted vault backups, opens and verifies each one before reporting it created, and scopes backups to the calling tenant. Backups can be created, verified, downloaded, uploaded and restored from the console or the API. A restore checks the master secret against the backup before changing anything, runs in a single transaction, and requires an administrator role and a typed confirmation. A standalone offline tool, zenxpam-recover, can verify a backup, list its contents by name, retrieve one credential or recover everything, without the platform. Backups persist on Kubernetes volumes, every backup, verification and restore is written to the audit trail, and neither the passphrase nor the master secret appears in logs. For the rest of the platform, there are database backup scripts that can back up from the streaming replica and upload to cloud storage, a restore script, application rollback to a previous release tag, and a high-availability topology with PostgreSQL replication and quarterly failover-drill guidance. Treat the vault master secret as a recovery secret and escrow it, as described above. See the platform or book a demo.
FAQ
Is a database replica a backup? No. A replica protects against hardware failure but copies mistakes and deletions. You need both.
How often should we back up the vault? Daily for production at a minimum, plus before every upgrade, with a restore drill each quarter.
What if we lose the master secret? The credentials cannot be read from the vault or any backup. You would rebuild and rotate every credential at its target, which is why the secret deserves escrow and a tested recovery.
Why not store the backup passphrase with the backup? Because then stealing the file is enough. Keep them separately, with different custodians.
Do we need to back up session recordings? Yes, if you need them for audits or investigations. They are not part of the vault backup.