Masterplan Optimiser
Root & controller

Operator Guide

Disaster Recovery

Recover from a lost peer, a lost holder, or complete loss of both VPSs without cloning node identities or reviving stale access.

Choose recovery by failure, not by convenience

Diagnose which trust or data boundary was lost before taking a mutating action.

What was lost?

Service, host, authority, data, or recovery material

One peer is lost

Keep serving, rebuild the lost peer, then reseed it

The holder is lost

Use verified failover only if the peer recovery point is acceptable

Both VPSs are lost

Rebuild from the external snapshot and its private recovery identity

Witness is unavailable

Keep reads available; do not bypass fail-closed writes

Root passkey is lost

Use guarded root recovery, then rotate affected access

Private recovery key is lost

Existing encrypted snapshots cannot be decrypted

One peer is lost

  1. Confirm the surviving holder still owns the lease and serves public traffic.
  2. Disable automatic failover before changing cluster infrastructure.
  3. Provision the replacement VPS at the same tested server commit.
  4. Create fresh node-local HA identity/TLS material; never copy the survivor's /etc/mp-opt-ha.
  5. If the original lost node token/identity cannot be recovered safely, commission a fresh witness cluster ID rather than impersonating it.
  6. Send and verify a complete copy, then test a planned switchover both ways.
  7. Repeat the required certification gate before enabling automatic failover.

The holder is lost and automatic failover is enabled

Do not manipulate DNS while the failover timer runs. Observe the witness and public endpoint. The peer may promote only after the safety delay and only with the accepted current-generation bundle.

  1. Confirm the generation increments and the peer becomes ready.
  2. Confirm the old pool is no longer preferred.
  3. Log in with a registered passkey and create a durable test write.
  4. Expect old sessions, links and desktop publishing credentials to fail.
  5. When the former holder returns, prove it remains fenced, then replicate current state back to it.

Complete loss of both VPSs

  1. Provision fresh VPS A and deploy the tested release in standalone mode.
  2. Configure the original domain, WebAuthn RP ID and origin so registered passkeys remain compatible.
  3. Derive and install only the public recovery recipient.
  4. Route the hostname to VPS A and install fresh TLS.
  5. Import the portable package through MP-OPT and provide its package SHA-256.
  6. Deep-verify it with the matching private identity.
  7. Restore it with rollback protection and require public health/readiness 200.
  8. Verify root login and durable data. Regenerate every desktop publishing secret.
  9. Create, deep-verify and export a fresh complete baseline.

The recovered application is intentionally standalone at this point.

Rebuild the two-node pair

  1. Provision VPS B and deploy exactly the same clean commit.
  2. Create fresh node-local HA identities, witness tokens and Origin CA material.
  3. Create a new cluster ID; a previously bootstrapped lost cluster cannot be bootstrapped again.
  4. Update/recreate the Cloudflare pools for the new addresses.
  5. Configure the same public snapshot recipient on both nodes.
  6. Bootstrap the new cluster with recovered VPS A as initial holder.
  7. Replicate to B and require matching generation, release, bundle ID/hash and recovery-recipient SHA-256.
  8. Require A /ha/ready 200 and B 503, then switch A → B → A.
  9. Repeat full active commissioning, including snapshot recovery and real automatic failover in both directions, before enabling unattended operation.

See HA Setup for commissioning and Active HA Verification for the acceptance gate.