Masterplan Optimiser
Root & controller

Operator Guide

HA Operations & Failover

Operate the pair from the current holder, prove every ownership change, and keep the former primary fenced until it has accepted a current copy.

Daily checks

Browser

Open the root High Availability tab. Confirm the holder, peer, generation, matching release, last complete copy, recovery-point age, peer reachability, snapshot status and recent incident history.

MP-OPT

Use High availability → Show lease, peer and replication state for the protected infrastructure view. UseSend a complete verified copy to the peer now when an immediate checkpoint is required.

Planned switchover

  1. Confirm both nodes are healthy, on the same commit and have matching recovery-recipient fingerprints.
  2. From the current holder select Replicate and hand ownership to the peer.
  3. The holder creates and transfers a fresh bundle before requesting the lease handoff.
  4. Require the peer /ha/ready to become 200 and the old holder to become 503.
  5. Confirm public health and root login, then create a durable test write.
  6. Verify the old node rejects writes and the web dashboard records the handoff.

Automatic failover

Enable it only after a full commissioning pass. The action is available on the live holder and requires the exact confirmation phrase. A sudden holder outage does not promote the peer before the configured five-minute safety delay.

After promotion, verify public recovery, the incremented generation, Cloudflare pool order, invalidation of old sessions/links and a new application write. When the former primary returns, it stays fenced until it accepts the current generation.

Peer unavailable

The holder continues serving reads and writes. Replication becomes degraded and recovery-point age grows. Do not perform a planned handoff. Restore SSH/network reachability, send a manual complete copy and require the receiver hash before considering the pair protected again.

Witness unavailable

The last holder continues serving reads but new writes fail HTTP 503 because no fresh permit can be proven. Do not bypass the permit or promote manually. Restore the Worker/Durable Object path, verify the same holder/generation, and then confirm writes recover.

Maintenance and upgrades

  1. Disable automatic failover.
  2. Create, deep-verify and export a fresh complete recovery snapshot.
  3. Deploy the non-holder, then the holder.
  4. Require both nodes to report the same clean Git commit.
  5. Send a complete copy, then switch ownership in both directions.
  6. Re-enable automatic failover only after readiness and application checks pass.