Active HA Verification

• Full commissioning: destructive; use only on a disposable two-node pair. It proves database, secrets and full restores, a clean rebuild, fencing, planned handover and real automatic failover in both directions.
• Operational smoke: non-destructive; use on a live pair after deployment or maintenance. It proves health, one-writer routing, immediate replication and snapshot create/deep-verify/export, but never wipes, restores, powers off or hands ownership to the peer.

conda activate mp-opt-soak-test
cd /d "C:\path\to\masterplanOptimiserV3---Server-Public"
python deploy\ha\ha_verification.py --config deploy\ha\ha-verification.local.json init
python deploy\ha\ha_verification.py --config deploy\ha\ha-verification.local.json

The local configuration stores only the public URL, SSH aliases and discovered public node metadata. Secret-, password-, token- and private-identity-like fields are rejected.

Continue shows exactly one checkpoint, its location and its exact completion phrase. Automated probes run again before acceptance. Status, Report, Verify evidence and Abort are safe across workstation restarts. There is no background monitor to keep alive.

1. Preflight proves SSH, TLS, public/origin health, lease ownership, exactly one ready writer, matching releases, clocks, services, audit chains, recovery recipients and replication receipts.
2. Create disposable application data and prove root login, publishing and the root HA panel.
3. Set replication temporarily to five minutes and wait for a genuinely scheduled copy—do not trigger it manually—then restore the desired interval.
4. Create, deep-verify and export database, secrets and full snapshots; run wrong-key, corrupted-package and stale-transfer negative controls.
5. Restore database-only and secrets-only snapshots, then wipe the disposable database and restore the full snapshot.
6. Prove portable recovery on a clean standalone rebuild and run the isolated recovery self-tests.
7. Prove peer-loss behavior and witness/lease-loss write fencing.
8. With automatic failover still disabled, power off the current holder in its provider panel. The peer must remain fenced for the complete 360-second observation window; then power the original holder on and require its unchanged generation to recover.
9. Perform a guarded planned handover to the peer and back.
10. Enable automatic failover only after the preceding recovery evidence passes.
11. In the current holder's provider panel, power it off. Promotion must not occur before 300 seconds and routing/public recovery must complete no later than 660 seconds.
12. Power the former holder on, prove it remains fenced, and replicate current state back to it.
13. Repeat the real provider power-off in the opposite direction. Both directions must pass.
14. Restore the intended holder, interval and healthy baseline; generate and verify the final evidence manifest.

Use the smoke profile on production. It verifies public and origin health, one-writer readiness, both SSH paths, lease consistency, matching releases, the root HA panel, an immediate peer-accepted replication, and snapshot create/deep-verify/export. It deliberately performs no restore, wipe, handover, witness manipulation or provider power-off.

A smoke result is labelled OPERATIONAL_SMOKE_PASS. It does not claim disaster recovery or automatic failover was re-proven.

The coordinator never reads the private recovery identity. When a checkpoint needs it, SSH interactively to the named node, open MP-OPT, choose the specified deep-verify or restore action, and paste the identity only into the hidden prompt. Return to the coordinator afterwards.

For exact snapshot behavior, read Recovery Snapshots. For operating the pair after commissioning, read HA Operations.