Operator Guide
High Availability & Redundancy
MP-OPT uses two symmetric VPSs, one external writer lease, provider-neutral SSH replication, and Cloudflare routing so either healthy node can become the primary.
Architecture
Symmetric two-VPS high availability
The witness grants ownership. The load balancer only sends traffic to the node that can prove it is ready.
Cloudflare Load Balancer
Public HTTPS traffic and /ha/ready health checks
VPS A
Application, PostgreSQL, local node identity
VPS B
Application, PostgreSQL, local node identity
Durable Object witness
One writer lease, generation, heartbeat, and short write permits
Independent external snapshot
Encrypted recovery copy plus an operator-held private identity kept away from both VPSs
Either VPS can become holder. The labels A and B identify nodes, not permanent primary and secondary roles.
- A Cloudflare Durable Object serialises ownership decisions and holds one writer lease.
- The lease holder serves reads and writes. The peer remains alive but returns not-ready.
- The holder sends encrypted complete point-in-time copies to the peer over authenticated SSH.
- The peer verifies the bundle, restores it into a staging database and atomically accepts it only after witness authorisation.
- Cloudflare Load Balancing routes public traffic to the holder only after that node reports ready.
Write fencing
The holder heartbeats every 15 seconds and receives short-lived write permits. FastAPI checks a permit before each mutating request and SQLAlchemy checks again before ORM commits. If the witness cannot be reached, writes return HTTP 503 instead of risking two writers.
/health proves the process and local database are alive. /ha/ready additionally proves current ownership and generation. Cloudflare pool monitors must use /ha/ready.
Replication and accepted data loss
The root administrator can choose a whole-minute interval from 5 to 1440 minutes and can request a copy immediately. The default is 15 minutes.
If the peer is unavailable, the current holder keeps serving and retains the last accepted recovery point on the peer. A later failover can lose every write made after that point. The web HA tab shows the last success, recovery-point age, peer reachability and current job.
Automatic failover
Automatic failover starts disabled. When enabled, the witness waits approximately five minutes without the holder heartbeat. It promotes the peer only when the peer is healthy, has the complete accepted bundle for the current generation, runs the same Git release and has no active transfer.
Promotion increments the generation, invalidates replayable access, restores local ownership state, waits for readiness and then updates Cloudflare routing. Registered passkeys and durable application data remain; sessions, activation/reset links, public schedule links and desktop publishing credentials are invalidated.
Commissioning proves this with a real provider power-off in both directions. There is no artificial multi-day waiting requirement; continued confidence comes from alerts, normal use, the root HA panel and periodic non-destructive operational smoke checks.
Automatic failover sequence
Promotion is deliberately gated to prevent both servers from accepting writes.
1
Heartbeat stops
The current holder disappears
2
Witness waits
Approximately five minutes
3
Peer qualifies
Healthy, matching release, accepted current bundle
4
Generation advances
Old write authority becomes invalid
5
Peer promotes
Database and local ownership activate
6
Routing follows
Cloudflare sees /ha/ready
What this protects—and what it does not
- It protects availability after one VPS or provider fails.
- It prevents split-brain writes while the witness is authoritative.
- It does not provide zero data loss or synchronous database replication.
- A total Cloudflare Worker outage pauses writes.
- Compromise or corruption on the holder can be copied to the peer.
- Node-local HA credentials are not part of normal snapshots.
Continue with HA Setup, then complete Active HA Verification before enabling automatic failover.