Automated Failover & Quorum Recovery in Galera

Galera has no elected primary to fail over to — every node is already a writable member — so “failover” in a MariaDB Galera Cluster means something different from a primary/replica database: it is the process by which the surviving nodes recompute quorum, keep the writable primary component alive, and either heal automatically or hand a deterministic recovery decision to your automation. This operational guide, part of MariaDB Galera Core Architecture & Fundamentals, turns that process into runbook-grade procedure: the quorum arithmetic that decides who stays writable, what happens when the primary component is lost, how pc.weight biases a tie, when recovery is safe to automate versus when it must be manual, how to pick the node that must bootstrap, how load balancers and proxies route around a failed member, and the Python that detects a non-Primary node and drives it back into service. The goal is a Galera cluster that survives a single-node or single-zone loss with zero operator action, and a documented, non-destructive path for the failures that genuinely require a human.

The Concept: Quorum, the Primary Component, and What “Failover” Means #

A Galera group is only writable while the members that can see each other hold a quorum — a strict majority of the total weight of the last agreed membership. The set of connected members that holds that majority is the primary component (PC); it continues to certify and commit writes. Any partition that finds itself in the minority transitions to non-Primary, refuses writes with WSREP has not yet prepared node for application use, and waits to rejoin. This is deliberate: by allowing only one partition to remain writable, Galera makes split-brain arithmetically impossible under default settings. The trade is that a Galera cluster which loses quorum stops accepting writes entirely rather than risk divergence — availability is sacrificed to preserve consistency, the same certification contract described in Understanding Galera Synchronous Replication.

The arithmetic is simple but unforgiving. Each member carries a vote weight (pc.weight, default 1). After a view change, a component is primary only if the sum of the weights it can see is strictly greater than half the weight of the previous primary component. In a three-node cluster of equal weight, any two nodes (weight 2 of 3) form a majority and stay writable; a lone survivor (weight 1 of 3) goes non-Primary. This is exactly why an odd node count matters and why a two-node cluster is a trap — lose either node and the survivor is a minority of one.

Prerequisites & Environment #

Automated recovery only works if the Galera cluster is built to make quorum decidable. Before wiring any of the automation below, confirm the baseline:

• An odd number of voting members — three full nodes, or two full nodes plus a Galera arbitrator (garbd) — so a single failure can never produce an even split. Node placement and weighting are laid out in Designing Multi-Master Topologies.
• pc.ignore_sb=false on every node (the default). Leaving split-brain protection disabled defeats the entire quorum model.
• A sized gcache so that a node returning after a short partition rejoins with a fast Incremental State Transfer rather than a full snapshot — the ring-buffer sizing lives in the wsrep.cnf Configuration Deep Dive.
• A monitoring account (GRANT PROCESS, REPLICATION CLIENT) reachable from your automation host, plus reachability to mysqladmin/mariadbd on each node for --wsrep-recover.
• A load balancer or proxy (MaxScale or HAProxy) in front of the Galera cluster that health-checks each node so client failover is automatic even while the group is still healing.

Step-by-Step: The Recovery Procedure #

The procedure below is the canonical sequence for a full or partial outage. Automate the safe steps; keep the destructive one (forcing a bootstrap) behind a deliberate gate.

Step 1 — Classify the failure: partition versus total loss #

The first decision is whether a writable partition still exists. Query any reachable node:

SHOW GLOBAL STATUS WHERE Variable_name IN
  ('wsrep_cluster_status', 'wsrep_cluster_size', 'wsrep_local_state_comment');

If any node reports wsrep_cluster_status = Primary with a wsrep_cluster_size equal to a majority of the intended count, quorum is intact — the failed nodes simply need to rejoin, and no bootstrap is required. If every reachable node reports non-Primary, the primary component is gone and you must reconstruct it from the survivor with the most complete state. Never bootstrap while a Primary partition still exists; doing so creates a second component and diverges the data.

Step 2 — When quorum holds: let the minority rejoin automatically #

A node that lost connectivity but is otherwise intact rejoins on its own the moment the network heals. Confirm it transitions Joining → Joined → Synced and that it took the cheap path:

# On the returning node, watch the state machine converge
mariadb -N -B -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';"
# Grep the error log to confirm IST, not a full SST
grep -E 'IST (received|receiver)|State transfer' /var/log/mysql/error.log | tail

This is the common, fully-automatable case: no operator action beyond restoring connectivity. Startup-log signatures for a rejoin that goes wrong are catalogued in Handling Galera Startup Errors & Logs.

Step 3 — When quorum is lost: find the highest-seqno survivor #

The node that must bootstrap the new primary component is the one holding the most-committed transactions — its state is a superset of every other survivor’s, so bootstrapping any other node would silently discard writes. Read each candidate’s last committed sequence number without starting replication:

# On each stopped survivor, recover the last committed seqno from the log
mariadbd --wsrep-recover 2>&1 | grep 'Recovered position'
# → WSREP: Recovered position: 6f2a...:184230

Alternatively, read grastate.dat directly — but a node shut down uncleanly writes seqno: -1 there, which is exactly why --wsrep-recover (which replays the InnoDB redo log to derive the true position) is authoritative. Compare the Recovered position across all survivors; the largest seqno wins.

Step 4 — Bootstrap the winner #

Bootstrap only the highest-seqno node, and only after confirming no Primary partition exists elsewhere:

# On the highest-seqno survivor ONLY
galera_new_cluster            # systemd wrapper: starts mariadbd with --wsrep-new-cluster
# Verify it is now a Primary component of size 1
mariadb -N -B -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';"   # → Primary

galera_new_cluster sets --wsrep-new-cluster, which tells the node to form a fresh primary component instead of searching for an existing one. It also rewrites safe_to_bootstrap semantics: MariaDB stores a safe_to_bootstrap: 1 flag in grastate.dat on the last node to leave a controlled shutdown, and refuses --wsrep-new-cluster on a node flagged 0 unless overridden — a guard that stops you from bootstrapping the wrong survivor.

Step 5 — Rejoin the remaining nodes #

Start the other survivors normally (systemctl start mariadb). Each contacts the freshly bootstrapped node, is certified into the group, and pulls the delta by IST if gcache still holds it or by full SST otherwise. Watch wsrep_cluster_size climb to the expected count and every node settle on Synced.

Parameter Deep-Dive: The Quorum Knobs #

pc.weight is the lever that turns an even-node or geographically-skewed layout into a decidable one. Give the node (or garbd instance) in your reference zone a weight of 2 and the Galera cluster keeps a writable majority even when a symmetric split would otherwise deadlock. pc.recovery is what lets a three-node cluster that lost power in all three machines re-form its primary component automatically once the nodes boot and re-establish the last-known view — without it, a full power event always requires a manual bootstrap. The low-latency variants of the EVS timeouts are compared in Configuring wsrep_provider_options for Low Latency.

Verification & Health Checks #

After any failover event, prove the group actually reconverged before you trust it with traffic. The status surface tells the whole story:

SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_cluster_size',          -- must equal the intended voting count
  'wsrep_cluster_status',        -- must be 'Primary' on every node
  'wsrep_local_state_comment',   -- must be 'Synced'
  'wsrep_ready',                 -- must be 'ON'
  'wsrep_cluster_conf_id',       -- identical on all nodes = same view
  'wsrep_evs_state'              -- must be 'OPERATIONAL'
);

wsrep_cluster_conf_id is the underused check: it is the monotonically-increasing view identifier, and if two nodes report different values they are in different components — a split you must resolve before routing writes. From the shell, a quick liveness gate for a health-checking proxy:

#!/usr/bin/env bash
set -euo pipefail
read -r size status state < <(mariadb -N -B -e \
  "SHOW GLOBAL STATUS WHERE Variable_name IN \
   ('wsrep_cluster_size','wsrep_cluster_status','wsrep_local_state_comment');" \
  | awk '{printf "%s ", $2}')
[[ "$status" == "Primary" && "$state" == "Synced" && "$size" -ge 2 ]] \
  || { echo "FAIL size=$size status=$status state=$state" >&2; exit 1; }
echo "OK: $size-node Primary, Synced"

Automation Integration: Detect and Recover a non-Primary Node #

The highest-value automation is a supervisor that watches every node, distinguishes a healthy minority-waiting-to-rejoin from a genuine total loss, and only escalates to a bootstrap when no Primary partition exists anywhere. The following controller targets Python 3.9+ with PyMySQL, reads each node’s status, and handles the wsrep contention error codes 1213 (deadlock/certification conflict) and 1205 (lock wait timeout) so a busy-but-healthy node is never misread as down.

import sys
import subprocess
import logging
from dataclasses import dataclass
import pymysql

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")

@dataclass
class NodeState:
    host: str
    reachable: bool
    status: str = ""          # wsrep_cluster_status: Primary / non-Primary
    state: str = ""           # wsrep_local_state_comment: Synced / ...
    size: int = 0
    seqno: int = -1           # last committed, from wsrep_last_committed


def probe(host: str, creds: dict) -> NodeState:
    """Read a node's quorum-relevant status without ever writing to it."""
    try:
        conn = pymysql.connect(
            host=host, user=creds["user"], password=creds["password"],
            connect_timeout=4, read_timeout=4,
        )
    except pymysql.err.OperationalError as exc:
        logging.warning("%s unreachable: %s", host, exc.args)
        return NodeState(host, reachable=False)

    try:
        with conn.cursor() as cur:
            cur.execute(
                "SHOW GLOBAL STATUS WHERE Variable_name IN "
                "('wsrep_cluster_status','wsrep_local_state_comment',"
                " 'wsrep_cluster_size','wsrep_last_committed')"
            )
            s = {name: val for name, val in cur.fetchall()}
    except pymysql.err.OperationalError as exc:
        # 1213 = certification deadlock, 1205 = lock wait timeout: transient, not down.
        if exc.args and exc.args[0] in (1213, 1205):
            logging.info("%s contended (%s); treating as reachable", host, exc.args[0])
            return NodeState(host, reachable=True, status="unknown")
        raise
    finally:
        conn.close()

    return NodeState(
        host, reachable=True,
        status=s.get("wsrep_cluster_status", ""),
        state=s.get("wsrep_local_state_comment", ""),
        size=int(s.get("wsrep_cluster_size", 0) or 0),
        seqno=int(s.get("wsrep_last_committed", -1) or -1),
    )


def recovered_seqno(host: str) -> int:
    """For a stopped node, derive the true last-committed seqno via --wsrep-recover."""
    out = subprocess.run(
        ["ssh", host, "mariadbd --wsrep-recover 2>&1 | grep 'Recovered position'"],
        capture_output=True, text=True, timeout=60,
    ).stdout
    # e.g. 'WSREP: Recovered position: <uuid>:184230'
    return int(out.rsplit(":", 1)[-1]) if ":" in out else -1


def decide(nodes: list[NodeState]) -> str:
    """Return the recommended action; NEVER bootstraps while a Primary partition exists."""
    if any(n.reachable and n.status == "Primary" for n in nodes):
        stragglers = [n.host for n in nodes if not (n.reachable and n.status == "Primary")]
        return (f"QUORUM OK — Primary partition alive. Rejoin: {stragglers or 'none'}")

    reachable = [n for n in nodes if n.reachable]
    if not reachable:
        return "TOTAL LOSS — every node down; recover seqno on each survivor before bootstrap"

    # All reachable nodes are non-Primary: a bootstrap is required. Pick the winner.
    winner = max(reachable, key=lambda n: n.seqno)
    return (f"NO QUORUM — bootstrap ONLY {winner.host} "
            f"(highest seqno {winner.seqno}); then start the rest normally")


if __name__ == "__main__":
    creds = {"user": "monitor", "password": "REDACTED"}
    hosts = sys.argv[1:] or ["10.0.1.11", "10.0.1.12", "10.0.1.13"]
    states = [probe(h, creds) for h in hosts]
    for st in states:
        logging.info("%s reachable=%s status=%s state=%s size=%s seqno=%s",
                     st.host, st.reachable, st.status, st.state, st.size, st.seqno)
    print(decide(states))

The design principle is that the controller recommends and only auto-executes the non-destructive branch (restarting a stopped node so it rejoins a live Primary). The bootstrap branch prints an instruction rather than firing galera_new_cluster, because forcing a bootstrap is the one decision that can lose data if the seqno comparison was made against an incomplete set of survivors. For fleets, wrap this in the same idempotent provisioning flow used for automating node provisioning with Ansible, and gate the bootstrap behind a manual approval step.

Client-side failover: proxies and load balancers #

While the group heals, clients must be steered to a writable node with zero application awareness. Two patterns dominate:

MaxScale understands Galera natively. Its galeramon monitor reads wsrep_local_state_comment and wsrep_cluster_status on each backend, marks any non-Synced or non-Primary node as down, and — with readwritesplit — routes writes to a single chosen Master while spreading reads. Because it speaks the wsrep status surface directly, it demotes a node that has gone non-Primary within one monitor interval:

[galera-monitor]
type=monitor
module=galeramon
servers=node1,node2,node3
user=maxmon
password=REDACTED
monitor_interval=1000ms
available_when_donor=false
disable_master_failback=true

HAProxy has no Galera awareness, so you give it one with an HTTP health endpoint. clustercheck (shipped with the Galera packages) or a tiny xinetd/systemd-socket service returns 200 only when the node is Synced and Primary, and 503 otherwise, so HAProxy ejects a non-Primary node from the pool automatically:

backend galera_write
    option httpchk GET /
    http-check expect status 200
    # single active writer avoids cross-node certification conflicts
    server node1 10.0.1.11:3306 check port 9200 inter 1s rise 2 fall 2
    server node2 10.0.1.12:3306 check port 9200 inter 1s rise 2 fall 2 backup
    server node3 10.0.1.13:3306 check port 9200 inter 1s rise 2 fall 2 backup

Routing all writes to a single active member (with the others as backup) is the recommended pattern even though Galera is multi-master: it sidesteps cross-node certification conflicts on hot rows while still failing over instantly when the active node drops. The read-routing counterpart, and when to spill reads to an async tier, is covered in Fallback Routing & Read-Only Nodes.

Troubleshooting #

A node that reaches Primary but is stuck in Joining/Donor rather than Synced is a state-machine problem, not a quorum one; the recovery steps are in Fixing wsrep_local_state_comment Issues.

Related #

• Recovering from a Non-Primary Component — the exact bootstrap-versus-wait decision and the pc.bootstrap runtime path
• Configuring a Galera Arbitrator with garbd — a voting, dataless tie-breaker for a third availability zone
• Galera vs Async Replication: Failover Trade-offs — how synchronous certification failover compares to classic primary/replica failover
• Designing Multi-Master Topologies — node placement, AZ layout, and pc.weight planning
• Handling Galera Startup Errors & Logs — reading the provider log lines a recovery produces