Troubleshooting SST Donor Selection

When a joiner triggers a State Snapshot Transfer, Galera elects one existing member to be its donor, and a poor election — the busiest node, a cross-AZ peer, or a node that then refuses to serve reads — turns a routine join into a group-wide stall. This guide extends Initial Data Synchronization Methods and explains exactly how the provider picks a donor, how wsrep_sst_donor and wsrep_sst_donor_rejects_queries change that choice, and how to keep an SST from desyncing the one node you cannot afford to lose.

Context: Why Donor Selection Decides the Blast Radius #

An SST is not a background copy. The elected donor enters the Donor/Desynced state for the entire transfer: with mariabackup it stays writable but stops advancing its own apply queue relative to the group, and with the blocking rsync method it flat-out refuses writes. Either way, the donor is degraded for the duration, and its receive queue can back up until flow control throttles every writer in the group. That means the identity of the donor is the single biggest lever over how much a join hurts. Pick a lightly-loaded, same-subnet, non-critical peer and the transfer is invisible to your application; let automatic selection grab your primary write node in another availability zone and you have coupled a routine reprovision to a latency spike across the whole cluster.

The decision is worth controlling precisely because the provider’s automatic logic optimizes for transfer feasibility, not for your operational priorities. It does not know which node fronts your write traffic or which peer sits across an expensive inter-region link. Donor pinning is how you inject that knowledge, and understanding the default election is how you know when pinning is necessary.

Solution: Control the Election, Then the Read Behavior #

How the default election works #

With no pinning, the provider builds the donor candidate list from members currently in the SYNCED state — a JOINER, DONOR, or DESYNCED node is never eligible, because it cannot serve a consistent snapshot. Among eligible nodes it favors one that shares the joiner’s group segment (gmcast.segment), which keeps the transfer inside a low-latency locality, and it avoids a node already acting as a donor. What it does not weigh is application load: the node fronting your writes is just as likely to be chosen as an idle spare. That gap is the root cause of most donor-selection incidents.

Pin the donor with wsrep_sst_donor #

Pinning names one or more preferred donors in priority order. The value is set on the joiner, since the joiner requests its donor:

[mysqld]
# Prefer db-node-03, then db-node-02; the trailing comma allows auto-fallback.
wsrep_sst_donor="db-node-03,db-node-02,"

Two rules govern the syntax. The list is tried left to right, so put your least-critical, same-AZ nodes first. The trailing comma is a fallback switch: with it, if none of the named nodes are SYNCED the joiner falls back to automatic selection; without it, the joiner refuses to join and stalls until a named donor becomes available. Omitting the comma is a deliberate choice — use it only when joining from the wrong donor is worse than not joining at all.

Keep the donor out of the read path with wsrep_sst_donor_rejects_queries #

A node in Donor/Desynced still accepts SQL by default, which is a trap: a load balancer that health-checks only for a live connection will keep routing reads to a desynced donor whose data is momentarily behind the group. Setting wsrep_sst_donor_rejects_queries=ON on candidate donors makes a desynced donor return ER_UNKNOWN_COM_ERROR to every query for the duration of the SST, which cleanly ejects it from any read pool that checks query success:

[mysqld]
# While acting as a donor, reject queries so load balancers route around it.
wsrep_sst_donor_rejects_queries=ON

This converts a silent stale-read risk into an explicit, health-checkable signal. Pair it with a probe that treats Donor/Desynced as unhealthy, so the node is drained the moment it is elected — the readiness-probe pattern lives in Monitoring Galera Cluster State with Python.

Detect a desynced donor from status #

A probe that decides whether a node is safe to route to must inspect wsrep_local_state_comment and refuse anything that is not Synced. This one uses PyMySQL and handles the wsrep contention codes 1213 (deadlock / certification conflict) and 1205 (lock wait timeout) so a transient conflict is not mistaken for a donor drain:

import sys
import pymysql
from pymysql.err import OperationalError, MySQLError

def donor_aware_health(host: str, user: str, password: str) -> bool:
    """Return False while a node is acting as an SST donor or is otherwise not Synced."""
    try:
        conn = pymysql.connect(host=host, user=user, password=password,
                               connect_timeout=5, read_timeout=5)
    except OperationalError as exc:
        # A donor with reject_queries=ON refuses connections mid-SST — treat as drained.
        print(f"[drain] {host} not accepting queries: {exc}", file=sys.stderr)
        return False
    try:
        with conn.cursor() as cur:
            cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'")
            state = cur.fetchone()[1]
    except MySQLError as exc:
        if exc.args and exc.args[0] in (1213, 1205):
            print(f"[warn] {host} contended ({exc.args[0]}); retry", file=sys.stderr)
            return False
        raise
    finally:
        conn.close()

    if state != "Synced":
        # 'Donor/Desynced', 'Joiner', 'Joined' all mean: do not route reads here.
        print(f"[drain] {host} state={state}")
        return False
    print(f"[ok] {host} Synced")
    return True

if __name__ == "__main__":
    sys.exit(0 if all(donor_aware_health(h, "monitor", "secret") for h in sys.argv[1:]) else 1)

Parameter Reference #

Setting gmcast.segment per availability zone is the low-effort way to make even automatic selection prefer a local donor, because the provider favors same-segment candidates. The full flow-control and provider-option matrix these keys live in is documented in the wsrep.cnf Configuration Deep Dive, and choosing the transfer backend itself is covered in Choosing the Right SST Method for Large Datasets.

Verification #

Watch the election happen and confirm the donor you expected was chosen:

# On the joiner: which donor did the provider negotiate?
journalctl -u mariadb -f | grep -E "WSREP:.*(Donor|donor|State transfer to)"

# On every node: who is currently desynced?
mysql -N -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'"

The donor logs a line like WSREP: Member ... requested state transfer ... selected ... as donor, naming the elected node — if it is not the one you pinned, the pinned node was not SYNCED and the fallback fired. During the transfer the donor reports Donor/Desynced and the joiner reports Joiner; both must return to Synced, and wsrep_flow_control_paused on the donor should stay near zero if gcs.fc_limit is sized correctly.

Edge Cases & Gotchas #

• Pinning a node that is never Synced hangs the join. If wsrep_sst_donor="db-node-03" (no trailing comma) and db-node-03 is down or itself joining, the joiner waits indefinitely rather than picking another donor. Always include the trailing comma unless a wrong-donor transfer is genuinely unacceptable.
• Donor/Desynced still holds a quorum vote. Desyncing a donor does not remove it from the primary component, so a three-node group with one donor is still a healthy quorum of three — but if a second node fails during the SST you can lose quorum. Avoid provisioning during a maintenance window that already has a node down.
• rejects_queries breaks naive backup jobs. A backup script pointed at a node that becomes a donor will suddenly get query errors when wsrep_sst_donor_rejects_queries=ON. Point backups at a dedicated non-donor node, or pin donors away from the backup target — the decoupling rationale is in When to Use Async Replicas with Galera.

Related #

• Initial Data Synchronization Methods — the SST/IST model and the donor state machine this page tunes
• Choosing the Right SST Method for Large Datasets — pick the backend before you tune who donates it
• Speeding Up SST with Parallel Compression — shorten the window a donor spends desynced
• wsrep.cnf Configuration Deep Dive — the provider-option and flow-control matrix behind donor behavior
• Monitoring Galera Cluster State with Python — a probe that drains a desynced donor from read routing