Troubleshooting Node Desync During Join

This guide extends the runbook in Graceful Node Join and Leave Procedures to answer one focused question: when a MariaDB Galera node hangs at Joining, flips to Donor/Desynced, or drops into Joiner: failed during a rejoin, how do you find the exact cause and clear it without forcing an avoidable State Snapshot Transfer (SST) or fracturing quorum? Node desync during join is the single most common failure a platform team hits after a maintenance window, and almost every case reduces to one of three root causes — an aged-out gcache window, a blocked transfer port, or a corrupted on-disk state file. This page walks the diagnostic path for each, gives the exact status variables and log signatures to read, and provides a Python pre-join probe that catches the failure before it starts.

Why Join-Time Desync Happens in a Multi-Master Cluster #

In a synchronous multi-master topology there is no “catching up later” — a node is either a full, voting member of the Primary Component or it is not trusted to serve reads and writes at all. When a member rejoins, Galera must reconcile the joiner’s last committed position with the group’s current position before it will promote the node to Synced. That reconciliation happens through one of two paths: an Incremental State Transfer (IST) that replays only the missing write-sets from a donor’s cache, or a full SST that streams the entire dataset. Desync is what you observe when that negotiation stalls: the joiner has requested state, but the transfer never completes, so the node cannot advance out of Joining.

The reason this matters more here than in primary-replica replication is quorum coupling. A stalled joiner can pull a donor into Donor/Desynced for the whole transfer, and if the donor was one of only three members, a second event during that window can drop the group below its floor(N/2)+1 threshold and turn the group non-Primary. Understanding the certification and membership rules behind that coupling is covered in how Galera synchronous replication works; this page assumes that model and focuses on the diagnostics.

Diagnosing the Stall: State, Sequence, and Log Signatures #

Start every investigation from the joiner’s reported state, because the state comment tells you which branch of the decision tree you are on. Run this on the joining node:

SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
SHOW GLOBAL STATUS LIKE 'wsrep_last_committed';
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';

A value stuck at Joining with no bytes moving points to a transport problem. A value cycling through Donor/Desynced on the donor side points to an in-progress transfer that is either slow or wedged. A Joiner: failed in the log means the SST script itself exited non-zero.

Next, decide whether IST was even possible. On a synced donor, compare the joiner’s last position against the donor’s retained cache window:

-- Run on the donor
SHOW GLOBAL STATUS LIKE 'wsrep_last_committed';       -- donor's current position
SHOW GLOBAL STATUS LIKE 'wsrep_local_cached_downto';  -- oldest seqno still in gcache

If the joiner’s wsrep_last_committed is lower than the donor’s wsrep_local_cached_downto, the missing write-set range has already been purged from gcache.size (wsrep_provider_options) and Galera has no choice but to fall back to a full SST. That is not a bug — it is the direct consequence of an undersized cache or an overlong outage.

Then read the log. Tail the journal on both nodes and filter for the transfer subsystems:

journalctl -u mariadb -f --no-pager | grep -E "WSREP|SST|IST"

The specific line you see maps directly to a cause:

• WSREP: Failed to prepare for incremental state transfer followed by Requesting state transfer naming mariabackup — IST was declined; the joiner’s range aged out of the donor cache and Galera is falling back to SST. Expected when the cache is too small, not a transport fault.
• WSREP: SST request failed: 113 (No route to host) — the SST channel could not be opened. Verify wsrep_sst_receive_address and the firewall path on port 4444.
• WSREP: Failed to read uuid:seqno from joiner script — the SST helper (mariabackup or rsync) crashed or lacks permissions; check its binary path and the wsrep_sst_auth user’s grants.
• WSREP: Node was not allowed to join: 113 — a quorum or membership mismatch, usually a drifted wsrep_cluster_address.

Fixing Each Root Cause #

Once the signature identifies the branch, the remediation is deterministic. Work through only the branch your diagnostics pointed to — do not blindly wipe state on a node that only had a blocked port.

Transport blocked (stuck at Joining, no bytes moving) #

Galera needs bidirectional TCP on 4567 (replication), 4568 (IST), and 4444 (SST). A rejoin that hangs before any data moves is almost always a dropped port or an MTU mismatch on the transfer path. Confirm the listeners and the reachability, then re-check the rules against the network security and firewall rules for Galera:

# On the donor: are the transfer ports listening?
ss -tlnp | grep -E '4444|4567|4568'
# From the joiner: can it actually reach the donor's SST port?
nc -vz 10.0.1.10 4444
# Watch the SST payload cross the wire (run during the join)
tcpdump -i any port 4444 -nn

A wsrep_sst_receive_address bound to 127.0.0.1 or left at 0.0.0.0 is a frequent silent failure: the donor connects but the payload never lands on a routable interface. Pin it to the node’s real address in the [mysqld] section and restart.

gcache aged out (falls back to full SST every rejoin) #

If diagnostics show the joiner’s position sits below the donor’s wsrep_local_cached_downto, IST is impossible and you have two levers: make future windows cheaper, or make the current SST safe. Widen the cache so the next rejoin qualifies for IST:

[mysqld]
# Size the cache to exceed the write-set volume of your longest maintenance window.
wsrep_provider_options="gcache.size=8G; gcache.page_size=256M"

Restart the donor during a maintenance window to apply the new allocation. The full tradeoff between a wider cache and the cost of the SST it avoids is covered in choosing the right SST method for large datasets.

Corrupted or uninitialized state (Joiner: failed, split UUID) #

Inspect the joiner’s saved state before doing anything destructive:

cat /var/lib/mysql/grastate.dat

A seqno: -1 means the node was stopped abruptly (a SIGKILL, OOM kill, or power loss) and Galera treats it as uninitialized — it will demand a full SST regardless of cache availability. A uuid that does not match wsrep_cluster_state_uuid on a synced donor means this node holds history from a different lineage (a split-brain or an accidental bootstrap) and must not be forced in. In both cases the safe recovery is to discard the local state and let the node re-provision cleanly from an authoritative member:

systemctl stop mariadb
rm -f /var/lib/mysql/grastate.dat /var/lib/mysql/gvwstate.dat
systemctl start mariadb

Removing grastate.dat and gvwstate.dat forces a fresh SST from a healthy donor rather than letting the node argue with the group over an inconsistent position. Never edit the seqno upward by hand to dodge an SST — a node that claims writes the group does not have will be rejected with a Reversing history error and can destabilize the Primary Component.

Parameter Reference #

These are the variables that govern whether a join succeeds via IST, falls back to SST, or stalls. Values assume a 3-node cluster on NVMe-backed storage with a moderate OLTP write rate.

Everything except wsrep_sst_method, wsrep_sst_auth, wsrep_sst_receive_address, and wsrep_sst_donor is set inside the wsrep_provider_options string in the [mysqld] section.

Pre-Join Validation Probe #

The cheapest fix for join-time desync is to never start a doomed join. Run a probe from your orchestration layer before systemctl start mariadb that verifies the SST port is reachable and that the donor still retains enough of its cache for IST. This snippet targets Python 3.9+, uses PyMySQL, and handles the two transient wsrep errors — 1213 (deadlock / certification conflict) and 1205 (lock wait timeout) — so a momentary conflict on the donor retries instead of aborting the check.

import socket
import time
import pymysql


def donor_status(donor_ip: str, user: str, password: str) -> dict:
    """Read the donor's committed and cached-down-to seqnos, retrying transient wsrep errors."""
    wanted = ("wsrep_last_committed", "wsrep_local_cached_downto")
    for attempt in range(3):
        try:
            conn = pymysql.connect(host=donor_ip, user=user, password=password,
                                   connect_timeout=5)
            try:
                with conn.cursor() as cur:
                    out = {}
                    for var in wanted:
                        cur.execute("SHOW GLOBAL STATUS LIKE %s", (var,))
                        row = cur.fetchone()
                        out[var] = int(row[1]) if row else 0
                    return out
            finally:
                conn.close()
        except pymysql.err.OperationalError as exc:
            code = exc.args[0]
            if code in (1213, 1205):  # deadlock / lock wait timeout — retry
                time.sleep(2 ** attempt)
                continue
            raise
    raise RuntimeError("donor status unavailable after retries")


def validate_join_readiness(donor_ip, joiner_last_committed, user, password, sst_port=4444):
    # 1. Transport: can the joiner open the SST channel at all?
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.settimeout(3)
        if s.connect_ex((donor_ip, sst_port)) != 0:
            raise ConnectionError(f"SST port {sst_port} unreachable on {donor_ip}")

    # 2. IST eligibility: is the joiner's position still inside the donor's cache window?
    status = donor_status(donor_ip, user, password)
    if joiner_last_committed < status["wsrep_local_cached_downto"]:
        raise ValueError("Joiner seqno aged out of donor gcache — join will force a full SST.")
    return True

Wire the raised exceptions into your Ansible or CI gate: a ConnectionError blocks the join until the firewall is fixed, while a ValueError lets you decide up front whether to widen gcache.size or schedule the SST during a low-traffic window. The Ansible role that renders a consistent wsrep.cnf across the fleet is in automating node provisioning with Ansible.

Verification #

After applying a fix and restarting, confirm the node genuinely reached full membership before the router sends it traffic:

-- Must read: Synced
SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
-- Must read: Primary
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';
-- Must equal your expected node count (e.g. 3)
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';
-- Must read: ON
SHOW GLOBAL STATUS LIKE 'wsrep_ready';
-- Should drain toward 0 shortly after Synced
SHOW GLOBAL STATUS LIKE 'wsrep_local_recv_queue';

A node reporting Synced and Primary with wsrep_cluster_size matching the full membership has cleared the desync. If wsrep_local_recv_queue stays elevated, the node reached membership but is still applying backlog — hold traffic until it drains. Wire these same variables into the alerting described in automated node health monitoring so the next stall is caught before it cascades.

Edge Cases & Gotchas #

• Docker and container images ship a seqno: -1 baseline. A freshly built image whose grastate.dat was never cleanly written will force an SST on first join even when a cache-eligible IST looks possible. Bake a clean state or expect the first join to be a full transfer, and give the container enough shutdown grace (stop_grace_period) that a normal stop is not escalated to SIGKILL.
• A systemd TimeoutStopSec shorter than an in-progress SST corrupts the donor’s state. If the donor is serving an SST and systemd kills it because the stop timed out, that donor now carries a -1 seqno and needs its own recovery. Raise TimeoutStopSec on donors or drain them before maintenance; the startup and shutdown log decoding is covered in handling Galera startup errors and logs.
• Mismatched wsrep_sst_method between donor and joiner terminates the handshake instantly. A donor set to mariabackup and a joiner set to rsync will fail at negotiation with no useful payload error. Enforce a single method across every node from one templated config rather than per-host edits.

Related #

• Graceful Node Join and Leave Procedures — the parent runbook for the full leave-and-rejoin lifecycle this page troubleshoots
• Fixing wsrep_local_state_comment Issues — resolving a node stuck outside the Synced state
• Choosing the Right SST Method for Large Datasets — controlling the cost when a desync forces a full SST fallback
• Network Security & Firewall Rules for Galera — opening 4444/4567/4568 so the transfer path is never the blocker