Restoring One Node Without Triggering a Full SST

This procedure extends Point-in-Time Recovery in Multi-Master Galera and answers a narrow, high-value question: how do you rebuild a single failed node’s datadir from a mariabackup image so it rejoins the Galera cluster with a fast Incremental State Transfer instead of forcing a full State Snapshot Transfer? The naive approach — wipe the datadir and let the node resync from scratch — always triggers a full SST, which desyncs a donor and copies the entire dataset over the network. The better approach restores a recent physical backup and, crucially, writes the backup’s saved replication position into grastate.dat so the provider knows exactly how far behind the node is and asks the donor for only the missing write-sets. Done correctly, a multi-hundred-gigabyte node rejoins in seconds.

Context: Why the Restore Method Decides IST vs SST #

When a node starts and tries to join, the provider compares the node’s saved position against what a donor still holds in its write-set cache. That saved position lives in grastate.dat as a Galera cluster UUID and a seqno. A node whose datadir was wiped has no position — it reports seqno: -1 — and a node with no known position cannot request an incremental transfer, so it is forced into a full SST. This is the same fork described in gcache Sizing for IST & Fast Recovery: IST is possible only when the joiner declares a valid seqno that still falls inside a donor’s cached window.

A mariabackup image captures more than the data files — it records the exact Galera state coordinates at the moment the backup was taken, in a file named xtrabackup_galera_info. Restoring that image and propagating its seqno and UUID into grastate.dat gives the rejoining node a valid, recent position. As long as that position is still inside a donor’s gcache, the provider serves an IST covering just the gap between the backup and now. The entire technique is a matter of not throwing away the position the backup already recorded for you.

Solution: Restore the Datadir and Preserve the Position #

The workflow is: prepare the backup, copy it into the datadir, extract the saved Galera position, write it into grastate.dat, and start the node so it requests IST.

Step 1 — Prepare and copy back the backup image #

Apply the backup’s redo log to make it internally consistent, then restore it into an emptied datadir on the failed node. --prepare rolls forward committed and rolls back uncommitted transactions; --copy-back places the files:

# Make the physical image consistent (idempotent; safe to re-run).
sudo mariabackup --prepare --target-dir=/backups/node3-2026-07-16

# Restore into an emptied datadir.
sudo systemctl stop mariadb
sudo rm -rf /var/lib/mysql/*
sudo mariabackup --copy-back --target-dir=/backups/node3-2026-07-16
sudo chown -R mysql:mysql /var/lib/mysql

Step 2 — Read the saved Galera position from the backup #

mariabackup writes the Galera cluster UUID and seqno captured at backup time into xtrabackup_galera_info inside the restored datadir. This is the position the node will declare when it rejoins:

sudo cat /var/lib/mysql/xtrabackup_galera_info
# Example contents:  5f3aab19-1e2c-11ef-9f3a-0242ac120003:918342
#                    ^--------------- cluster UUID -----------^ ^seqno^

The value left of the colon is the Galera cluster UUID; the value right of it is the seqno S. Both must land in grastate.dat for the provider to trust the position.

Step 3 — Write the position into grastate.dat #

Galera reads grastate.dat at startup to decide its rejoin path. Populate it with the UUID and seqno from the backup, and set safe_to_bootstrap: 0 because this node is joining an existing cluster, not forming one:

sudo tee /var/lib/mysql/grastate.dat >/dev/null <<'EOF'
# GALERA saved state
version: 2.1
uuid:    5f3aab19-1e2c-11ef-9f3a-0242ac120003
seqno:   918342
safe_to_bootstrap: 0
EOF
sudo chown mysql:mysql /var/lib/mysql/grastate.dat

A seqno of -1 here is exactly what you are avoiding — it would force SST. Setting the real backup seqno is what makes IST possible. Never set safe_to_bootstrap: 1 on a rejoining node, since that authorizes it to form a competing primary component.

Step 4 — Start the node and let it request IST #

Start MariaDB normally (not with galera_new_cluster). The provider reads the position, contacts a donor, and — if seqno S is still in the donor’s cache — receives an IST for just the missing range:

sudo systemctl start mariadb
sudo tail -f /var/log/mysql/error.log | grep -E "IST|SST|Synced"
# Success: 'WSREP: Receiving IST: 40217 writesets, seqnos 918343-958559'

If the log instead shows a fallback to wsrep_sst_mariabackup, seqno S had aged out of every donor’s gcache — the diagnosis and fix are in Diagnosing IST Failures & gcache Eviction.

Parameter Reference #

The interplay to internalize: this technique only keeps you on IST if the donor’s gcache still holds seqno S. If your backups are older than the ring’s coverage window, restoring them still lands you in SST — so pair a fresh backup cadence with a ring sized per Sizing gcache to Avoid a Full SST on Rejoin.

Verification #

Confirm the node took IST and reached Synced as a full member. First check the transfer path in the log, then the live state:

SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_local_state_comment',   -- expect 'Synced'
  'wsrep_cluster_status',        -- expect 'Primary'
  'wsrep_cluster_size'           -- expect the full node count
);

To prove it was IST rather than SST specifically, the joiner log is authoritative — Receiving IST with a bounded seqno range confirms only the gap transferred. You can also compare the restored seqno against the donor’s cached window before starting the node, to predict the path in advance:

# On a healthy donor: is the backup's seqno still cached?
mysql -N -B -e "SHOW STATUS LIKE 'wsrep_local_cached_downto';"
# If this value <= 918342 (the backup seqno), IST is available.

Edge Cases & Gotchas #

• A stale backup defeats the whole technique. If the backup’s seqno is older than the donor’s oldest cached write-set, the node falls back to SST no matter how carefully you set grastate.dat. Keep backups recent relative to your gcache window, or accept that a very old restore implies a full resync.
• A wrong UUID gets the node rejected, not just SST-ed. The uuid in grastate.dat must exactly match the live cluster’s UUID (visible as wsrep_cluster_state_uuid on any member). A mismatched UUID makes the provider treat the node as belonging to a different cluster and refuse the join outright; copy it verbatim from xtrabackup_galera_info.
• Restoring while the datadir is partially populated causes silent corruption. --copy-back expects an empty target. If a crashed node left partial files, mariabackup may refuse or, worse, mix old and new files. Always fully clear the datadir first, and confirm ownership is mysql:mysql before starting, per the lifecycle rules in Graceful Node Join and Leave Procedures.

Related #

• Point-in-Time Recovery in Multi-Master Galera — the full-cluster recovery this single-node restore is a building block of
• gcache Sizing for IST & Fast Recovery — sizing the donor ring so a restored seqno stays cached
• Diagnosing IST Failures & gcache Eviction — what to do when a restore still falls back to SST
• Sizing gcache to Avoid a Full SST on Rejoin — matching backup freshness to the ring window
• Graceful Node Join and Leave Procedures — the node lifecycle rules the rejoin follows