Backup & Restore Sync for MariaDB Galera Clusters

Protecting a MariaDB Galera Cluster is a different discipline from backing up a single server: because every node holds an identical certified copy of the dataset, the questions that matter are not “where is my one good copy” but “how do I capture a consistent snapshot without desyncing a live member, how do I rebuild a wiped node from the write-set history instead of a full copy, and how do I roll the whole group back to a moment in time when replication faithfully propagated a bad DELETE to all three nodes at once.” This is the home guide for those concerns. It covers non-blocking physical backups with mariabackup, the state-transfer machinery (SST and IST) that rebuilds a node, gcache sizing so a rejoin stays incremental, point-in-time recovery across a multi-master group, and the verification and automation patterns that make a restore something you have actually tested rather than something you hope works. It targets MariaDB 10.6 through 11.x with Galera 4; return to the wider Galera automation library for the adjacent setup, monitoring, and architecture domains.

Architecture Overview: Where Backups, SST, and IST Fit #

Synchronous replication changes what a backup is. The Galera provider keeps every member byte-for-consistent at the transaction level, so the live cluster is not itself a backup — a logical corruption or an erroneous statement is certified and applied everywhere in milliseconds, and the redundancy that protects you from a dead disk does nothing against a bad WHERE clause. Data protection therefore splits into three distinct mechanisms that operate at different layers of the Galera core architecture, and confusing them is the root of most restore failures.

The three mechanisms are:

• State Snapshot Transfer (SST) — a full physical copy of one node’s data directory streamed to a node that has no usable state. In production this is driven by mariabackup, which reads InnoDB pages without a blocking table lock, so the donor stays writable while it streams. SST rebuilds a node from scratch and moves over port 4444.
• Incremental State Transfer (IST) — a replay of only the write-sets a rejoining node is missing, served from the donor’s gcache ring buffer over port 4568. IST is cheap and fast, but only possible when the joiner’s last committed sequence number is still inside the donor’s cache window.
• Scheduled backups & point-in-time recovery (PITR) — physical mariabackup snapshots taken on a cadence and archived off-cluster, combined with binary logs so you can restore to an arbitrary transaction rather than only to the last full backup. This is the layer that protects against logical damage the Galera cluster faithfully replicated.

Figure: a scheduled hot backup and the two state-transfer paths, overlaid on the synchronous write-set flow between cluster members.

Three baseline settings make every one of these mechanisms work, and each must be identical on every node. wsrep_sst_method=mariabackup selects the non-blocking physical transfer instead of the donor-locking rsync default; wsrep_provider_options must carry a gcache.size large enough to keep rejoins on the IST path; and log_bin with log_slave_updates must be enabled if you intend to do point-in-time recovery, because without a binary log there is nothing to replay past the last full snapshot. The mandatory baseline for a backup-capable cluster looks like this:

[mysqld]
# --- Baseline for a backup- and recovery-capable Galera node ---
wsrep_on                 = ON
wsrep_provider           = /usr/lib/galera/libgalera_smm.so
binlog_format            = ROW
default_storage_engine   = InnoDB
innodb_autoinc_lock_mode = 2

# --- State transfer: non-blocking physical SST ---
wsrep_sst_method         = mariabackup
wsrep_sst_auth           = sstuser:REPLACE_WITH_VAULT_SECRET

# --- Keep rejoins incremental, and enable PITR ---
wsrep_provider_options   = "gcache.size=4G"
log_bin                  = /var/log/mysql/mariadb-bin
log_slave_updates        = ON
expire_logs_days         = 7

The SST method choice is consequential enough to warrant its own guide — the non-blocking streaming mechanics, compression, and encryption options are detailed in mariabackup Streaming SST & Backups, and the broader menu of transfer methods and how a node decides between them is documented under Initial Data Synchronization Methods.

How State Protection and Recovery Work End to End #

A Galera node lives in one of a small set of lifecycle states, and every recovery path is a route through that state machine back to Synced. Understanding which route a given failure forces you onto is the whole game, because the routes differ in cost by orders of magnitude.

When a node starts and discovers it has some prior state — a populated data directory and a grastate.dat recording the UUID and sequence number it last committed — it asks the group whether the gap between its seqno and the current cluster position can be served from a donor’s gcache. If yes, the donor streams the missing write-sets as an Incremental State Transfer: the joiner applies them in order, catches up to the live position, and transitions to Synced in seconds to minutes with no bulk data copy. The gcache is a fixed-size ring buffer written to the galera.cache file, so the determining factor is simple: if the joiner’s required seqno has already been overwritten by newer write-sets, IST is impossible. Sizing that buffer to cover your longest realistic outage is the single highest-leverage tuning decision for fast recovery, and it has a page of its own in gcache Sizing for IST & Fast Recovery.

When IST cannot be served — a wiped data directory, a corrupted grastate.dat reporting seqno: -1, or a gap larger than the cache — the group falls back to a State Snapshot Transfer. With mariabackup the donor takes a hot physical backup of its live data directory and streams it to the joiner while continuing to serve queries. Internally the donor marks itself Donor/Desynced so it stops sending flow-control feedback that would otherwise throttle the whole cluster during the transfer, but it does not stop accepting writes; the write-sets it accepts during the backup are captured and shipped as the incremental tail so the joiner ends up exactly consistent. This is why the method matters: the legacy rsync method takes a blocking read lock and freezes the donor for the duration, whereas mariabackup does not. The full comparison of donor-blocking behaviour, throughput, and encryption is laid out in mariabackup vs rsync SST: Choosing a State-Transfer Method.

Both SST and IST rebuild a node to the current cluster state. Neither can take you backwards in time, and that is the gap point-in-time recovery fills. Because a certified write-set is applied on every node, a destructive but valid statement — a DROP TABLE, an unfiltered UPDATE, an application bug that deletes the wrong rows — is replicated to all members before anyone notices. No amount of node redundancy helps; the only recovery is to restore a physical mariabackup snapshot taken before the damage and replay the binary log forward to the transaction immediately preceding it. This requires binary logging to have been enabled on the node the backup came from, which is not the Galera default, and it requires a coordinate system — GTID — to identify the stop point precisely. The mechanics of enabling logs and executing the rollback across a multi-master group are the subject of Point-in-Time Recovery in Multi-Master.

The subtle interaction is that PITR and the state-transfer machinery cooperate. You do not restore every node from the archive — you restore one node to the target point, bootstrap it as the new primary component, and let the remaining nodes rebuild from it via SST or IST. The recovered node’s grastate.dat seqno determines whether those rejoins are incremental or full, which is why a restore procedure that lands inside the gcache window can bring an entire cluster back far faster than one that forces three full snapshots.

Configuration Reference #

The parameters that govern backup and recovery fall into three domains: state transfer, cache sizing, and binary logging for PITR. Every key below lives in the [mysqld] section unless noted as a mariabackup command-line flag.

The state-transfer and PITR settings interact with the wider provider-options string, which is a single semicolon-delimited value; splitting gcache keys across two declarations silently drops all but the last. A representative production block that keeps rejoins incremental and PITR possible:

[mysqld]
# --- State transfer ---
wsrep_sst_method                = mariabackup
wsrep_sst_auth                  = sstuser:REPLACE_WITH_VAULT_SECRET
wsrep_sst_donor                 = "galera-node-3,"   # trailing comma = any if node-3 down

# --- Provider options: gcache tuned for IST ---
wsrep_provider_options          = "gcache.size=8G; gcache.recover=yes; gcache.page_size=256M"

# --- Binary logging for point-in-time recovery ---
log_bin                         = /var/log/mysql/mariadb-bin
log_slave_updates               = ON
binlog_format                   = ROW
gtid_strict_mode                = ON
expire_logs_days                = 7

The trailing comma in wsrep_sst_donor is deliberate: it tells the joiner to prefer galera-node-3 as the donor but to fall back to any available member if that node is down, rather than failing the join outright. Donor pinning matters because a full SST loads the donor’s disk and network, so directing it at a dedicated node keeps that cost off the members serving application writes.

Automation Patterns #

The backup you have never restored is a hypothesis, not a backup. The most valuable automation in this domain is not the backup job itself — cron plus mariabackup is trivial — but the verification harness that proves each snapshot is restorable. The following Python 3.9+ orchestrator takes a streaming mariabackup, then validates it by preparing it in a scratch directory and asserting the InnoDB recovery succeeded, and it handles the Galera contention error codes 1213 (deadlock/certification conflict) and 1205 (lock wait timeout) explicitly so a busy source does not read as a backup failure.

import subprocess
import datetime
import pathlib
import sys
import pymysql

BACKUP_ROOT = pathlib.Path("/var/backups/galera")

def source_is_safe(host: str, user: str, password: str) -> bool:
    """Refuse to back up a node that is itself unhealthy or flow-control paused."""
    try:
        conn = pymysql.connect(host=host, user=user, password=password,
                               connect_timeout=5)
        with conn.cursor() as cur:
            cur.execute("SHOW GLOBAL STATUS WHERE Variable_name IN "
                        "('wsrep_local_state_comment','wsrep_flow_control_paused')")
            status = {name: val for name, val in cur.fetchall()}
    except pymysql.err.OperationalError as exc:
        # 1213 = certification deadlock, 1205 = lock wait timeout: transient, retry.
        if exc.args and exc.args[0] in (1213, 1205):
            print(f"[WARN] {host} contended ({exc.args[0]}); retry backup shortly")
            return False
        print(f"[FATAL] {host} unreachable: {exc}", file=sys.stderr)
        return False
    finally:
        try:
            conn.close()
        except Exception:
            pass
    if status.get("wsrep_local_state_comment") != "Synced":
        print(f"[FAIL] {host} not Synced: {status}")
        return False
    if float(status.get("wsrep_flow_control_paused", 0)) > 0.1:
        print(f"[FAIL] {host} is flow-control throttled; pick another node")
        return False
    return True

def take_and_verify_backup(host: str, sst_user: str, sst_pass: str) -> pathlib.Path:
    stamp = datetime.datetime.now().strftime("%Y%m%dT%H%M%S")
    target = BACKUP_ROOT / stamp
    target.mkdir(parents=True, exist_ok=True)

    # 1. Streaming hot backup — no blocking table lock on the donor.
    with open(target / "backup.xb", "wb") as fh:
        subprocess.run(
            ["mariabackup", "--backup", f"--host={host}",
             f"--user={sst_user}", f"--password={sst_pass}",
             "--stream=xbstream"],
            stdout=fh, check=True,
        )

    # 2. Prepare in a scratch dir: this replays the InnoDB redo log and is the
    #    real test that the snapshot is internally consistent and restorable.
    extract = target / "prepared"
    extract.mkdir(exist_ok=True)
    subprocess.run(["mbstream", "-x", "-C", str(extract)],
                   stdin=open(target / "backup.xb", "rb"), check=True)
    result = subprocess.run(
        ["mariabackup", "--prepare", f"--target-dir={extract}"],
        capture_output=True, text=True,
    )
    if result.returncode != 0 or "completed OK!" not in result.stderr:
        raise RuntimeError(f"Backup {stamp} FAILED to prepare — not restorable")
    print(f"[OK] {stamp} verified restorable at {extract}")
    return extract

if __name__ == "__main__":
    node = sys.argv[1] if len(sys.argv) > 1 else "127.0.0.1"
    if not source_is_safe(node, "monitor", "REDACTED"):
        sys.exit(1)
    take_and_verify_backup(node, "sstuser", "REDACTED")

The discipline that matters is the --prepare step: mariabackup reports completed OK! only when the InnoDB redo log applied cleanly, so gating on that string turns an unverified archive into a proven one. A lighter shell gate is enough for a cron wrapper that only needs to confirm the target node is a safe backup source before it runs:

#!/usr/bin/env bash
set -euo pipefail

query() { mysql -N -B -e "SHOW GLOBAL STATUS LIKE '$1';" | awk '{print $2}'; }

state=$(query wsrep_local_state_comment)
paused=$(query wsrep_flow_control_paused)

if [[ "$state" != "Synced" ]] || (( $(echo "$paused > 0.1" | bc -l) )); then
  echo "Refusing backup: state=$state fc_paused=$paused" >&2
  exit 1
fi
echo "Node safe to back up (Synced, not throttled)"

Wire these into the fleet tooling that already renders configuration and health-gates restarts; the Python status-probe patterns that these reuse are built out under Automated Node Health Monitoring. Every scheduled backup should also record the source node’s GTID position at snapshot time, because that coordinate is what a later point-in-time restore replays forward from.

Infrastructure & Topology Considerations #

Backup infrastructure amplifies the same physical constraints that govern replication. A full SST reads an entire data directory and pushes it across the network, so on a multi-terabyte dataset the transfer is bounded by the slower of donor disk throughput and inter-node bandwidth — plan the SST network path with the same care as the replication path, and prefer a dedicated donor so that load never lands on a node serving application writes. Where SST cost is dominated by dataset size, the parallel-compression and method-selection trade-offs in Choosing the Right SST Method for Large Datasets become the deciding factor in how long a rebuild takes.

Backup storage must live off the Galera cluster. Archiving snapshots to a node’s local disk protects against nothing that also loses that node; ship them to object storage or a separate backup host, and treat the binary logs as part of the archive rather than an afterthought, because a full snapshot without the binlog tail can only restore to the snapshot instant, not to the transaction before an incident. Retention is a quorum-independent decision: expire_logs_days should hold enough binary log to cover the window between your backup cadence and your worst-case detection-to-recovery time, or PITR will run out of log before it reaches the damage.

Placement also governs how fast a whole-cluster rebuild completes. If you keep one node’s gcache sized generously and use it as the restore seed, the other members rejoin by IST rather than each taking an independent full SST — turning an O(nodes) copy into a single copy plus incremental catch-up. This makes the recovery-seed node a deliberate topology role, not an accident of which node you happened to restore first.

Failure Modes & Remediation #

The incidents that page an on-call engineer in this domain cluster around a handful of causes. The table maps the symptom you will actually see to its root cause and first action.

Recovery from logical damage always begins by stopping the bleeding — halt the offending application path before you restore, or PITR will replay against a moving target. Then identify the most recent verified snapshot whose GTID precedes the damage, restore it to a single seed node, replay the binary log up to the transaction before the bad statement, and rebuild the rest of the Galera cluster from that seed. Controlled add and remove sequencing that keeps those rebuilds on the fast path is covered in Graceful Node Join and Leave Procedures.

Monitoring & Telemetry #

Backup and recovery health is observable through a small set of wsrep_ status variables plus the exit status of the backup jobs themselves. Scrape the status vars with SHOW GLOBAL STATUS LIKE 'wsrep_%' and pair them with a freshness metric emitted by the backup wrapper.

The most under-monitored value here is wsrep_local_cached_downto: the distance between it and wsrep_last_committed is your IST headroom in write-sets, and watching that margin erode is how you catch a gcache that has silently become too small for your write rate before the next rejoin forces a surprise full SST. Diagnosing exactly that eviction condition is the focus of Diagnosing IST Failures & gcache Eviction.

Frequently Asked Questions #

Does Galera’s synchronous replication mean I do not need backups? No. Replication protects against hardware loss, not logical damage. A DROP TABLE or an erroneous UPDATE is certified and applied on every node within milliseconds, so all your redundant copies are corrupted identically. You still need periodic physical mariabackup snapshots and binary logs for point-in-time recovery, because those are the only mechanism that can take the data backwards to a moment before the damage.

Why does a rejoining node sometimes take a full SST and sometimes a quick IST? The provider serves an Incremental State Transfer only when the joiner’s last committed sequence number is still held in a donor’s gcache ring buffer. If the node was down long enough that its required seqno was overwritten by newer write-sets, the buffer no longer contains the gap and the group falls back to a full State Snapshot Transfer. Sizing gcache.size to cover your longest expected outage is what keeps rejoins on the fast IST path.

Can I take a backup without slowing down the Galera cluster? Yes, if you use mariabackup. It reads InnoDB pages without a blocking table lock, so the source node keeps serving reads and writes while it streams. Point the job at a dedicated donor or an async replica rather than a primary write node, and gate it on the node being Synced and not flow-control paused, and a scheduled backup has no visible impact on application traffic.

Related #

• mariabackup Streaming SST & Backups — non-blocking state transfer and scheduled physical backups end to end
• gcache Sizing for IST & Fast Recovery — sizing the write-set ring buffer so rejoins stay incremental
• Point-in-Time Recovery in Multi-Master — rolling a Galera cluster back to a transaction with binary logs and GTID
• Initial Data Synchronization Methods — the full SST/IST decision model behind these recovery paths
• MariaDB Galera Core Architecture & Fundamentals — the replication and certification model these backups protect