mariabackup Streaming SST & Backups: Non-Blocking State Transfer

This guide builds on the recovery model in Backup & Restore Sync and solves one operational problem: how to rebuild or seed a MariaDB Galera node without freezing the member that feeds it. The legacy rsync transfer method takes a global read lock on the donor for the entire copy, so a joining node can stall production writes for minutes on a large dataset. mariabackup replaces that with a hot physical backup — it copies InnoDB pages while the donor keeps committing, captures the concurrent write-sets as an incremental tail, and hands the joiner a byte-consistent snapshot. This page shows how to configure wsrep_sst_method=mariabackup, how the streaming State Snapshot Transfer actually moves data across the wire, and how to reuse the same tooling for scheduled off-cluster backups with compression and encryption.

Concept: How a Streaming mariabackup SST Works #

A State Snapshot Transfer fires whenever a node needs a complete copy of the dataset — a fresh node, a wiped data directory, or a rejoin whose sequence-number gap is too large for the gcache to serve as an Incremental State Transfer. Galera does not perform the copy itself; it invokes an SST script, wsrep_sst_mariabackup, on both donor and joiner, and that script orchestrates mariabackup plus a streaming transport.

The defining property is that the donor is never globally locked. mariabackup reads the live InnoDB data files page by page while recording every change to the redo log, so a page copied early and modified during the backup is reconciled at prepare time by replaying the captured redo. For the brief window it needs a consistent view of non-transactional metadata, it uses MariaDB’s backup locks (BACKUP STAGE statements) rather than a full FLUSH TABLES WITH READ LOCK, so DML on InnoDB tables is never blocked. The donor does mark itself Donor/Desynced for the duration — which suppresses its flow-control feedback so it does not throttle the healthy members while it is busy — but it keeps accepting reads and writes throughout.

Figure: the donor streams a hot backup to the joiner over socat, then ships the write-sets it accepted during the copy as an incremental tail.

Because the transfer is physical rather than logical, its cost scales with data size and disk throughput, not with row count or query complexity — a terabyte of InnoDB pages moves at streaming speed regardless of how many tables it spans. That physicality is also why mariabackup is the same tool you use for scheduled off-cluster backups: an SST is simply a streaming backup whose destination is another node’s data directory instead of an archive.

Prerequisites & Environment #

A streaming SST touches software, network state, and privileges on every node, and a gap in any of them turns the first join into a hard failure.

Software — install the same mariabackup version as the server on every node; it ships in the mariadb-backup package and a missing binary is the single most common SST failure. Add socat (the default streaming transport) and, for compressed backups, qpress or the lz4/zstd utilities the SST script can invoke.

Network ports — the joiner listens for the stream, so port state must match the config:

Locking these to known peers is covered in Network Security & Firewall Rules for Galera; a closed 4444 makes SST hang with no useful error.

Privileges & secrets — the donor’s mariabackup process authenticates with the account named in wsrep_sst_auth, and that account needs a specific least-privilege grant set. The full privilege model, secret storage, and TLS-for-SST setup are the subject of Securing mariabackup SST Credentials & Privileges; at minimum the account needs RELOAD, PROCESS, LOCK TABLES, and REPLICATION CLIENT (renamed BINLOG MONITOR on MariaDB 10.5+).

Step-by-Step: Configure and Drive a mariabackup SST #

The workflow is: select the method, provision the SST account, verify the transport, trigger a join, and confirm the joiner reaches Synced.

Step 1 — Select mariabackup as the SST method #

Set the method identically on every node, because either side of a transfer can be donor or joiner. Keep these keys in a single version-controlled drop-in, per the precedence discipline in the wsrep.cnf Configuration Deep Dive:

[mysqld]
wsrep_sst_method = mariabackup
wsrep_sst_auth   = sstuser:REPLACE_WITH_VAULT_SECRET

[sst]
# streaming and packaging options consumed by wsrep_sst_mariabackup
streamfmt   = xbstream
transferfmt = socat

xbstream is mandatory for mariabackup streaming (the older tar format cannot carry incremental or encrypted backups), and socat is the default transport that also carries the TLS options used later.

Step 2 — Create the SST account #

On any one node — the write is replicated to the rest — create the account the donor uses. Grant only what mariabackup needs:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'REPLACE_WITH_VAULT_SECRET';
GRANT RELOAD, PROCESS, LOCK TABLES, BINLOG MONITOR
  ON *.* TO 'sstuser'@'localhost';
-- MariaDB 10.4 and earlier: use REPLICATION CLIENT in place of BINLOG MONITOR
FLUSH PRIVILEGES;

Step 3 — Prove a manual streaming backup works #

Before you rely on it for a join, run the exact backup the SST script will run, streamed to a file, so any privilege or tooling gap surfaces in isolation rather than mid-SST:

# Hot streaming backup to an archive — same mechanics as an SST
mariabackup --backup --user=sstuser --password="$SST_PASS" \
  --stream=xbstream --parallel=4 | \
  zstd -T4 > /var/backups/galera/$(date +%FT%H%M%S).xb.zst

--parallel=4 copies multiple InnoDB files concurrently; zstd -T4 compresses the stream with four threads so the archive is small without bottlenecking on a single core. This same command is your scheduled-backup building block.

Step 4 — Trigger the join and watch the transfer #

Start the joining node normally. The provider detects it needs state, selects a donor, and both sides invoke wsrep_sst_mariabackup. Watch the donor’s error log for the transfer lifecycle:

# On the donor
sudo journalctl -u mariadb -f | grep -Ei 'wsrep_sst|mariabackup|Donor|Desynced'

You should see the donor enter Donor/Desynced, mariabackup report completed OK!, and the donor return to Synced when the joiner finishes.

Step 5 — Confirm the joiner reached Synced #

On the joiner, verify it certified into the group rather than starting standalone:

SHOW GLOBAL STATUS WHERE Variable_name IN
  ('wsrep_local_state_comment', 'wsrep_cluster_status', 'wsrep_cluster_size');
-- expect: Synced, Primary, and the full node count

Parameter Deep-Dive: The Knobs That Move SST Behaviour #

A handful of options in the [sst] and [mysqld] sections control transfer speed, safety, and encryption. Tune these deliberately.

wsrep_sst_donor deserves a trailing comma — wsrep_sst_donor="galera-node-3," prefers node-3 but falls back to any member if it is down, whereas omitting the comma fails the join when the named donor is unavailable. parallel is the highest-leverage speed knob on datasets with many tablespaces, but it competes with live traffic for donor I/O, so raise it on a dedicated donor and keep it modest on a node still serving reads. encrypt=4 is the current TLS mode; the older encrypt=2/encrypt=3 modes are deprecated and should not be used for new builds.

Compression, Encryption, and the socat Transport #

Three moving parts sit between the donor’s mariabackup process and the joiner’s data directory, and each is configurable in the [sst] block: the stream format, the transport, and the optional compression and encryption filters layered on top. Understanding how they compose is what lets you tune SST for a slow WAN link or a security-zone crossing without breaking the transfer.

The transport is socat by default, invoked as transferfmt=socat. On the joiner it listens on port 4444 and writes the incoming bytes to mbstream; on the donor it connects to that listener and feeds it the backup stream. socat is preferred over the legacy netcat transport precisely because it can wrap the connection in TLS natively — the encrypt=4 mode is implemented by handing socat an OPENSSL address with the tca/tcert/tkey certificates rather than a plain TCP one, so encryption is transport-level and adds no separate tunnel to manage. The full certificate setup, and reusing the same authority as group communication, is covered in Securing mariabackup SST Credentials & Privileges.

Compression is a filter the SST script inserts into the pipe on the donor and reverses on the joiner. Rather than relying on a built-in mariabackup compressor, the robust pattern is to name external commands so you control the algorithm and thread count:

[sst]
streamfmt     = xbstream
transferfmt   = socat
# Compress on the donor, decompress on the joiner — both nodes need the binary.
compressor    = "zstd -T4"
decompressor  = "zstd -dc"

zstd at a low level gives most of the size reduction of heavier codecs at a fraction of the CPU, which matters because the donor is compressing while still serving traffic. On a fast intra-datacenter 10/25 GbE link, compression is usually a net loss — the CPU spent shrinking the stream exceeds the time saved on the wire — so enable it only when the inter-node link is genuinely the bottleneck, such as a metered or cross-region path. The order of operations is fixed and worth internalizing: mariabackup produces the xbstream, the compressor shrinks it, socat encrypts and ships it, and the joiner reverses that chain (decrypt, decompress, extract) before --prepare reconciles the redo log. A binary present on only one side — a common failure after a partial package rollout — breaks the transfer at exactly the filter that node is missing.

Verification & Health Checks #

After a transfer, confirm both the joiner’s membership and the donor’s return to health. Start with live status:

-- On the joiner: did it certify into the group?
SHOW GLOBAL STATUS WHERE Variable_name IN
  ('wsrep_local_state_comment','wsrep_cluster_status','wsrep_ready');

-- On the donor: did it return from Donor/Desynced to Synced?
SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';

Then validate any scheduled backup the same way the recovery guide insists on — a backup is not restorable until --prepare says so. This Python 3.9+ check drives a backup against a node, prepares it, and gates on the success string, handling Galera’s contention error codes 1213 (deadlock/certification conflict) and 1205 (lock wait timeout):

import subprocess
import sys
import pymysql

def node_is_synced(host: str, user: str, password: str) -> bool:
    try:
        conn = pymysql.connect(host=host, user=user, password=password,
                               connect_timeout=5)
        with conn.cursor() as cur:
            cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'")
            state = cur.fetchone()[1]
    except pymysql.err.OperationalError as exc:
        # 1213 = certification deadlock, 1205 = lock wait timeout: transient.
        if exc.args and exc.args[0] in (1213, 1205):
            print(f"[WARN] {host} contended ({exc.args[0]}); retry")
            return False
        print(f"[FATAL] {host} unreachable: {exc}", file=sys.stderr)
        return False
    finally:
        try:
            conn.close()
        except Exception:
            pass
    return state == "Synced"

def verify_backup(archive: str, workdir: str) -> bool:
    subprocess.run(f"zstd -dc {archive} | mbstream -x -C {workdir}",
                   shell=True, check=True)
    result = subprocess.run(
        ["mariabackup", "--prepare", f"--target-dir={workdir}"],
        capture_output=True, text=True,
    )
    ok = result.returncode == 0 and "completed OK!" in result.stderr
    print("[OK] restorable" if ok else "[FAIL] not restorable")
    return ok

if __name__ == "__main__":
    if not node_is_synced("127.0.0.1", "monitor", "REDACTED"):
        sys.exit(1)
    sys.exit(0 if verify_backup(sys.argv[1], "/tmp/verify") else 1)

Reusable versions of the status probe, wired into alerting, are covered in Monitoring Galera Cluster State with Python.

Automation Integration #

Manual SSTs are fine for a first build; a fleet needs the method rendered from a single source and the backup schedule owned by the same automation. Template the SST block with the rest of wsrep.cnf and gate the change on validation before a restart, exactly as in Automating Node Provisioning with Ansible. For the scheduled-backup side, a systemd timer that runs the Step 3 stream against a Synced, unthrottled node — and pushes the verified archive to object storage — turns backups into a monitored service rather than a fragile cron line. Record the source node’s GTID position with every archive, because that coordinate is what a later point-in-time recovery replays forward from.

Troubleshooting #

For a node stuck reporting the wrong lifecycle state after a failed transfer, the state-machine recovery steps are in Fixing wsrep_local_state_comment Issues, and the broader donor-selection failures are catalogued in Troubleshooting SST Donor Selection.

Related #

• Backup & Restore Sync — the parent guide to protecting and rebuilding Galera cluster state
• mariabackup vs rsync SST: Choosing a State-Transfer Method — the donor-blocking, speed, and encryption trade-offs in full
• Securing mariabackup SST Credentials & Privileges — the least-privilege grant set, secret storage, and TLS for the stream
• gcache Sizing for IST & Fast Recovery — keeping rejoins on the cheap incremental path instead of a full SST
• Choosing the Right SST Method for Large Datasets — method selection when transfer time is dominated by data size