Speeding Up SST with Parallel Compression

A mariabackup State Snapshot Transfer on a multi-terabyte dataset can pin a network link for an hour, and most of that time is wasted on a single-threaded copy of already-compressible pages — parallel reader threads, a fast streaming compressor, and the right transport turn that hour into minutes. This guide extends Initial Data Synchronization Methods with the concrete tuning that shortens the window a donor spends desynced: thread counts, qpress versus zstd, socat versus nc, and how to actually measure the improvement.

Context: Where SST Time Actually Goes

An SST has three serial costs stacked on top of each other: the donor reads InnoDB pages off disk, the stream is pushed across the network, and the joiner writes and prepares the datadir. On a default configuration every one of these runs with minimal parallelism, so the transfer is bounded by whichever single resource saturates first — usually the network on a busy link, or disk read on a large cold dataset. The lever that matters depends on which is the bottleneck, which is why measuring comes before tuning.

Compression trades CPU for bytes-on-the-wire. On a bandwidth-constrained or cross-datacenter link, compressing the stream on the donor before it hits the network can cut transfer time by half or more, because database pages are highly compressible and the CPU cost of a fast codec is far cheaper than the extra seconds of network transfer it saves. On a fast local link where the network is not the constraint, the same compression is pure overhead — it adds CPU latency without shortening anything. Parallelism attacks a different axis: mariabackup --parallel=N reads N tablespace files concurrently, which helps when disk read throughput, not the wire, is the limit. Getting this right directly shrinks the Donor/Desynced window that couples an SST to group-wide flow control, so it compounds with the donor-selection tuning in Troubleshooting SST Donor Selection.

The mariabackup SST pipeline and where each tuning lever applies The mariabackup SST pipeline runs donor to joiner in four stages. Parallel reader threads read InnoDB pages from the donor disk. A streaming compressor such as zstd or qpress shrinks the stream on the donor CPU. A transport such as socat or nc carries the compressed stream across the network. The joiner decompresses, writes, and prepares the datadir. Parallel threads target the disk-read stage, compression targets the network stage, and both shorten the total donor desync window. DONOR JOINER Parallel read --parallel=N threads disk-bound lever Compress zstd / qpress network lever Transport socat / nc · 4444 compressed stream over the wire Receive socat / nc Decompress + write datadir --prepare apply redo measure first: threads help a disk-bound copy, compression helps a network-bound one
Parallel reader threads attack the disk-read stage; a streaming compressor attacks the network stage. Which lever helps depends on which resource saturates first.

Solution: Tune the mariabackup Stream

mariabackup SST tuning lives in two directives: wsrep_sst_method selects the backend, and wsrep_sst_mariabackup_options passes flags straight through to the mariabackup invocation on both donor and joiner. Set them identically across the fleet.

Enable parallel reads and streaming compression

The baseline high-throughput block enables parallel tablespace reads and a fast streaming compressor. zstd is the modern default — it compresses database pages tightly at low CPU cost and decompresses fast — and it has superseded the older qpress/--compress path on current MariaDB:

[mysqld]
wsrep_sst_method=mariabackup

# Parallel readers + zstd streaming compression, donor and joiner.
wsrep_sst_mariabackup_options="--parallel=4 --compress=zstd --compress-threads=4"

--parallel=4 opens four concurrent file-copy threads; on a many-tablespace dataset this scales disk read throughput close to linearly until the disk or the wire saturates. --compress-threads=4 runs the zstd codec across four cores so compression itself is not the new bottleneck — pairing parallel readers with single-threaded compression just moves the stall. Size both to leave headroom on the donor: a donor already serving production writes should not surrender every core to the SST, so --parallel and --compress-threads together should stay under the donor’s spare core count.

Choose the compressor deliberately

The compressor is the highest-leverage choice on a constrained link. The trade is compression ratio against CPU:

Compressor Ratio CPU cost When to use
zstd (--compress=zstd) High Low–moderate, tunable by level Default for any bandwidth-constrained or cross-DC transfer
qpress (--compress) Moderate Low Legacy path; only where zstd is unavailable in the build
none 1:1 Zero Fast local link where the network is not the bottleneck

On a saturated 1 GbE link between datacenters, zstd typically halves wall-clock transfer time because the bytes removed from the wire outvalue the CPU spent removing them. On a 25 GbE local fabric the same zstd can slow the transfer, because compression throughput becomes the ceiling below line rate — there, drop compression and let --parallel carry the speedup. This is why the measurement step is not optional.

Pick the transport: socat versus nc

The SST script streams the payload over port 4444 using a transport helper. The default on most builds is socat, and it is the right default: socat supports TLS (openssl mode) so the stream can be encrypted in flight, and it handles half-close and buffering more robustly than plain nc. Plain nc (netcat) has marginally lower overhead but no encryption and flakier connection teardown, so reserve it for trusted, non-encrypted local segments. When transfers cross any network boundary, use socat with TLS rather than sending an unencrypted physical copy of your database across the wire:

[mysqld]
# socat transport (default); enable encryption for cross-boundary transfers.
[sst]
encrypt=4
ssl-ca=/etc/mysql/ssl/ca.pem
ssl-cert=/etc/mysql/ssl/server-cert.pem
ssl-key=/etc/mysql/ssl/server-key.pem

Securing that credential and certificate surface is covered in Securing mariabackup SST Credentials & Privileges, and the full streaming-backup mechanics behind the method are in mariabackup Streaming SST & Backups.

Parameter Reference

Flag / key Where Default Recommended Effect
--parallel=N wsrep_sst_mariabackup_options 1 4–8 (≤ spare cores) Concurrent tablespace-copy threads; scales a disk-bound read
--compress=zstd wsrep_sst_mariabackup_options off zstd on constrained links Streaming compression to cut bytes on the wire
--compress-threads=N wsrep_sst_mariabackup_options 1 match --parallel Parallelizes the compressor so it is not the new bottleneck
--compress-level wsrep_sst_mariabackup_options codec default 3–6 for zstd Higher ratio at more CPU; raise only if network-bound
encrypt [sst] 0 4 (TLS via socat) Encrypts the stream across a network boundary
wsrep_sst_method [mysqld] build-dependent mariabackup Non-blocking physical SST that keeps the donor writable

These flags interact with flow control: the faster the transfer completes, the shorter the donor is desynced and the less risk it stalls the group. Size gcs.fc_limit to 256+ alongside this tuning, as detailed in the wsrep.cnf Configuration Deep Dive, and choose the backend against a large-dataset benchmark in Choosing the Right SST Method for Large Datasets.

Verification: Measure the SST Duration

Never tune by feel — time each transfer and compare. The SST script logs start and end timestamps to the joiner’s error log; extract the delta:

# Duration of the most recent SST on the joiner, in seconds
journalctl -u mariadb | grep -E "WSREP_SST.*(Preparing|Total time|Galera SST completed)" \
  | tail -5

# Live throughput while a transfer runs (watch the replication NIC)
sar -n DEV 2 | grep -E "eth0|ens"

For a repeatable measurement, capture the wall-clock of a full join and the peak network utilization together, then change exactly one variable — thread count, compressor, or level — and re-run. This Python helper polls the joiner until it reaches Synced and reports the elapsed time, handling the wsrep contention codes 1213 (deadlock / certification conflict) and 1205 (lock wait timeout) so a busy donor does not abort the measurement:

import sys
import time
import pymysql
from pymysql.err import OperationalError, MySQLError

def time_sst(host: str, user: str, password: str, timeout_s: int = 3600) -> float | None:
    """Poll a joiner until Synced; return elapsed seconds, or None on timeout."""
    start = time.monotonic()
    while time.monotonic() - start < timeout_s:
        try:
            conn = pymysql.connect(host=host, user=user, password=password,
                                   connect_timeout=5, read_timeout=5)
        except OperationalError:
            # Node still mid-SST refuses connections; keep waiting.
            time.sleep(5)
            continue
        try:
            with conn.cursor() as cur:
                cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'")
                state = cur.fetchone()[1]
        except MySQLError as exc:
            if exc.args and exc.args[0] in (1213, 1205):
                time.sleep(5)
                continue
            raise
        finally:
            conn.close()
        if state == "Synced":
            elapsed = time.monotonic() - start
            print(f"[done] {host} reached Synced in {elapsed:.1f}s")
            return elapsed
        time.sleep(5)
    print(f"[timeout] {host} never reached Synced", file=sys.stderr)
    return None

if __name__ == "__main__":
    time_sst("10.0.1.20", "monitor", "secret")

Edge Cases & Gotchas

  • Over-threading starves the live donor. Setting --parallel and --compress-threads to the full core count on a donor that is also serving production writes spikes its apply latency and can itself trip flow control — the opposite of the goal. Leave headroom, or pin the SST to a non-critical donor as in Troubleshooting SST Donor Selection.
  • Compression on a fast link is negative tuning. If the network is not the bottleneck, --compress=zstd adds CPU latency for no wall-clock gain and can reduce throughput below line rate. Measure first; only compress when the wire is the constraint.
  • Donor and joiner options must match. wsrep_sst_mariabackup_options is applied on both ends, so a joiner missing the matching decompression capability (zstd not installed) fails the transfer partway with a stream error. Install the compressor and keep the option string identical across the fleet, validated by the gate in Validating Galera Config in CI Before Restart.