Sizing gcache to Avoid a Full SST on Rejoin

This worksheet extends the configuration guide in gcache Sizing for IST & Fast Recovery and answers one precise question: what number do you put in gcache.size so a node that was down for your worst-case outage rejoins with an Incremental State Transfer instead of a full State Snapshot Transfer? The answer is arithmetic, not guesswork — it is the product of the rate at which write-sets accumulate and the length of the outage you must survive, with headroom for bursts. This page shows you how to measure the write-set rate on a live cluster, gives you the formula, and walks a full worked example from raw counters to a rounded gcache.size you can commit.

Context: Why the Right Number Matters in Multi-Master #

In a synchronous multi-master group, every node caches the same ordered stream of write-sets, and a rejoining member can only skip a full snapshot if the write-sets it missed are still in a donor’s ring buffer. Undersize gcache.size and the arithmetic works against you constantly: a brief kernel upgrade, a leaf-switch reboot, or a five-minute cloud maintenance event produces enough write history to roll the oldest cached seqno past the point the returning node needs, and the provider has no choice but to fall back to SST. That fallback is not free — it desyncs a donor, saturates its disk and network for the duration, and can itself trip flow control across the whole cluster, as covered in Initial Data Synchronization Methods. Getting the number right once, from real measurements, removes an entire class of recovery incidents.

The variable you are solving for is a duration expressed as bytes. The ring holds a fixed number of bytes; at a given write-set rate that translates to a fixed number of seconds of history. Your job is to make that number of seconds comfortably exceed your longest realistic outage.

Solution: Measure the Rate, Then Apply the Formula #

The formula is deliberately simple:

gcache.size (bytes)  =  write_rate (bytes/sec)  ×  outage_seconds  ×  burst_factor

Everything hinges on a trustworthy write_rate. Two counters give it to you directly, and which you pick shapes the answer. wsrep_replicated_bytes counts only the write-sets this node originated; Bytes_received at the GCS level and the applied-byte counters reflect the full group stream. For sizing a ring that must cover any node’s absence, measure the total write-set volume flowing through the group, because a donor must retain everything every peer might need — not just its own writes.

Step 1 — Sample the write-set volume over a peak window #

Take two readings of the replicated-byte counter a fixed interval apart, during your heaviest sustained write load. A 5-minute sample smooths out per-second jitter while still capturing a realistic peak:

#!/usr/bin/env bash
# Measure group-wide write-set byte rate over a 300s peak window.
set -euo pipefail

read_bytes() { mysql -N -B -e "SHOW GLOBAL STATUS LIKE 'wsrep_replicated_bytes';" | awk '{print $2}'; }

INTERVAL=300
before=$(read_bytes)
sleep "$INTERVAL"
after=$(read_bytes)

rate=$(( (after - before) / INTERVAL ))
echo "write-set rate: ${rate} bytes/sec  ($(( rate / 1024 / 1024 )) MB/sec)"

Run this on the busiest writer node, ideally overlapping a known heavy job (nightly ETL, a bulk import, month-end batch) so your rate reflects the worst realistic case rather than a quiet afternoon.

Step 2 — Choose the outage window and burst factor #

outage_seconds is a policy decision, not a measurement: it is the longest a node may be gone while still rejoining on IST. Anchor it to your real operational events — a rolling MariaDB minor upgrade, a kernel reboot, or a cloud instance stop/start typically falls between 5 and 30 minutes. The burst_factor (1.5–2.0) absorbs the reality that your averaged rate hides short spikes far above the mean; without it, a write burst during the outage silently shortens the covered window.

Step 3 — Compute and round up #

Multiply, convert to gigabytes, and round up to a clean value — over-provisioning the ring costs only disk, while under-provisioning costs an SST:

import math

def gcache_size_bytes(write_rate_bps: int, outage_seconds: int, burst: float = 1.75) -> int:
    """Return the minimum gcache.size in bytes to keep a rejoin on IST."""
    return int(write_rate_bps * outage_seconds * burst)

def ceil_to_gib(nbytes: int) -> int:
    """Round a byte count up to whole GiB for a clean gcache.size value."""
    return math.ceil(nbytes / (1024 ** 3))

if __name__ == "__main__":
    rate = 4 * 1024 * 1024          # 4 MB/s measured in Step 1
    window = 30 * 60                # 30-minute worst-case outage
    need = gcache_size_bytes(rate, window)
    print(f"minimum gcache.size: {need:,} bytes  (~{need / 1024**3:.1f} GiB)")
    print(f"set gcache.size={ceil_to_gib(need)}G")
    # -> minimum gcache.size: 13,212,057,600 bytes  (~12.3 GiB)
    # -> set gcache.size=13G   (round to 14G for extra burst margin)

Worked example, end to end #

Suppose Step 1 reports 4 MB/s of write-set volume at peak, and policy sets the outage window to 30 minutes to cover a rolling upgrade. With a burst factor of 1.75:

4 MB/s × 1800 s × 1.75  =  12,600 MB  ≈  12.3 GiB

Round up and set gcache.size=14G. That ring holds roughly 52 minutes of average-rate history, so even a 30-minute outage that coincides with a 1.7× write spike still rejoins on IST. Commit the value in a single provider-options string:

[mysqld]
wsrep_provider_options = "gcache.size=14G; gcache.page_size=1G; gcache.recover=yes"

Parameter Reference #

The burst_factor is not a MariaDB parameter — it is your safety margin. On workloads with predictable, flat write rates, 1.5 is enough; on spiky OLTP with batch overlays, use 2.0.

Verification #

After applying the value, confirm the achieved window matches the target you computed. Convert the retained transaction count into a duration using the live commit rate:

SELECT
  (SELECT VARIABLE_VALUE FROM information_schema.GLOBAL_STATUS
     WHERE VARIABLE_NAME='wsrep_last_committed')
  -
  (SELECT VARIABLE_VALUE FROM information_schema.GLOBAL_STATUS
     WHERE VARIABLE_NAME='wsrep_local_cached_downto')
  AS transactions_retained;

Divide transactions_retained by your measured commits-per-second to get the covered seconds; it should exceed outage_seconds. For the definitive proof, restart one node and confirm the joiner log shows Receiving IST with a seqno range rather than Running: wsrep_sst_mariabackup. If it fell back to SST despite the sizing, the diagnostics are in Diagnosing IST Failures & gcache Eviction.

Edge Cases & Gotchas #

• Averaged rates hide the spike that bites you. A 24-hour mean write rate will badly undersize the ring if your damage window overlaps a nightly batch that writes ten times the average. Always measure during the peak, and keep the burst factor at 2.0 if batch and OLTP write paths share the Galera cluster.
• Large write-sets shrink the effective window. The ring counts bytes, not transactions, so a workload with big BLOB rows or wide multi-row INSERTs fills it far faster than the transaction rate suggests. If your measured rate is dominated by a few huge write-sets, raise both gcache.size and gcache.page_size together.
• The datadir must actually have the space. gcache.size=14G reserves that much on the gcache.dir filesystem in addition to InnoDB. Sizing the ring generously is worthless if it fills the disk and crashes the node — verify free space against the hardware plan in Galera Cluster Hardware Requirements before committing a large value.

Related #

• gcache Sizing for IST & Fast Recovery — the full configuration reference this worksheet feeds
• Diagnosing IST Failures & gcache Eviction — what to do when a rejoin fell back to SST despite sizing
• Initial Data Synchronization Methods — the cost of the SST fallback you are sizing to avoid
• Galera Cluster Hardware Requirements — ensuring the datadir has room for a large ring