gcache Sizing for IST & Fast Recovery

This guide builds on the backup and state-transfer model in Backup & Restore Sync and solves one specific operational problem: how to size the Galera write-set cache so a rejoining node replays a few minutes of missed transactions with a fast Incremental State Transfer instead of copying the entire dataset with a disruptive full snapshot. The gcache is a fixed-size ring buffer of recently replicated write-sets held by every node; when a member restarts after a patch, a crash, or a network blip, the donor can stream just the write-sets that member missed — but only if those write-sets are still in the ring. Size the buffer too small and every rejoin degrades into a full State Snapshot Transfer that desyncs a donor and saturates its I/O; size it deliberately and node recovery becomes a seconds-long, non-disruptive event. This page is the sizing and configuration reference for database administrators and platform teams running MariaDB 10.6 through 11.x with Galera 4.

Concept: The Write-Set Ring Buffer and the IST/SST Fork #

Every write-set the Group Communication System delivers is appended to a local, ordered log called the gcache (GCache, the group cache). Each entry carries the global sequence number (seqno) assigned at certification, so the cache is effectively a contiguous window of history [first_cached .. last_cached]. The buffer is circular: once it fills, the oldest write-sets are overwritten by the newest, so the low-water mark first_cached continuously advances. This window is the entire basis of Incremental State Transfer — the mechanism introduced alongside SST in Initial Data Synchronization Methods.

When a node rejoins, it announces the last seqno it applied before leaving. The provider on the group selects a donor and asks one question: is the joiner’s next required seqno still inside my gcache window? If yes, the donor streams only the missing write-sets over port 4568 — an IST — and the joiner replays them in seconds while the donor stays fully writable. If the required seqno has already aged out of the ring, IST is impossible, and Galera falls back to a full SST over port 4444: the donor snapshots its entire datadir, desyncs for the duration, and the joiner discards and rebuilds its state from scratch. The single variable that decides which path you get is gcache.size relative to how much write history accumulated during the outage.

The practical lesson is that gcache.size is not a performance knob in the usual sense — it is an insurance policy against SST. A node down for ninety seconds during a package upgrade should never trigger a multi-hour full copy just because the buffer defaulted to 128 MB and a write burst rolled the window forward. The rest of this page turns that principle into a concrete sizing procedure, a parameter reference, and the verification you need to prove a rejoin actually took the IST path.

Prerequisites & Environment #

Because several gcache parameters are read only when the provider loads, and because the buffer lives on disk, validate the following before you change anything:

• MariaDB 10.6 LTS or later with the Galera 4 provider libgalera_smm.so. Provider option defaults shift between releases, so confirm the running defaults with SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options' rather than assuming.
• A writable, monitored data directory for the gcache file. By default the ring lives as galera.cache in the datadir; a full filesystem there will crash the node, so the gcache.size you choose must fit alongside InnoDB with headroom.
• A baseline write-rate measurement. You cannot size the ring without knowing how fast write-sets accumulate. Capture wsrep_replicated_bytes over a representative window — the full measurement routine is in Sizing gcache to Avoid a Full SST on Rejoin.
• A change window for a rolling restart. gcache.size, gcache.page_size, and gcache.recover are all load-time options; changing them requires restarting the node, one at a time, so the group never drops below quorum. The safe sequencing is in Graceful Node Join and Leave Procedures.
• Consistent provider options across the group. Divergent gcache settings do not fail loudly; they surface later as a page-size mismatch on rejoin. Own the value in exactly one drop-in per the discipline in wsrep.cnf Configuration Deep Dive.

Step-by-Step: Size and Configure gcache for Reliable IST #

The workflow is: measure the write rate, multiply by your worst-case outage window, set the ring size (plus the page and recovery options), then roll the change out and prove a rejoin stays on IST.

Step 1 — Measure the sustained write-set rate #

gcache fills at the rate write-sets are replicated, not at the rate of raw SQL. Sample the replicated byte counter twice over a fixed interval during peak write load and divide by the elapsed seconds:

# Two samples 300 s apart; bytes/sec of write-set volume entering gcache
before=$(mysql -N -B -e "SHOW GLOBAL STATUS LIKE 'wsrep_replicated_bytes';" | awk '{print $2}')
sleep 300
after=$(mysql -N -B -e "SHOW GLOBAL STATUS LIKE 'wsrep_replicated_bytes';" | awk '{print $2}')
echo "write-set rate: $(( (after - before) / 300 )) bytes/sec"

Run this during your heaviest sustained write window, not an idle one — the ring must survive an outage that happens to coincide with a batch job. Treat the result as bytes per second; the sizing arithmetic that turns it into a gcache.size is worked through in full on the gcache sizing worksheet.

Step 2 — Set gcache.size to cover the worst-case outage #

Multiply the measured rate by the longest outage you must survive on IST, then add generous headroom (a factor of 1.5–2×) for write bursts above the average. If a node accumulates 4 MB/s of write-sets and you want to cover a 30-minute maintenance window, that is 4 MB/s × 1800 s ≈ 7 GB; round up to 12G:

[mysqld]
# gcache and its siblings must all live in ONE provider-options string
wsrep_provider_options = "gcache.size=12G; gcache.page_size=1G; gcache.recover=yes"

Remember that wsrep_provider_options is a single semicolon-delimited value: a second declaration replaces the whole string rather than appending, so every gcache key must sit in the same line as your gcs and evs tuning.

Step 3 — Set gcache.page_size for large transactions #

When a single write-set (or the in-memory ring) exceeds the RAM allotment, Galera spills to on-disk page-store files sized by gcache.page_size. A page size that is too small forces many tiny files and slows large-transaction handling; too large wastes space. Match it to your largest expected transaction — 1G is a sane production default and comfortably absorbs bulk INSERT ... SELECT or schema changes:

[mysqld]
wsrep_provider_options = "gcache.size=12G; gcache.page_size=1G; gcache.keep_pages_size=2G"

gcache.keep_pages_size tells the provider to retain that many bytes of allocated page-store files rather than deleting them the instant they drain, which avoids churn when write volume is spiky. Its role in eviction is examined in Diagnosing IST Failures & gcache Eviction.

Step 4 — Enable gcache.recover for restart-surviving IST #

By default a clean or crash restart discards the on-disk gcache, so the very first rejoin after a restart is forced to SST even though the write history existed moments earlier. Setting gcache.recover=yes makes the provider validate and reuse the existing galera.cache at startup, preserving the cached window across the restart so the node can still donate — and receive — IST:

[mysqld]
wsrep_provider_options = "gcache.size=12G; gcache.page_size=1G; gcache.recover=yes"

This single option is the difference between a rolling restart that stays entirely on IST and one that triggers an SST storm the moment the first node comes back.

Step 5 — Roll out and confirm the ring window #

Apply the change with a rolling restart, waiting for each node to reach Synced before the next, then confirm the cache actually spans a useful window. The provider reports the current low- and high-water seqnos as status variables:

SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_local_cached_downto',  -- oldest seqno still in gcache (the F in the diagram)
  'wsrep_last_committed'        -- newest applied seqno (approx. L)
);

The difference wsrep_last_committed − wsrep_local_cached_downto is the number of transactions your ring currently holds; divide by your commit rate to get the outage duration it covers right now.

Parameter Deep-Dive: The gcache Knobs That Matter #

These six provider options govern the ring buffer. Tune size, page_size, and recover deliberately; leave the rest at defaults unless a specific symptom points at them.

gcache.size dominates every recovery outcome: it must exceed the write volume produced across your longest realistic outage plus burst headroom. gcache.page_size only engages for write-sets larger than the in-memory allocation, so raise it if you run large bulk loads or online DDL. gcache.keep_pages_size trades disk for stability — keep a couple of gigabytes allocated on write-spiky systems so the provider is not constantly creating and deleting page files. gcache.recover should be yes on essentially every production node; its only cost is a brief cache-validation pass at startup. gcache.dir matters for I/O: the ring is written on the hot path of every commit, so it belongs on the same class of low-latency local storage as your redo log, never on NFS or a network block device where a stall would propagate straight into flow control.

Verification & Health Checks #

After the rolling change, prove two things: the ring is large enough, and a real rejoin took IST rather than SST. First, quantify the window in time, not just bytes:

SELECT
  @cached := (SELECT VARIABLE_VALUE FROM information_schema.GLOBAL_STATUS
              WHERE VARIABLE_NAME='wsrep_local_cached_downto') AS oldest_cached_seqno,
  @last := (SELECT VARIABLE_VALUE FROM information_schema.GLOBAL_STATUS
            WHERE VARIABLE_NAME='wsrep_last_committed') AS newest_seqno,
  (@last - @cached) AS transactions_retained;

Then, after restarting a node, confirm which path it took by reading the donor and joiner logs — an IST logs Receiving IST and a seqno range, while an SST logs Running: wsrep_sst_mariabackup. A Python probe makes this a gate. It uses PyMySQL, computes the retained-window duration from the commit rate, and handles the two contention error codes Galera surfaces under load — 1213 (deadlock / certification conflict) and 1205 (lock wait timeout) — so a busy node is not misread as unhealthy:

import sys
import time
import pymysql

def status(cur, name):
    cur.execute("SHOW GLOBAL STATUS LIKE %s", (name,))
    row = cur.fetchone()
    return int(row[1]) if row else 0

def gcache_window_seconds(host: str) -> float:
    """Estimate how many seconds of outage the current gcache ring covers."""
    try:
        conn = pymysql.connect(host=host, user="monitor", password="secret",
                               connect_timeout=5, read_timeout=5)
    except pymysql.err.OperationalError as exc:
        print(f"[FATAL] {host} unreachable: {exc}", file=sys.stderr)
        return -1.0
    try:
        with conn.cursor() as cur:
            cached_downto = status(cur, "wsrep_local_cached_downto")
            last_1 = status(cur, "wsrep_last_committed")
            time.sleep(10)
            last_2 = status(cur, "wsrep_last_committed")
    except pymysql.err.OperationalError as exc:
        # 1213 = certification deadlock, 1205 = lock wait timeout; transient under load.
        if exc.args and exc.args[0] in (1213, 1205):
            print(f"[WARN] {host} contended ({exc.args[0]}); retry", file=sys.stderr)
            return -1.0
        raise
    finally:
        conn.close()

    commit_rate = (last_2 - last_1) / 10.0  # transactions/sec
    retained = last_1 - cached_downto
    if commit_rate <= 0:
        print(f"[OK] {host} idle; ring holds {retained} transactions")
        return float("inf")
    seconds = retained / commit_rate
    print(f"[OK] {host} gcache covers ~{seconds:,.0f}s at {commit_rate:,.0f} txn/s")
    return seconds

if __name__ == "__main__":
    covered = gcache_window_seconds(sys.argv[1] if len(sys.argv) > 1 else "127.0.0.1")
    # Fail the gate if the ring covers less than a 15-minute outage.
    sys.exit(0 if covered >= 900 else 1)

Wire this into the reusable probes in Monitoring Galera Cluster State with Python and alert when the covered window drops below your longest planned maintenance outage — that is the early warning that your next rejoin will fall back to SST.

Automation Integration #

Sizing is not a one-time act: write volume grows, and a ring that covered thirty minutes last quarter may cover eight today. Treat the covered-window metric as a first-class SLO. The following Ansible task renders the provider-options string from an inventory variable so the whole fleet shares one source of truth, and gates the restart behind the config validation described in Validating Galera Config in CI Before Restart:

- name: Render Galera provider options with sized gcache
  ansible.builtin.template:
    src: 60-galera.cnf.j2
    dest: /etc/mysql/mariadb.conf.d/60-galera.cnf
  register: galera_cfg

- name: Validate merged configuration before any restart
  ansible.builtin.command: mariadbd --validate-config --defaults-file=/etc/mysql/mariadb.cnf
  changed_when: false

- name: Rolling restart one node at a time when config changed
  ansible.builtin.systemd:
    name: mariadb
    state: restarted
  when: galera_cfg.changed
  throttle: 1   # serialize so the group never drops below quorum

The template that renders gcache.size from write_rate_bytes × outage_seconds × burst_factor closes the loop between measurement and configuration; the provisioning pattern it plugs into is Automating Node Provisioning with Ansible.

Troubleshooting #

These are the failures that trace directly to gcache sizing or configuration rather than a network fault.

For the deeper diagnostic workflow — reading the exact donor/joiner log lines that reveal why a seqno was not found, and how page eviction interacts with gcache.keep_pages_size — work through Diagnosing IST Failures & gcache Eviction. For the arithmetic that prevents these symptoms in the first place, use the gcache sizing worksheet.

Related #

• Sizing gcache to Avoid a Full SST on Rejoin — the write-rate measurement and the sizing formula with a worked example
• Diagnosing IST Failures & gcache Eviction — why a rejoin fell back to SST and how to read the eviction logs
• Backup & Restore Sync — the parent guide to state transfer, backups, and recovery
• Initial Data Synchronization Methods — the full SST versus IST decision model behind the ring buffer
• wsrep.cnf Configuration Deep Dive — owning provider options in one validated drop-in