Configuring wsrep_provider_options for Low Latency

This page extends the Write-Set Certification Process Explained reference and answers one focused question: which wsrep_provider_options directives actually reduce commit latency in a MariaDB Galera group, and how do you apply them without triggering flow-control storms or EVS view changes? Synchronous multi-master replication pays for its consistency guarantee with a group-communication round trip on every COMMIT, so when latency creeps up the fix is rarely a bigger machine — it is a precise change to the Group Communication System (GCS) and Extended Virtual Synchrony (EVS) knobs inside the provider-options string.

Why This Tuning Matters in a Multi-Master Group #

In an active-active Galera group, a client that issues COMMIT blocks until its write-set is ordered by GCS and certified on every node, exactly as described in the Write-Set Certification Process Explained walkthrough. That means three separate latency sources stack on top of raw network round-trip time: the GCS ordering delay, the depth of the receive queue before flow control throttles writers, and the per-frame CPU cost of checksums and message coalescing. The wsrep_provider_options (deep-dived in the wsrep.cnf configuration guide) string is where all three are tuned, so it is the single most productive place to spend a latency budget once the synchronous replication commit path itself is understood.

The trap is that these options interact. Push evs.user_send_window too high and a transient partition floods a peer’s memory; drop gcs.fc_limit too low and you trade queue latency for premature backpressure. The goal is not the smallest possible value for each knob but a balanced string that lowers the median commit while keeping the tail bounded during network jitter.

Solution: A Low-Latency Provider-Options String #

The provider-options string is a single semicolon-delimited value. A second wsrep_provider_options declaration replaces the whole string rather than merging into it, so every knob you tune must live in one line. Apply this in the [mysqld] section of a version-controlled drop-in:

[mysqld]
wsrep_provider_options="gcs.fc_limit=32; gcs.fc_factor=0.7; evs.user_send_window=512; evs.send_window=512; cert.optimistic_pa=YES; socket.checksum=0"

Reading each significant directive:

• gcs.fc_limit=32 raises the receive-queue depth a node tolerates before it halts the group for flow control. The default of 16 stalls high-throughput OLTP by throttling on brief bursts; a moderate lift absorbs the burst so writers keep committing.
• gcs.fc_factor=0.7 sets the fraction of gcs.fc_limit the queue must drain back to before the pause releases. Pairing a higher limit with 0.7 widens the hysteresis band and stops the rapid throttle/unthrottle oscillation that shows up as latency spikes.
• evs.user_send_window=512 and evs.send_window=512 let a node keep up to 512 messages in flight before it must block for acknowledgement. On a low-latency, high-bandwidth LAN this hides round-trip wait during bulk commits. Keep evs.send_window ≥ evs.user_send_window, and never exceed 2048, or a partition can pin excessive memory.
• cert.optimistic_pa=YES keeps optimistic parallel apply on so apply threads start non-conflicting write-sets before the certification interval fully closes — a direct commit-latency win on low-conflict workloads. It is already the Galera 4 default; the reason to state it explicitly is to stop a stray rollback string from disabling it.
• socket.checksum=0 disables the per-frame CRC32C checksum, saving CPU on every replicated packet. Only do this on a trusted internal network that already error-checks at the transport or link layer.

Most of these are dynamically settable, so you can validate on one node before persisting. Set it live, then write the same value into the drop-in so a restart does not revert it:

SET GLOBAL wsrep_provider_options =
  'gcs.fc_limit=32; gcs.fc_factor=0.7; evs.user_send_window=512; cert.optimistic_pa=YES; socket.checksum=0';

Parameter Reference #

Types, defaults, and low-latency starting points for the directives involved. Tune against measured load rather than treating these as constants.

Verification #

Confirm the running provider actually loaded the string you set — a typo in one directive is silently ignored, leaving the rest applied:

SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options';

Then watch the counters that prove the tuning helped rather than hurt. Persistent increments on the flow-control counters mean you traded queue latency for backpressure and should back the change out:

SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_flow_control_sent',     -- times this node throttled the group; should stay flat
  'wsrep_flow_control_paused',   -- fraction of time paused; keep well below 0.05
  'wsrep_local_recv_queue_avg',  -- avg receive-queue depth; should track apply threads
  'wsrep_cert_deps_distance'     -- apply parallelism the workload actually allows
);

To attribute latency to the transport rather than certification, capture GCS traffic on port 4567 and isolate the round trips that exceed your budget:

tcpdump -i any port 4567 -w galera_latency.pcap &
sleep 30 && kill %1
tshark -r galera_latency.pcap -Y "tcp.analysis.ack_rtt > 0.05" \
  -T fields -e frame.time_delta -e ip.src -e tcp.analysis.ack_rtt

If network RTT stays flat but wsrep_local_cert_failures climbs, the bottleneck is in the certification layer, not the wire — no GCS/EVS knob will help, and you should return to the certification process reference.

A small Python probe turns the check into a gate. It targets Python 3.9+, uses PyMySQL, and treats the two wsrep conflict codes — 1213 (deadlock / certification failure) and 1205 (lock-wait timeout) — as transient rather than as a failed tuning run:

import sys
import pymysql

FC_PAUSE_MAX = 0.05  # alert if the node spends >5% of the interval paused

def flow_control_ok(host: str, user: str, password: str) -> bool:
    try:
        conn = pymysql.connect(host=host, user=user, password=password,
                               connect_timeout=5, read_timeout=5)
    except pymysql.err.OperationalError as exc:
        print(f"[FAIL] {host}: cannot connect: {exc}", file=sys.stderr)
        return False
    try:
        with conn.cursor() as cur:
            cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_flow_control_paused'")
            paused = float(cur.fetchone()[1])
    except pymysql.err.OperationalError as exc:
        if exc.args and exc.args[0] in (1213, 1205):  # transient wsrep conflict
            print(f"[RETRY] {host}: transient conflict {exc.args[0]}", file=sys.stderr)
            return False
        raise
    finally:
        conn.close()

    ok = paused < FC_PAUSE_MAX
    print(f"[{'OK' if ok else 'FAIL'}] {host}: flow_control_paused={paused:.4f}")
    return ok

if __name__ == "__main__":
    sys.exit(0 if flow_control_ok(sys.argv[1], "monitor", "monitor-pass") else 1)

Edge Cases & Gotchas #

• A second wsrep_provider_options line drops your settings. Because the string is one value, splitting gcache, gcs, and evs keys across two declarations means the last line wins and the earlier keys vanish with no error. Keep every provider option in a single declaration and diff the live value against your template.
• gcs.max_packet_size and aggressive send windows must match on every node. A per-node mismatch, or a send window so large it overruns a slow peer during a blip, triggers repeated EVS view changes. Recover with a coordinated graceful restart following the node join and leave procedures, and confirm wsrep_cluster_size returns to the expected count.
• Cloud and container images ship conservative defaults. Managed MariaDB images, Docker entrypoints, and systemd drop-ins frequently inject their own wsrep_provider_options at boot, silently overriding your file. Cross-AZ links also have higher jitter than a LAN, so socket.checksum=0 is unsafe there and the EVS timeout defaults evict too eagerly — verify the effective value after every image or unit change, and keep low-latency profiles for LAN groups only.

Related #

• Write-Set Certification Process Explained — the parent reference on how certification and flow control decide commit latency
• wsrep.cnf Configuration Deep Dive — load precedence and validation for the provider-options string
• Understanding Galera Synchronous Replication — the commit path that sets the latency floor these knobs work within
• Network Security & Firewall Rules for Galera — securing the 4567/4568 ports these directives depend on