Tuning Flow Control to Prevent Cluster Stalls

This guide extends the commit-path model in Understanding Galera Synchronous Replication and answers a single sharp question: how do you tune Galera flow control so one slow node stops throttling the entire group without letting the receive queue grow into a consistency risk? Flow control is the back-pressure valve that keeps fast writers from outrunning slow appliers, and its defaults were chosen for a small LAN cluster, not a busy production OLTP group. When a member’s apply queue crosses gcs.fc_limit it broadcasts a pause message and every node stops replicating new write-sets until the laggard drains — so a single overloaded disk or an undersized apply pipeline manifests as a group-wide stall. This page shows exactly which knobs govern that behaviour, what production values to set, and how to prove the pauses are gone.

Context: Why Flow Control Halts the Whole Group

Flow control is not a per-connection throttle; it is a group-wide freeze. Every node tracks its own wsrep_local_recv_queue — the count of write-sets that have been received and totally ordered but not yet applied to InnoDB. When that queue length reaches gcs.fc_limit, the node emits an FC_PAUSE message over the Group Communication System, and the provider on every member stops delivering new write-sets until the slow node’s queue drains back below a release threshold and it sends FC_CONT. Because the pause is broadcast, the throughput ceiling of the group is set by whichever member has the shortest fuse relative to its apply speed — usually the node with the slowest storage, the fewest apply threads, or the most contended workload.

That is why raising gcs.fc_limit in isolation is a trap. A higher limit lets the slow node buffer more write-sets before it pauses the group, which smooths bursty traffic, but it also widens the window in which that node holds data the rest of the group has already committed. The real fix is usually to make the queue drain faster by widening apply parallelism with wsrep_slave_threads — a lever detailed in how Galera handles concurrent writes in multi-master — and only then to size the limit for the burst profile. Flow control and apply threads are two ends of the same pipe: threads set the drain rate, fc_limit sets the bucket depth, and gcs.fc_factor sets how far the bucket must empty before writers are released.

Flow-control hysteresis and the group-wide pause it triggers A slow node's receive queue fills toward gcs.fc_limit. When the queue depth reaches fc_limit the node broadcasts an FC_PAUSE message that stops every member from replicating new write-sets. Apply threads keep draining the queue; once it falls back to fc_limit multiplied by gcs.fc_factor the node broadcasts FC_CONT and the whole group resumes. Wider wsrep_slave_threads drains the queue faster and keeps the queue below fc_limit. Slow node · receive queue gcs.fc_limit queue = limit → PAUSE fc_limit × fc_factor queue drains here → CONT apply threads drain (wsrep_slave_threads) FC_PAUSE broadcast over GCS ENTIRE GROUP · replication frozen Node A writes blocked Node B writes blocked Node C writes blocked Slow node draining queue throughput ceiling = slowest member's drain rate every writer waits for the laggard's FC_CONT resume on FC_CONT
A single node's queue crossing gcs.fc_limit freezes replication on every member; the group resumes only when that node drains to fc_limit × fc_factor and sends FC_CONT.

Solution: Tune the Drain Rate First, Then the Bucket

The correct order of operations is to eliminate the avoidable pauses before you loosen the safety limit. Work through it deliberately.

1. Widen apply parallelism so the queue drains faster. wsrep_slave_threads is the number of parallel apply threads on the receive side. Raising it lets non-conflicting write-sets apply concurrently, which is the direct lever on drain rate. Set it live and persist it to a version-controlled drop-in — the precedence rules are in the wsrep.cnf configuration deep dive:

SET GLOBAL wsrep_slave_threads = 8;
[mysqld]
# Match to available apply cores; do not exceed the workload's
# average wsrep_cert_deps_distance or threads sit idle.
wsrep_slave_threads = 8

Parallelism above the workload’s natural dependency distance buys nothing — a serial hot-row workload pins to one apply thread no matter how many you provision. Diagnose that ceiling with wsrep_cert_deps_distance before adding threads.

2. Set the limit and the release factor as one provider-options string. gcs.fc_limit is the queue depth that triggers the pause; gcs.fc_factor is the fraction of that limit the queue must fall back to before writers resume. They are two keys inside the single semicolon-delimited wsrep_provider_options value — a second declaration replaces the whole string rather than merging, so keep every key on one line:

[mysqld]
wsrep_provider_options = "gcs.fc_limit=256; gcs.fc_factor=0.8; gcs.fc_single_primary=NO"

A low limit paired with a high factor (say 16 and 1.0) produces oscillation — the node pauses, drains one write-set, resumes, and immediately re-pauses. A limit of 256 with a factor of 0.8 gives the apply pipeline room to drain roughly 20% of the bucket before writers are released, damping that flapping. The low-latency variants of this string are compared in configuring wsrep_provider_options for low latency.

3. Declare whether writes are single-primary. gcs.fc_single_primary (the current name for the legacy gcs.fc_master_slave) tells the provider whether more than one node originates writes. In multi-master mode (NO, the default) Galera inflates the effective limit as members are added, on the assumption that any node can source a burst. If you pin all writes to one node — the single-node write-affinity pattern that also cuts certification conflicts — set it to YES so the effective limit is not silently multiplied and back-pressure engages at the value you actually configured.

Parameter Reference

Parameter Scope Type Default Recommended
gcs.fc_limit wsrep_provider_options integer 16 128–512 on fast networks with wide apply parallelism
gcs.fc_factor wsrep_provider_options float 0–1 1.0 0.8 to release once the queue drains ~20% and damp oscillation
gcs.fc_single_primary wsrep_provider_options boolean NO YES only when all writes are pinned to one node
wsrep_slave_threads [mysqld] integer 1 8–16, matched to apply cores and wsrep_cert_deps_distance
wsrep_local_recv_queue_avg status (read) float 0.0 investigate a sustained trend above 1.0
wsrep_flow_control_paused status (read) float 0–1 0.0 alert on sustained values above 0.05

gcs.fc_limit is the highest-leverage knob once apply is already wide: too low and a normal write burst pauses the group; too large and a lagging node holds an unbounded backlog of committed-elsewhere data. gcs.fc_factor at 0.8 is the near-universal production choice because the historical default of 1.0 releases the pause the instant the queue drops by a single write-set, causing pause/resume flapping under sustained load.

Verification

Confirm the tuning took effect and that pauses have actually stopped. First read the live counters across the group:

SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_flow_control_paused',   -- fraction of the interval spent paused
  'wsrep_flow_control_sent',     -- FC_PAUSE messages this node emitted
  'wsrep_flow_control_recv',     -- FC_PAUSE messages this node received
  'wsrep_local_recv_queue_avg',  -- mean apply-queue depth
  'wsrep_cert_deps_distance'     -- available apply parallelism
);

wsrep_flow_control_sent climbing on one specific node identifies the laggard that is throttling everyone — that is the node to give more apply threads or faster storage, not the whole fleet. wsrep_flow_control_paused is a fraction of wall-clock time; anything sustained above 0.05 means writers are being throttled 5% of the time or more.

Because wsrep_flow_control_paused is cumulative since the last FLUSH STATUS, measure its rate over a window rather than reading the raw total. This Python 3.9+ probe samples twice, computes the pause delta, and handles the two contention codes a busy node surfaces — 1213 (deadlock / certification conflict) and 1205 (lock wait timeout):

import sys
import time
import pymysql

PAUSE_ALERT = 0.05  # max acceptable fraction of an interval spent paused

def flow_control_pause_rate(host: str, window_s: float = 10.0) -> float:
    conn = pymysql.connect(host=host, user="monitor", password="secret",
                           connect_timeout=5, read_timeout=5)
    try:
        with conn.cursor() as cur:
            def paused():
                cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_flow_control_paused'")
                return float(cur.fetchone()[1])
            first = paused()
            time.sleep(window_s)
            second = paused()
    except pymysql.err.OperationalError as exc:
        # 1213 = certification conflict, 1205 = lock wait timeout: transient.
        if exc.args and exc.args[0] in (1213, 1205):
            print(f"[RETRY] {host}: transient wsrep conflict {exc.args[0]}", file=sys.stderr)
            return 0.0
        raise
    finally:
        conn.close()
    # wsrep_flow_control_paused already resets its window internally on read,
    # so the second sample is the fraction paused over the elapsed window.
    return second

if __name__ == "__main__":
    rate = flow_control_pause_rate(sys.argv[1] if len(sys.argv) > 1 else "127.0.0.1")
    print(f"flow-control paused fraction: {rate:.3f}")
    sys.exit(1 if rate > PAUSE_ALERT else 0)

Wire the same sample into a scrape loop and page on a sustained breach, using the pattern in alerting on flow control and receive-queue depth and the broader automated node health monitoring guide.

Edge Cases & Gotchas

  • A single oversized transaction pauses everyone at once. A multi-gigabyte write-set serializes the apply pipeline on every node, so the receive queue on all members spikes and flow control fires cluster-wide even though nothing is misconfigured. No fc_limit value fixes this — batch bulk DML into chunks of a few thousand rows so no one write-set dominates the total order.
  • Raising gcs.fc_limit without raising apply threads just hides the symptom. A larger bucket delays the pause but does nothing for the drain rate, so the lagging node accumulates a deeper backlog and its recovery pause, when it finally comes, is longer. Always confirm wsrep_local_recv_queue_avg is falling after the change, not just that pauses are less frequent.
  • Cross-availability-zone jitter can look like a flow-control problem but is a topology problem. If pauses persist after apply is wide and the limit is generous, the bottleneck is inter-node round-trip time, not queue depth — measured and remediated in measuring replication latency and RTT impact. Stretching a synchronous group across a high-latency link is a design error that no flow-control tuning repairs.