Alerting on Flow Control & Receive-Queue Depth

This guide extends the alerting design in Python Monitoring & Alerting Patterns for Galera and answers a single operational question: which wsrep_ metrics warn you that a MariaDB Galera cluster is about to stall, and where do you set the thresholds? A synchronous group throttles its own write throughput to protect consistency, so the earliest sign of trouble is not an error — it is a rising apply backlog and a growing fraction of time spent paused. Alert on wsrep_flow_control_paused, wsrep_local_recv_queue_avg, and wsrep_flow_control_sent correctly and you get minutes of warning before commit latency spikes cluster-wide; alert on them naively and you drown in false pages during every batch job. This page gives the exact thresholds, the PromQL and Python alert rules, and the causal reasoning that ties each metric to an imminent stall.

Context: Why These Metrics Predict a Stall

Galera commits at the speed of its slowest member because every write-set must be received and eventually applied on every node. When one node applies slower than the group produces write-sets, its received-but-not-yet-applied queue grows. wsrep_local_recv_queue_avg is the average depth of that queue since the metric was last read — it is the leading indicator, rising before any throttling happens. Once the queue crosses the provider’s gcs.fc_limit, that node sends a flow-control pause message to the whole group, incrementing its wsrep_flow_control_sent counter and forcing every other member to stop accepting new writes until the backlog drains. wsrep_flow_control_paused is the fraction of wall-clock time the node spent in that paused state — the lagging indicator that the stall is already happening.

Read as a chain, the three metrics form an early-warning sequence: recv-queue climbs → this node sends flow control → the whole cluster pauses. That is why alerting on the queue depth buys you lead time that alerting on paused-time alone does not. The throttle itself — the gcs.fc_limit and gcs.fc_factor hysteresis these metrics measure — is tuned in Tuning Flow Control to Prevent Cluster Stalls, and the certification pipeline that feeds the apply queue is explained in Write-Set Certification Process Explained.

Escalation chain from receive-queue growth to a group-wide stall Three stages escalate left to right. Stage one, the leading indicator: wsrep_local_recv_queue_avg rises as one node applies write-sets slower than the group produces them, warning at twenty and critical at one hundred. Stage two: when the queue crosses gcs.fc_limit the node sends a flow-control pause, incrementing wsrep_flow_control_sent, which alerts on any sustained rate above zero. Stage three, the lagging indicator: the whole cluster pauses and wsrep_flow_control_paused rises toward one, warning at 0.1 and critical at 0.25, meaning commit latency is already degraded everywhere. 1 · LEADING INDICATOR recv_queue_avg ↑ one node applies slower than the group warn ≥ 20 crit ≥ 100 2 · TRIGGER flow_control_sent ↑ queue crosses gcs.fc_limit alert on any sustained rate > 0 3 · LAGGING INDICATOR flow_control_paused ↑ whole cluster pauses writes warn ≥ 0.10 crit ≥ 0.25 earlier warning ←··· queue depth buys lead time over paused-time ···→ commit latency already degraded
Alert on the leading indicator (recv-queue depth) to get minutes of warning; by the time paused-time is high, the whole cluster is already throttled.

Solution: The Alert Rules

The correct approach differs by metric type. wsrep_flow_control_paused is an interval average reset on read, so in Prometheus you should alert on the rate of the monotonic wsrep_flow_control_paused_ns counter instead, which is scrape-timing-independent. wsrep_flow_control_sent is a monotonic counter, so alert on its rate(). wsrep_local_recv_queue_avg is a gauge you can threshold directly, but require it to hold across a for: window so a single burst does not page.

PromQL alert rules

These consume the gauges from Building a Prometheus Exporter for wsrep Status:

groups:
  - name: galera_flow_control
    rules:
      # Leading indicator: apply backlog building on a member.
      - alert: GaleraRecvQueueHigh
        expr: galera_wsrep_local_recv_queue_avg > 20
        for: 2m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.node }} apply backlog rising (recv_queue_avg {{ $value | printf \"%.1f\" }})"

      # Lagging indicator: paused fraction from the monotonic ns counter.
      - alert: GaleraFlowControlPaused
        expr: rate(galera_wsrep_flow_control_paused_ns[5m]) / 1e9 > 0.10
        for: 3m
        labels: {severity: warning}
        annotations:
          summary: "Cluster throttled: {{ $labels.node }} paused fraction {{ $value | printf \"%.2f\" }}"

      - alert: GaleraFlowControlStalled
        expr: rate(galera_wsrep_flow_control_paused_ns[5m]) / 1e9 > 0.25
        for: 2m
        labels: {severity: critical}
        annotations:
          summary: "Cluster stalling: {{ $labels.node }} paused over 25% of wall-clock time"

      # Trigger: identify which member is the flow-control culprit.
      - alert: GaleraFlowControlSender
        expr: rate(galera_wsrep_flow_control_sent[5m]) > 0.1
        for: 5m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.node }} is sending flow-control pauses to the group"

rate(...paused_ns[5m]) / 1e9 yields the paused fraction over the window: a value of 0.10 means the node was throttled 10% of the time. The GaleraFlowControlSender rule is the one that names the culprit — the member with a non-zero wsrep_flow_control_sent rate is the node whose apply pipeline is holding the group back, which is where you look first.

Python threshold alerting

If you alert from the Python collector rather than Prometheus, the same thresholds apply, but you must compute the paused fraction yourself from successive wsrep_flow_control_paused_ns samples. The evaluator handles the Galera contention codes 1213 (deadlock / certification conflict) and 1205 (lock wait timeout) so a busy node is not misread as unreachable:

"""Flow-control alerting from the Python collector (PyMySQL, 3.9+)."""
import time
import pymysql

WARN_PAUSED, CRIT_PAUSED = 0.10, 0.25
WARN_QUEUE, CRIT_QUEUE = 20.0, 100.0

def _read(host, user, pw):
    conn = pymysql.connect(host=host, user=user, password=pw,
                           connect_timeout=4, read_timeout=4)
    try:
        with conn.cursor() as cur:
            cur.execute("SHOW GLOBAL STATUS WHERE Variable_name IN "
                        "('wsrep_flow_control_paused_ns','wsrep_local_recv_queue_avg',"
                        "'wsrep_flow_control_sent')")
            return {n: v for n, v in cur.fetchall()}
    except pymysql.err.OperationalError as exc:
        if exc.args and exc.args[0] in (1213, 1205):
            return None  # transient contention, skip this sample
        raise
    finally:
        conn.close()

def check_flow_control(host, user, pw, interval=15):
    """Sample twice to derive the paused fraction from the ns counter."""
    first = _read(host, user, pw)
    if first is None:
        return
    time.sleep(interval)
    second = _read(host, user, pw)
    if second is None:
        return

    paused_delta_ns = int(second["wsrep_flow_control_paused_ns"]) - \
                      int(first["wsrep_flow_control_paused_ns"])
    paused_frac = paused_delta_ns / (interval * 1e9)   # ns paused / ns elapsed
    queue = float(second["wsrep_local_recv_queue_avg"])

    if paused_frac >= CRIT_PAUSED:
        yield "critical", f"{host} paused {paused_frac:.0%} of the interval"
    elif paused_frac >= WARN_PAUSED:
        yield "warning", f"{host} paused {paused_frac:.0%} of the interval"
    if queue >= CRIT_QUEUE:
        yield "critical", f"{host} recv_queue_avg={queue:.0f} (apply backlog)"
    elif queue >= WARN_QUEUE:
        yield "warning", f"{host} recv_queue_avg={queue:.0f} (apply backlog)"

Deriving the fraction from the delta of the nanosecond counter over a known elapsed interval avoids the reset-on-read trap that makes wsrep_flow_control_paused unreliable when more than one tool reads status.

Parameter & Threshold Reference

Metric Type Healthy Warn Critical Meaning of a breach
wsrep_local_recv_queue_avg gauge (interval avg) < 5 ≥ 20 ≥ 100 Received write-sets queuing faster than this node applies them
wsrep_flow_control_paused float 0–1 (reset on read) ~0.0 ≥ 0.10 ≥ 0.25 Fraction of the interval the group was throttled; use the ns counter in Prometheus
wsrep_flow_control_paused_ns counter (nanoseconds) flat rising rising fast Cumulative pause time; rate()/1e9 gives the paused fraction
wsrep_flow_control_sent counter flat any sustained rate rising fast This node is the one requesting pauses — the group’s bottleneck
wsrep_flow_control_recv counter flat — — Pause requests received; high everywhere but low _sent means the culprit is elsewhere
wsrep_local_recv_queue gauge (instant) < 5 ≥ 40 ≥ 200 Instantaneous backlog; noisier than the average, use for spot checks

The gcs.fc_limit provider option is the queue depth at which a node actually sends flow control, so tune your recv_queue_avg warning threshold below it — alerting at 20 when gcs.fc_limit is 256 gives ample lead time. The relationship between fc_limit, fc_factor, and these metrics is developed in the wsrep.cnf Configuration Deep Dive.

Verification

Prove the alert fires on a real backlog before trusting it in production. Read the live values on each node and compare _sent across the group to confirm which member is the source:

-- Run on every node; the one with a rising _sent is the bottleneck.
SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_flow_control_paused',      -- interval fraction, reset on read
  'wsrep_flow_control_paused_ns',   -- monotonic; sample twice for the rate
  'wsrep_flow_control_sent',        -- non-zero here = this node throttles the group
  'wsrep_local_recv_queue_avg'      -- the leading indicator
);

Force a backlog safely on a test cluster by throttling one node’s apply pipeline (drop its wsrep_slave_threads to 1 and drive a write burst), then watch wsrep_local_recv_queue_avg climb on that node and wsrep_flow_control_paused_ns rise across all members. Confirm the PromQL expression returns the expected fraction:

rate(galera_wsrep_flow_control_paused_ns[5m]) / 1e9

Edge Cases & Gotchas

  • Never alert on wsrep_flow_control_paused as if it were cumulative. It resets each time status is read, so if your exporter and an ad-hoc SHOW STATUS both read it, each sees a fraction of the true window and under-reports. Alert on rate(wsrep_flow_control_paused_ns[...]) instead — the nanosecond counter is monotonic and immune to who read it last.
  • A single spike is not a stall. wsrep_flow_control_paused legitimately jumps for one interval during a large transaction or a batch load. Require the condition to hold across a for: 2m–3m window (or two to three consecutive Python samples) so momentary bursts do not page; the throttle is doing its job, not failing.
  • High _recv with low _sent means look elsewhere. If a node shows lots of wsrep_flow_control_recv but near-zero wsrep_flow_control_sent, it is a victim of the pause, not its cause — another member is the bottleneck. Always compare wsrep_flow_control_sent across every node to attribute the stall to the correct host before you touch its wsrep_slave_threads or disk.