Measuring Replication Latency & RTT Impact

This guide extends Understanding Galera Synchronous Replication and answers a measurement question rather than a tuning one: how much of your commit latency is inter-node round-trip time, and how do you prove it with numbers instead of guessing? Because a Galera commit blocks until its write-set is ordered by every reachable member, the network round-trip to the slowest peer is added to every write — it is a floor no wsrep_slave_threads setting can remove. Before you re-architect a topology or blame the application, you need a defensible commit-latency budget: how many milliseconds the network contributes, how much the apply queue adds, and where the line sits at which a cross-availability-zone link makes a synchronous group untenable. This page gives you the exact status variables, network probes, and arithmetic to draw that budget.

Context: Why RTT Taxes Every Commit #

In an asynchronous topology the source commits locally and ships changes afterward, so replication distance is invisible to the writing client. Galera inverts that: the originating connection is held from the moment it sends COMMIT until the Group Communication System has ordered the write-set across the primary component. That ordering requires at least one network round trip to the furthest voting member, so if the round-trip time (RTT) to your most distant node is 8 ms, every commit on the group carries an irreducible ~8 ms consensus tax on top of local fsync and apply time. Read-only statements are unaffected — they never enter the write-set path — which is why a group can show fast SELECT latency while writes crawl.

Two distinct costs hide inside a slow commit, and measuring them separately is the whole discipline. The first is the ordering cost: the GCS round trip, governed purely by network RTT and packet loss on port 4567. The second is the apply cost: how long the write-set waits in a receiver’s queue behind other work, which surfaces as flow control when it grows — the subject of tuning flow control to prevent cluster stalls. A latency problem you can fix with more apply threads is an apply-cost problem; a latency problem that persists no matter how you tune is an ordering-cost problem, and the only cure for that is shorter network distance.

Solution: Measure the Network and the Queue Separately #

Draw the budget in two independent measurements — the raw network path, then the provider’s own view of latency and queueing.

1. Baseline the raw inter-node RTT. Measure the network floor before touching the database. Use ping for ICMP RTT and a TCP-level probe against the GCS port, because ICMP and TCP can diverge under a busy or shaped link:

# ICMP round-trip to the most distant peer (50 tight samples)
ping -c 50 -i 0.2 10.0.1.13 | tail -3

# TCP connect time to the GCS port — the path a write-set actually travels
for i in $(seq 20); do
  curl -o /dev/null -s -w "%{time_connect}\n" telnet://10.0.1.13:4567
done | awk '{s+=$1; n++} END {printf "avg tcp connect: %.2f ms\n", (s/n)*1000}'

Record the mean and the tail (p99). A stable 0.3 ms mean with a 40 ms p99 is a jitter problem that will trip flow control intermittently even though the average looks fine.

2. Read the provider’s replication-latency counters. Modern MariaDB exposes the commit round-trip directly, so you do not have to infer it. Capture these under real write load, not idle:

SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_avg_replication_latency',  -- mean seconds a write-set spends being ordered
  'wsrep_local_send_queue_avg',     -- mean depth of the outbound send queue
  'wsrep_local_recv_queue_avg',     -- mean depth of the inbound apply queue
  'wsrep_evs_delayed',              -- peers currently flagged as slow/unreachable
  'wsrep_evs_repl_latency'          -- EVS-layer replication latency histogram (min/avg/max)
);

wsrep_avg_replication_latency is the closest thing to a direct commit-tax reading: it is the average time a write-set spends between broadcast and ordered delivery, so it tracks RTT plus GCS overhead. A rising wsrep_local_send_queue_avg means this node is producing write-sets faster than the network can ship them — an outbound-side symptom of an undersized or congested link, distinct from the inbound apply pressure that wsrep_local_recv_queue_avg reports.

3. Watch the EVS delayed list for the node that is dragging the group. The Extended Virtual Synchrony (EVS) layer maintains a per-peer view of responsiveness. wsrep_evs_delayed names any member the local node currently considers slow or intermittently unreachable, and wsrep_evs_repl_latency gives a min/avg/max/stddev histogram of the messaging latency the EVS layer actually observed:

SHOW GLOBAL STATUS LIKE 'wsrep_evs_delayed';
SHOW GLOBAL STATUS LIKE 'wsrep_evs_repl_latency';

A non-empty wsrep_evs_delayed is the provider telling you which peer is the tax collector — that is the node to move closer or drop from the synchronous membership.

Parameter / Metric Reference #

wsrep_avg_replication_latency compared against the raw ping mean is the single most useful cross-check: if the replication latency is far larger than RTT, the cost is GCS-side queueing, not the wire; if the two track closely, you are latency-bound by distance and no tuning will help. On cross-AZ links, loosen evs.suspect_timeout toward PT10S so normal jitter does not cause a false eviction — the timeout mechanics are detailed in the wsrep.cnf configuration deep dive.

Verification #

Turn the readings into a repeatable budget check. This Python 3.9+ probe records wsrep_avg_replication_latency, compares it to a measured RTT floor, and flags when the ordering tax exceeds budget. It handles the two contention codes a busy node returns — 1213 (deadlock / certification conflict) and 1205 (lock wait timeout):

import subprocess
import sys
import mysql.connector
from mysql.connector import errorcode

RTT_BUDGET_MS = 3.0  # ceiling for a workable same-region synchronous group

def measured_rtt_ms(peer: str) -> float:
    out = subprocess.run(["ping", "-c", "10", "-i", "0.2", peer],
                         capture_output=True, text=True, check=True).stdout
    # parse the "rtt min/avg/max/mdev = a/b/c/d ms" summary line
    avg = out.strip().splitlines()[-1].split("=")[1].split("/")[1]
    return float(avg)

def replication_latency_ms(host: str) -> float:
    conn = mysql.connector.connect(host=host, user="monitor", password="secret",
                                   connection_timeout=3, database="information_schema")
    try:
        cur = conn.cursor()
        cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_avg_replication_latency'")
        row = cur.fetchone()
        return float(row[1]) * 1000.0 if row else 0.0
    except mysql.connector.Error as err:
        if err.errno in (errorcode.ER_LOCK_DEADLOCK, errorcode.ER_LOCK_WAIT_TIMEOUT):
            print(f"[RETRY] {host}: transient wsrep conflict {err.errno}", file=sys.stderr)
            return 0.0
        raise
    finally:
        conn.close()

if __name__ == "__main__":
    host, peer = sys.argv[1], sys.argv[2]
    rtt = measured_rtt_ms(peer)
    repl = replication_latency_ms(host)
    print(f"RTT={rtt:.2f}ms  repl_latency={repl:.2f}ms  overhead={repl - rtt:.2f}ms")
    sys.exit(1 if repl > RTT_BUDGET_MS else 0)

Feed the same series into your telemetry pipeline so the trend is visible over time, following monitoring Galera cluster state with Python. A latency budget is only useful as a tracked line, because the day a peer moves to a new rack or a route changes is the day the tax quietly doubles.

Edge Cases & Gotchas #

• A low average RTT with a high tail is worse than a slightly higher stable RTT. Galera’s EVS layer reacts to the worst-case delay, so a link with 0.3 ms mean but frequent 30 ms spikes flags peers into wsrep_evs_delayed and trips flow control intermittently, producing latency that looks random. Always measure the p99, not just the mean, and treat wsrep_evs_repl_latency’s max column as a first-class signal.
• wsrep_avg_replication_latency is a running average, so a brief spike is diluted. A short severe stall can leave the average looking acceptable while individual commits time out. Correlate it with wsrep_flow_control_sent and the EVS latency max rather than trusting the mean alone.
• A cross-region synchronous member is almost always the wrong tool. Once RTT reaches double digits, every commit pays that tax and throughput collapses no matter how you tune apply threads. The correct pattern is a synchronous core with an asynchronous replica in the far region, whose trigger metrics and provisioning are covered in when to use async replicas with Galera, with the placement rules in designing multi-master topologies.

Related #

• Understanding Galera Synchronous Replication — the commit path whose latency this page decomposes
• Tuning Flow Control to Prevent Cluster Stalls — reducing the apply-queue component once RTT is ruled out
• Designing Multi-Master Topologies — placing nodes to keep the RTT tax inside budget
• When to Use Async Replicas with Galera — the escape hatch when RTT makes a synchronous member untenable
• Network Security & Firewall Rules for Galera — the 4567 path whose RTT you are measuring