When to Use Async Replicas with Galera: Architectural Triggers and Automation

This decision guide extends the routing model in Fallback Routing & Read-Only Nodes and answers one focused question: at what point should you attach an asynchronous replica to a MariaDB Galera cluster instead of adding another synchronous member? Async replicas are not a default scaling lever — they are a deliberate trade of the synchronous certification guarantee for geographic reach, analytical isolation, and operational decoupling. Get the trigger wrong and you introduce silent replication lag, stale reads, and topology drift; get it right and you extend read capacity without slowing the write path. This page gives you the exact metrics that justify an async node, a safe provisioning routine, and the recovery drill for when a replica falls behind.

Context: Why This Matters in a Multi-Master Cluster

Every write on a Galera member must pass through the write-set certification process on all nodes before it commits, so the group commits only as fast as its slowest synchronous member can certify and apply. Adding a synchronous node therefore adds a certification participant — it does not offload the write path, and if that node is slow or distant it actively throttles the whole group through flow control. This is the property described in Understanding Galera Synchronous Replication: consistency is cluster-wide and synchronous, and latency is shared.

An asynchronous replica sidesteps that contract. It consumes the primary’s binary log outside the write-set path, so it never votes in certification, never participates in flow control, and never blocks a commit. The cost is measurable lag — the replica applies transactions after they have already committed on the source. That single distinction is the whole decision: reach for an async replica precisely when you need read capacity or isolation that must not be paid for in write latency, and you can tolerate reads that are a few seconds stale.

Synchronous certification core versus an out-of-boundary async replica Writes enter a certification boundary holding three fully-meshed Galera nodes that certify every write-set synchronously under shared flow control. One node streams its binary log one-way, out of the boundary, to a read-only async replica that never certifies and carries measurable lag; stale-tolerant reads are served from that replica while writes stay on the synchronous core. Synchronous core · certification boundary every commit certified on all members · shared flow control Writes synchronous write-sets certified round-trip · every member votes Galera node 1 voting member Galera node 2 voting member Galera node 3 binlog source asynchronous binlog stream no flow control · measurable lag leaves the certification boundary Async replica read-only · non-voting Reads (stale-tolerant)

Solution: The Four Triggers That Justify an Async Replica

Provision an async replica only when synchronous replication introduces measurable degradation or violates an operational boundary. In practice that reduces to four concrete triggers.

1. Geographic latency beyond the certification budget. Galera certifies synchronously across every node, so a distant member adds its round-trip time to every commit. When inter-node RTT consistently exceeds roughly 10–15 ms, wsrep_flow_control_paused climbs and the whole group stalls waiting for the far node. An async replica in the remote region absorbs local read traffic there without joining the certification round-trip, keeping the primary region’s commit velocity intact.

2. Analytical and OLAP isolation. Full-table scans, reporting joins, and batch aggregation consume buffer pool and I/O that transactional writes need. Run them on a Galera member and they inflate the apply queue and lengthen certification, raising wsrep_local_cert_failures. An async replica gives those queries a dedicated node whose slowness can never propagate back into the synchronous group.

3. Backup and disaster-recovery decoupling. A State Snapshot Transfer locks a donor and saturates its I/O; the method trade-offs are covered in Choosing the Right SST Method for Large Datasets. Pointing backups at a dedicated async replica lets mariabackup run continuously without ever desyncing a live cluster member or tripping flow control.

4. Version and patch staging. Galera enforces strict protocol compatibility across synchronous members, so you cannot mix binaries in the group during validation. An async replica can run a newer minor version and consume the same binlog stream, giving you a production-shaped canary before a rolling upgrade touches the synchronous cluster.

If none of these apply, add a synchronous member instead — you keep zero-lag reads and stay inside the consistency guarantee.

Confirm the trigger with real metrics, not intuition

Before provisioning anything, prove the synchronous path is actually the bottleneck. Capture the state vector and the network baseline together:

# Certification and flow-control health across the group
mysql -u root -p -e "SHOW GLOBAL STATUS LIKE 'wsrep_%';" \
  | grep -E 'flow_control_paused|local_recv_queue_avg|cert_failures|cluster_size|ready'

# Long-running transactions that stall certification behind them
mysql -u root -p -e "SELECT trx_id, trx_state, trx_started, trx_mysql_thread_id \
  FROM information_schema.innodb_trx WHERE trx_state = 'LOCK WAIT';"

# Inter-node round-trip — sync certification wants this under ~10 ms
ping -c 50 -i 0.2 10.0.1.11 | tail -2

If wsrep_flow_control_paused stays above 0.15 (15% of wall-clock spent paused) after you have tuned gcs.fc_limit and gcs.fc_factor, the bottleneck is network- or I/O-bound, not configuration-bound — that is the signal an async offload is warranted rather than more flow-control tuning.

Provision the replica safely with a pre-flight check

Never attach an async node by hand mid-incident. The routine below validates that the source is healthy, captures exact binary-log coordinates, and only then starts replication. It uses PyMySQL and handles the Galera contention error codes 1213 (deadlock) and 1205 (lock wait timeout) explicitly, since a busy source can surface either during the status probe.

import subprocess
import logging
import pymysql

logging.basicConfig(level=logging.INFO)

def provision_async_replica(source_host: str, creds: dict) -> None:
    """Attach an async replica only if the source is healthy; capture binlog coords first."""
    try:
        conn = pymysql.connect(
            host=source_host, user=creds["user"], password=creds["password"],
            connect_timeout=5, cursorclass=pymysql.cursors.DictCursor,
        )
        with conn.cursor() as cur:
            # 1. Refuse to clone from a source that is itself under flow control.
            cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_flow_control_paused'")
            if float(cur.fetchone()["Value"]) > 0.15:
                raise RuntimeError("Source under flow control; abort provisioning.")

            # 2. MariaDB 10.5+: SHOW BINLOG STATUS replaces the deprecated SHOW MASTER STATUS.
            cur.execute("SHOW BINLOG STATUS")
            binlog = cur.fetchone()
            if not binlog:
                raise RuntimeError("Binary logging is not enabled on the source.")
    except pymysql.err.OperationalError as exc:
        code = exc.args[0]
        if code in (1213, 1205):
            logging.warning("Source contended (error %s); retry provisioning shortly.", code)
        raise
    finally:
        conn.close()

    # 3. MariaDB 10.5+: CHANGE REPLICATION SOURCE TO replaces CHANGE MASTER TO.
    change_cmd = (
        "CHANGE REPLICATION SOURCE TO "
        f"SOURCE_HOST='{source_host}', SOURCE_USER='{creds['repl_user']}', "
        f"SOURCE_PASSWORD='{creds['repl_password']}', "
        f"SOURCE_LOG_FILE='{binlog['File']}', SOURCE_LOG_POS={binlog['Position']};"
    )
    for stmt in (change_cmd, "START REPLICA;"):
        subprocess.run(
            ["mysql", "-u", creds["user"], "-p" + creds["password"], "-e", stmt], check=True,
        )
    logging.info("Async replica attached at %s:%s.", binlog["File"], binlog["Position"])

The key discipline is the pre-flight gate: capturing SOURCE_LOG_FILE/SOURCE_LOG_POS from a node that is not paused guarantees the replica starts from a consistent, quiescent coordinate. For repeatable fleet rollouts, wrap this in the same provisioning flow used for automating node provisioning with Ansible so replica attachment is idempotent rather than a one-off script.

Configure the replica as a pure binlog consumer

An async replica must have the Galera provider switched off entirely — leaving it on forces the node into synchronous certification and triggers immediate cluster eviction. Tune the same node for read-heavy, parallel apply:

[mysqld]
# Disable the Galera provider — this node is NOT a cluster member.
wsrep_on=OFF
wsrep_provider=

# Read-optimised, parallel binlog apply.
read_only=1
super_read_only=1
innodb_read_io_threads=8
innodb_write_io_threads=4
slave_parallel_threads=4
slave_parallel_mode=optimistic

Setting super_read_only=1 alongside read_only=1 closes the SUPER-privilege write bypass, so migration tooling or an admin session can never silently diverge the replica from its source.

Parameter Reference

Parameter Scope Type Default Recommended for async offload
wsrep_on [mysqld] on replica boolean ON OFF — the replica must not certify or join the group
wsrep_provider [mysqld] on replica path none empty — no libgalera_smm.so on an async node
wsrep_flow_control_paused status (source) float 0–1 0.0 < 0.15; sustained higher after tuning = offload trigger
wsrep_local_recv_queue_avg status (source) float 0.0 < 20; a rising queue precedes certification failures
wsrep_local_cert_failures status (source) counter 0 watch the rate; OLAP contention inflates it
gcs.fc_limit wsrep_provider_options integer 16 128 on write-heavy members before concluding you need offload
slave_parallel_threads [mysqld] on replica integer 0 4–8 to keep Seconds_Behind_Source low under bursts
slave_parallel_mode [mysqld] on replica enum conservative optimistic for higher apply throughput on a read replica

The flow-control knobs live in wsrep.cnf; tune them on the synchronous members first, because an offload that was really a mis-tuned gcs.fc_limit just moves the problem.

Verification

Confirm the replica is attached, streaming, and inside its lag budget before you route any reads to it:

SHOW REPLICA STATUS\G
-- Inspect: Replica_IO_Running=Yes, Replica_SQL_Running=Yes, Seconds_Behind_Source < 5

Then verify the node is genuinely outside the Galera cluster — a correct async replica reports no Galera membership at all:

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';   -- expect: empty / 0 (not a member)
SELECT @@global.wsrep_on;                        -- expect: 0

Finally, prove data actually flows: write a sentinel row on the source and read it back from the replica within your lag SLA. Wire the same probe into a monitoring loop by reusing the Python cluster-state monitoring patterns, alerting when Seconds_Behind_Source crosses the threshold at which reads must fail back to the synchronous group.

Edge Cases & Gotchas

  • Never leave wsrep_on=ON on the replica. A node with the provider active tries to certify the binlog stream it is applying, immediately conflicts with the real cluster, and gets evicted. If a replica keeps dropping out of “async” mode, check SELECT @@global.wsrep_on first — it is the single most common misconfiguration.
  • Runaway lag or a duplicate-key error needs a re-clone, not a nudge. When Seconds_Behind_Source grows without bound or Last_SQL_Error shows a duplicate key, stop and reset the channel (STOP REPLICA; RESET REPLICA ALL;), re-seed from a mariabackup snapshot with --copy-back, then re-attach at fresh coordinates using the provisioning routine above. Validate row-level consistency with pt-table-checksum before re-enabling read routing.
  • Docker and systemd images silently re-enable the provider. A base MariaDB image or a distro galera.cnf drop-in can ship wsrep_on=ON, and a systemd drop-in that sets wsrep_provider overrides your [mysqld] block. On containerised or packaged deployments, assert the provider is off at boot rather than trusting the config file precedence — an unexpected drop-in re-enrols the replica the moment it restarts.