MariaDB Galera Core Architecture & Fundamentals

MariaDB Galera Cluster delivers synchronous multi-master replication through the wsrep (Write-Set Replication) API, giving every node an identical, certified copy of the dataset and eliminating the replication lag that makes asynchronous failover lossy. When an application can write to any node and read a consistent result from every other node, the hard operational problems shift: instead of chasing binlog drift you now engineer around certification conflicts, flow control back-pressure, quorum arithmetic, and state-transfer storms. This guide is the architectural home base for those concerns — it explains the core components, the end-to-end write-set lifecycle, the baseline parameters that keep a Galera cluster certified, and the automation and monitoring patterns platform teams need to run production-grade active-active MariaDB on releases 10.6 through 11.x with Galera 4. From here you can descend into every subsystem; return to the wider Galera automation library for adjacent operational domains.

Architecture Overview: Components, Data Flow & Baseline Parameters #

The defining trait of the Galera architecture is that it decouples the database engine from the replication transport. The MariaDB server does not ship binary logs to peers; instead the Galera provider library (libgalera_smm.so) hooks into transaction commit through the wsrep API, captures the row changes as a write-set, and hands that write-set to the Group Communication System (GCS) for totally-ordered delivery to every member. Because ordering and certification are deterministic, each node reaches the same commit-or-rollback verdict independently, which is what makes the group genuinely multi-master rather than a primary with hot standbys. The mechanics of that guarantee are unpacked in Understanding Galera Synchronous Replication.

The moving parts break down into four cooperating layers:

• MariaDB server + InnoDB — parses SQL, runs the transaction, and produces the row-level change set at commit time. Galera requires a transactional engine, so InnoDB (or XtraDB) is mandatory; MyISAM writes are not replicated safely.
• wsrep provider (libgalera_smm.so) — the certification and replication engine loaded via wsrep_provider. It builds write-sets, assigns them a global sequence number (seqno), and applies remote write-sets through parallel apply threads.
• Group Communication System (GCS) — the messaging layer that guarantees total ordering and manages cluster membership, view changes, and flow control over TCP/UDP port 4567.
• State Transfer subsystem — the SST/IST machinery that brings a joining or lagging node back to a consistent state, covered in depth under Initial Data Synchronization Methods.

Figure: the MariaDB server talks to the Galera provider through the wsrep API; the Group Communication System replicates write-sets to peer nodes.

Three parameters are non-negotiable on every node, and drift on any of them silently breaks the certification contract. wsrep_on=ON activates replication; binlog_format=ROW guarantees deterministic row images so that apply is reproducible across nodes; and innodb_autoinc_lock_mode=2 (interleaved) prevents auto-increment gaps from serializing writers and stalling the group. Galera 4 adds streaming replication and improved parallel apply, but parallel apply only helps if wsrep_slave_threads is tuned to the workload — leave it at 1 and a busy donor will build an apply backlog that eventually trips flow control. The baseline every node must share looks like this:

[mysqld]
# --- Mandatory baseline (identical on every node) ---
wsrep_on                     = ON
wsrep_provider               = /usr/lib/galera/libgalera_smm.so
binlog_format                = ROW
default_storage_engine       = InnoDB
innodb_autoinc_lock_mode     = 2
innodb_flush_log_at_trx_commit = 2
wsrep_slave_threads          = 8

For the full annotated parameter tree, deprecations, and load-precedence rules, the canonical reference is the wsrep.cnf Configuration Deep Dive, which treats the configuration file as a declarative artifact you can validate before a node ever starts.

How Synchronous Replication Works End to End #

A Galera commit is best understood as a certification-based variant of virtually-synchronous replication rather than a classic two-phase commit. The originating node executes the transaction locally and optimistically, touching only its own storage engine until the client issues COMMIT. At that point the provider assembles the write-set — the modified rows plus a set of certification keys derived from primary keys, unique keys, and foreign keys — and broadcasts it to the GCS. The GCS delivers that write-set to every node, including the originator, in the same global order. Each node then runs the same deterministic certification test against the gap of transactions that were in flight when the write-set was generated. If no key in the incoming write-set collides with a concurrently-committed transaction, certification passes everywhere; if it collides, it fails everywhere. The Write-Set Certification Process Explained walks through the exact key-set comparison and the sequence-number windows that decide the verdict.

The consequence of this design is that the client acknowledgement is delayed only until the write-set is ordered and certified — not until it is fully applied on remote nodes. Remote apply happens asynchronously behind the total order, which is why Galera is described as “virtually synchronous”: there is no replication lag in the sense of stale reads on a certified transaction, but there is a bounded apply queue that flow control keeps from growing without limit. When a slower node’s receive queue exceeds gcs.fc_limit, it emits a flow-control pause message and the whole cluster throttles new writes until the laggard catches up. This is the single most important behaviour to internalize: in a synchronous cluster your write throughput is governed by your slowest healthy node and your inter-node round-trip time, not by the fastest writer. Concurrent writes to the same rows on different nodes are resolved by the certification order — the transaction with the lower global sequence number wins and the loser receives a deterministic deadlock — a scenario examined in How Galera Handles Concurrent Writes in Multi-Master.

Because the loser of a certification conflict is rolled back after the client already issued its statements, applications must treat commit as a fallible operation. In practice that means catching MariaDB error 1213 (deadlock, surfaced for wsrep certification failures) and 1205 (lock wait timeout) and retrying the whole transaction from the beginning rather than assuming success. Designing transactions to be short, idempotent, and single-node-affine (route related writes to one node where possible) dramatically reduces the certification conflict rate.

Configuration Reference #

Galera’s parameters fall into four operational domains. Getting the domain boundaries right — identity, state transfer, flow control, and network — is what turns a fragile lab cluster into a deterministic one. The tables below list the highest-impact keys per domain; every one of them lives inside the [mysqld] section.

Identity, SST, and network options that live under wsrep_provider_options deserve special care because they are passed as a single semicolon-delimited string and a typo disables the whole option set silently. A representative production block:

[mysqld]
# --- Identity ---
wsrep_cluster_name    = galera_prod_eu
wsrep_cluster_address = gcomm://10.0.1.11,10.0.1.12,10.0.1.13
wsrep_node_name       = galera-node-1
wsrep_node_address    = 10.0.1.11

# --- State transfer (SST) ---
wsrep_sst_method      = mariabackup
wsrep_sst_auth        = sst_user:REPLACE_WITH_VAULT_SECRET

# --- Flow control & apply ---
wsrep_slave_threads   = 8

# --- Provider options: gcache, flow control, EVS, quorum ---
wsrep_provider_options = "gcache.size=2G; gcs.fc_limit=256; gcs.fc_factor=0.8; evs.keepalive_period=PT1S; evs.suspect_timeout=PT10S; pc.weight=1; pc.ignore_sb=false"

The gcache.size setting is the write-set ring buffer that lets a rejoining node catch up with a fast Incremental State Transfer (IST) instead of a full SST — size it to hold at least the volume of writes produced during your longest expected node outage. Latency-sensitive tuning of these provider options, including the trade-offs of evs.send_window and gcs.fc_factor, is the subject of Configuring wsrep_provider_options for Low Latency. Never leave pc.ignore_sb=true in a production three-node cluster — it disables split-brain protection and lets a partitioned minority keep accepting writes.

Automation Patterns #

Galera exposes its entire runtime as pollable status variables, which makes it an ideal target for programmatic health gates. The single most valuable automation is a readiness probe that a load balancer or orchestrator can call before routing traffic: it must confirm the node is a member of the primary component, is Synced, and is not paused by flow control. The following probe targets Python 3.9+ with mysql-connector-python and handles the wsrep-specific error codes explicitly so that a transient certification conflict does not read as an outage.

import sys
import mysql.connector
from mysql.connector import errorcode

FC_PAUSE_ALERT = 0.05  # fraction of time paused by flow control

def read_wsrep(cur, name):
    cur.execute("SHOW GLOBAL STATUS LIKE %s", (name,))
    row = cur.fetchone()
    return row[1] if row else None

def node_is_healthy(host):
    try:
        conn = mysql.connector.connect(
            host=host, user="monitor", password="REDACTED",
            connection_timeout=3, database="information_schema",
        )
    except mysql.connector.Error as err:
        print(f"UNREACHABLE {host}: {err}")
        return False

    try:
        cur = conn.cursor()
        ready   = read_wsrep(cur, "wsrep_ready")
        state   = read_wsrep(cur, "wsrep_local_state_comment")
        status  = read_wsrep(cur, "wsrep_cluster_status")
        paused  = float(read_wsrep(cur, "wsrep_flow_control_paused") or 0)

        healthy = (ready == "ON" and state == "Synced"
                   and status == "Primary" and paused < FC_PAUSE_ALERT)
        if not healthy:
            print(f"UNHEALTHY {host}: ready={ready} state={state} "
                  f"status={status} fc_paused={paused:.3f}")
        return healthy
    except mysql.connector.Error as err:
        # 1213 = deadlock / certification conflict, 1205 = lock wait timeout.
        if err.errno in (errorcode.ER_LOCK_DEADLOCK, errorcode.ER_LOCK_WAIT_TIMEOUT):
            print(f"TRANSIENT {host}: retryable wsrep conflict {err.errno}")
            return True  # transient, not a node-down condition
        print(f"ERROR {host}: {err}")
        return False
    finally:
        conn.close()

if __name__ == "__main__":
    ok = node_is_healthy(sys.argv[1] if len(sys.argv) > 1 else "127.0.0.1")
    sys.exit(0 if ok else 1)

The same status surface underpins a lightweight shell gate for systemd ExecStartPost hooks or CI smoke tests, which is often enough for bootstrap validation:

#!/usr/bin/env bash
set -euo pipefail

query() { mysql -N -B -e "SHOW GLOBAL STATUS LIKE '$1';" | awk '{print $2}'; }

size=$(query wsrep_cluster_size)
status=$(query wsrep_cluster_status)
state=$(query wsrep_local_state_comment)

if [[ "$status" != "Primary" || "$state" != "Synced" || "$size" -lt 2 ]]; then
  echo "Galera not healthy: size=$size status=$status state=$state" >&2
  exit 1
fi
echo "Galera OK: $size-node primary component, node Synced"

Wire these gates into the tools that already own your fleet. Config templating with Ansible or Terraform should render wsrep.cnf from a single source of truth and run a dry-run parse before restarting any service — the end-to-end provisioning workflow is documented in the cross-domain Automated Node Health Monitoring and its Python-first companion Monitoring Galera Cluster State with Python. Continuous validation should assert that wsrep_last_committed advances uniformly across all nodes; a node whose committed seqno stalls while peers advance is applying behind the group and will trip flow control before it affects your SLA.

Infrastructure & Topology Considerations #

Synchronous replication amplifies every physical weakness in the underlying infrastructure, so topology is an architectural decision, not an afterthought. Because each commit waits for total ordering, inter-node round-trip time is added to the latency of every write; an RTT above roughly 10 ms will begin to trigger flow control on write-heavy workloads and visibly degrade throughput. Keep the replication path on a dedicated low-latency, high-bandwidth network segment, and bind wsrep_node_address to that interface rather than a public one. The full port map — 4567 for GCS replication, 4568 for IST, 4444 for SST, and 3306 for clients — plus the ingress rules to lock it down are enumerated in Network Security & Firewall Rules for Galera, and encrypting that traffic is covered by Setting Up Secure TLS for Galera Cluster Communication.

Quorum arithmetic dictates node count. Galera keeps the group writable only while a node can see a majority of the last-known membership weight, so an odd number of members (three is the practical minimum) avoids the deadlocked even split where neither half holds a majority. Spreading three nodes across three availability zones survives a single-AZ loss, but a two-AZ layout is a trap: losing the AZ that holds two of three nodes takes the whole cluster read-only. Where a third full node is uneconomical, deploy a lightweight arbitrator (garbd) in a third zone to hold a tie-breaking vote without storing data. Cross-region stretch clusters are almost always the wrong tool — the RTT tax on every commit is punishing — and the right pattern there is a synchronous core with an asynchronous replica, described in Fallback Routing & Read-Only Nodes and its decision guide When to Use Async Replicas with Galera. Full multi-master node placement, bootstrap sequencing, and quorum weighting are laid out in Designing Multi-Master Topologies.

Capacity planning must budget for Galera’s own memory and disk overhead: the certification index, the flow-control receive queue, and the gcache ring buffer all consume resources beyond the InnoDB buffer pool. NVMe-backed data directories, vm.swappiness=1, disabled transparent huge pages, and generous open-file limits are baseline expectations — the full sizing matrix lives in Galera Cluster Hardware Requirements.

Failure Modes & Remediation #

The failures that page an on-call engineer cluster around a handful of root causes. The table maps the symptom you will actually see in logs or status to its cause and the first corrective action.

Recovery from a full outage always starts by identifying the node with the highest committed seqno — that node holds the most complete state and must be the one to bootstrap the new primary component with galera_new_cluster. Detailed log-parsing and startup-error diagnostics are collected in Handling Galera Startup Errors & Logs, and controlled add/remove sequencing that avoids these failure modes in the first place is covered in Graceful Node Join and Leave Procedures.

Monitoring & Telemetry #

Effective Galera monitoring is a small, well-chosen set of wsrep_ status variables scraped on a tight interval, each with a threshold tied to a specific failure mode. Scrape them with SHOW GLOBAL STATUS LIKE 'wsrep_%' or from information_schema.GLOBAL_STATUS, and export them through a Prometheus exporter (the mysqld-exporter surfaces the wsrep collector) or an OpenTelemetry collector so they land next to the rest of your service metrics.

Alert on state transitions, not just absolute values: a node moving to Donor/Desynced is expected during an SST but alarming if unplanned, and a wsrep_cluster_size that decrements without a corresponding maintenance window is a genuine incident. Pair the numeric alerts with a log stream filtered for WSREP: events so that a certification storm or view change is visible in context. The reusable exporter and alert-rule patterns for this domain are built out in Automated Node Health Monitoring.

Frequently Asked Questions #

Why does my application see deadlock errors under Galera that it never saw on a single MariaDB server? Those are certification conflicts. When two nodes commit transactions that touch the same rows concurrently, the one ordered later loses certification and is rolled back with error 1213. Retry the transaction from the start, keep transactions short, and route related writes to a single node to cut the conflict rate.

How many nodes do I actually need? Three is the practical minimum for a fault-tolerant cluster, because quorum requires a strict majority of the membership weight. Two nodes cannot survive one failure without going read-only. If a third full node is uneconomical, add a garbd arbitrator in a separate failure domain to hold the tie-breaking vote.

What is the difference between SST and IST, and why does it matter? SST is a full state snapshot used when a node has no usable state; IST replays only the missing write-sets from a donor’s gcache. IST is far cheaper, so sizing gcache.size to cover your longest expected outage lets rejoining nodes take the fast IST path instead of a disruptive full SST.

Related #

• Understanding Galera Synchronous Replication — the commit lifecycle and latency model in depth
• Write-Set Certification Process Explained — how concurrent-write conflicts are decided deterministically
• Designing Multi-Master Topologies — node placement, AZ layout, and quorum weighting
• Network Security & Firewall Rules for Galera — the required port map and ingress hardening
• Fallback Routing & Read-Only Nodes — extending the synchronous core with async replicas
• wsrep.cnf Configuration Deep Dive — the full annotated parameter reference and load precedence