Bootstrapping Your First Galera Cluster: Production-Grade Initialization & Multi-Master Sync

This procedure builds on the node lifecycle model described in Galera Cluster Setup & Node Management, and solves a single, high-stakes operational problem: bringing a brand-new MariaDB Galera deployment safely from zero nodes to a healthy, writable multi-master group without introducing a split-brain. Unlike an asynchronous primary-replica topology, a synchronous group has no “default primary” — exactly one node must be told to form the initial Primary Component, and every other node must join rather than bootstrap. Get the sequence wrong and you either end up with two independent single-node clusters that silently diverge, or a group that refuses to become writable at all. This guide treats bootstrap as a deterministic, validated, and repeatable workflow suitable for database administrators, DevOps engineers, and platform teams automating cluster provisioning.

The Primary Component and the Bootstrap State Machine #

A Galera cluster is only writable when a quorum of nodes agrees on a shared view of membership called the Primary Component (PC). On a first-time deployment there is no existing PC to join, so the group cannot form by consensus — one node must be granted the authority to declare itself the founding member of a component of size 1. That single act is what “bootstrapping” means. Once the seed node holds the Primary Component, additional nodes contact it over the group communication channel, negotiate a state transfer, and are admitted to the component one at a time until membership reaches its target size.

The distinction that trips up most first deployments is bootstrap versus join. Bootstrapping sets wsrep_cluster_address to the empty group gcomm:// for one node only, telling Galera “start a new group here.” Every other node keeps the full member list (gcomm://ip1,ip2,ip3) so it knows where to find the existing group. The membership handshake and how concurrent writes are ordered once the group is live are governed by the write-set certification process, which is the mechanism that lets every node accept writes while still guaranteeing a single global transaction order.

The broader theory of how synchronous replication keeps all nodes byte-for-byte identical is covered in Understanding Galera Synchronous Replication; this page focuses on the concrete initialization mechanics.

Prerequisites & Environment Requirements #

Bootstrap is unforgiving of a misconfigured environment because the seed node commits to being the source of truth for the entire group. Validate every item below on all candidate nodes before you run a single initialization command.

Software versions

• MariaDB 10.6 LTS or later (11.4 LTS recommended for new builds); the Galera wsrep provider (galera-4, libgalera_smm.so) ships with the server package.
• A backup binary for state transfer — mariabackup for MariaDB 10.6+ (bundled with the server), used by the mariabackup SST method.
• Consistent versions across all nodes. A minor-version skew between donor and joiner can cause SST to fail during the prepare/apply phase.

Network ports — bidirectional reachability is mandatory on:

Latency should stay under ~1 ms intra-availability-zone and under ~5 ms cross-AZ; beyond that, flow control begins throttling writers. Opening these ports safely (and locking them to known peers) is detailed in Network Security & Firewall Rules for Galera.

System settings

• innodb_flush_log_at_trx_commit=1 and sync_binlog=1 for crash safety on the seed node — the founding member must survive a restart without losing committed history.
• Swap disabled or vm.swappiness=0; the gcache and certification index live in RAM, and an OOM kill mid-bootstrap corrupts grastate.dat.
• SELinux/AppArmor policies that allow mysqld to bind 4567, 4568, and 4444. Sizing guidance for CPU, RAM, and disk lives in Galera Cluster Hardware Requirements.

Run this validation pass across all nodes before deploying configuration:

#!/usr/bin/env bash
set -euo pipefail

REQUIRED_PORTS=(3306 4567 4568 4444)
NODES=("10.0.1.10" "10.0.1.11" "10.0.1.12")

echo "=== Pre-Bootstrap Infrastructure Validation ==="

# Verify swap is disabled — an OOM kill mid-bootstrap corrupts grastate.dat
if [[ $(swapon --show=NAME --noheadings | wc -l) -gt 0 ]]; then
  echo "[FAIL] Active swap detected. Disable: swapoff -a && sed -i '/swap/d' /etc/fstab"
  exit 1
fi

# Kernel memory pressure threshold
SWAPPINESS=$(sysctl -n vm.swappiness)
if [[ "$SWAPPINESS" -ne 0 ]]; then
  echo "[WARN] vm.swappiness=${SWAPPINESS}. Set to 0 to prevent gcache eviction."
fi

# Port reachability matrix
for node in "${NODES[@]}"; do
  for port in "${REQUIRED_PORTS[@]}"; do
    if ! timeout 2 bash -c "echo > /dev/tcp/${node}/${port}" 2>/dev/null; then
      echo "[FAIL] ${node}:${port} unreachable. Verify firewall/iptables rules."
    fi
  done
done

echo "[PASS] Infrastructure validation complete. Proceed to configuration."

Step-by-Step Bootstrap Procedure #

The bootstrap process is strictly sequential: form the Primary Component on exactly one node, confirm it, then add the remaining nodes one at a time so each state transfer completes cleanly before the next begins.

Step 1 — Deploy identical configuration to every node #

Before anything starts, each node needs a wsrep-enabled config. The key insight is that the same file ships everywhere — only wsrep_node_address and wsrep_node_name differ per host. wsrep_cluster_address lists the full membership on every node; you never bake gcomm:// (empty) into a config file, because that would make a node bootstrap a fresh group every time it restarts.

[mysqld]
# Core Galera provider
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name=prod_galera_cluster
wsrep_cluster_address=gcomm://10.0.1.10,10.0.1.11,10.0.1.12
wsrep_node_address=10.0.1.10
wsrep_node_name=node-01

# State transfer & cache
wsrep_sst_method=mariabackup
wsrep_sst_auth="sst_user:secure_password"
wsrep_provider_options="gcache.size=8G; gcache.page_size=1G; gcs.fc_limit=256"

# ACID compliance & crash safety
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=1
sync_binlog=1

binlog_format=ROW and innodb_autoinc_lock_mode=2 are hard requirements — Galera refuses to replicate statement-based binlog events, and interleaved auto-increment lock mode is what lets multiple masters generate non-colliding IDs. Full parameter semantics, loading precedence, and the syntax of wsrep_provider_options (wsrep.cnf Configuration Deep Dive) are documented separately. Provision wsrep_sst_auth credentials from a secrets manager and inject them at runtime rather than committing plaintext to the config.

Step 2 — Initialize the Primary Component on the seed node #

On the single designated seed node, run the bootstrap wrapper. Modern MariaDB packages ship galera_new_cluster, which starts mariadbd with --wsrep-new-cluster — the safe, idempotent way to inject the empty gcomm:// group for one boot without editing the config file on disk.

sudo galera_new_cluster

Never run galera_new_cluster on more than one node, and never run it again once the group exists — a second bootstrap creates a rival Primary Component and is the classic split-brain trigger.

Step 3 — Confirm the seed formed a writable component #

Before adding any joiner, verify the seed is genuinely Primary and writable. A component that reports non-Primary will silently reject writes.

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';    -- expect 1
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';  -- expect Primary
SHOW GLOBAL STATUS LIKE 'wsrep_ready';           -- expect ON
SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'; -- expect Synced

Step 4 — Create the SST account used by joiners #

The joining nodes authenticate to the donor using wsrep_sst_auth. Create that account on the seed now, while it is the only node, so the grant replicates to every joiner as part of their state transfer.

CREATE USER 'sst_user'@'localhost' IDENTIFIED BY 'secure_password';
GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT,
      BINLOG MONITOR ON *.* TO 'sst_user'@'localhost';
FLUSH PRIVILEGES;

Step 5 — Join the remaining nodes one at a time #

On each additional node, start MariaDB the normal way. Because its config already lists the full membership, it discovers the live group, requests a state transfer, and is admitted to the Primary Component. Start them sequentially — waiting for each to reach Synced — so you never run two simultaneous SSTs against the same donor.

sudo systemctl start mariadb

A joiner with an empty or divergent datadir receives a full State Snapshot Transfer (SST); one whose last position is still held in the donor’s gcache receives a faster Incremental State Transfer (IST). Choosing and tuning the transfer mechanism for large datasets is covered in Choosing the Right SST Method for Large Datasets. OS-specific package paths, systemd unit overrides, and service hardening for a common target platform live in How to Bootstrap MariaDB Galera on Ubuntu 22.04.

Step 6 — Verify multi-master write propagation #

Once every node reports wsrep_cluster_size = N, prove the group is genuinely multi-master by writing on one node and reading the row back on another.

-- On node 1
CREATE TABLE IF NOT EXISTS test.sync_check (id INT PRIMARY KEY AUTO_INCREMENT, ts DATETIME);
INSERT INTO test.sync_check (ts) VALUES (NOW());

-- On node 2 and node 3
SELECT * FROM test.sync_check;

The row must appear on all nodes effectively instantly. If it does not, the nodes are not in the same Primary Component — stop and investigate before putting traffic on the group.

Parameter Deep-Dive #

These are the knobs that most directly determine whether a first bootstrap succeeds and whether joiners come up cleanly.

gcache.size is the single most consequential tuning decision for painless node additions: it is the buffer that decides IST-versus-SST. Detailed flow-control and provider-option tuning is documented in the wsrep.cnf Configuration Deep Dive.

Verification & Health Checks #

Beyond the ad-hoc SHOW STATUS checks above, standardize a health probe so both humans and automation read cluster state the same way. The variables that matter most immediately after bootstrap:

• wsrep_cluster_status — must be Primary on every node. Any non-Primary means that node has lost quorum.
• wsrep_cluster_size — must equal your intended node count on all nodes. Two different sizes on two nodes is split-brain.
• wsrep_local_state_comment — must read Synced. Donor/Desynced, Joiner, or Initialized mean the node is mid-transfer or unhealthy.
• wsrep_ready — ON means the node accepts queries; OFF means it will reject them.
• wsrep_flow_control_paused — should sit near 0.0; sustained values above 0.05 indicate a node throttling the writers.

A Python readiness probe suitable for a provisioning gate:

import sys
import pymysql
from pymysql.err import OperationalError, MySQLError


def check_galera_health(host: str, user: str, password: str, expected_size: int = 3) -> bool:
    """Return True only when the node is a Synced member of a Primary Component."""
    try:
        conn = pymysql.connect(host=host, user=user, password=password,
                               database="mysql", connect_timeout=5)
    except OperationalError as exc:
        print(f"[ERROR] cannot reach {host}: {exc}", file=sys.stderr)
        return False

    checks = {
        "wsrep_ready": "ON",
        "wsrep_cluster_status": "Primary",
        "wsrep_local_state_comment": "Synced",
    }
    try:
        with conn.cursor() as cur:
            for var, want in checks.items():
                cur.execute("SHOW GLOBAL STATUS LIKE %s", (var,))
                row = cur.fetchone()
                if not row or row[1] != want:
                    print(f"[WARN] {host}: {var}={row[1] if row else 'n/a'} (want {want})")
                    return False
            cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'")
            size = int(cur.fetchone()[1])
    except MySQLError as exc:
        print(f"[ERROR] status query failed on {host}: {exc}", file=sys.stderr)
        return False
    finally:
        conn.close()

    if size < expected_size:
        print(f"[WARN] {host}: cluster_size={size} (want >= {expected_size})")
        return False
    return True


if __name__ == "__main__":
    if check_galera_health("10.0.1.10", "monitor", "secure_pass", expected_size=3):
        print("[OK] Primary Component stable — safe to route traffic.")
    else:
        sys.exit(1)

Ongoing state polling and alert thresholds are expanded in Automated Node Health Monitoring and its Python-focused companion, Monitoring Galera Cluster State with Python.

Automation Integration #

Manual bootstrap is fine for a lab; production wants the bootstrap-exactly-once invariant enforced by tooling. The pattern that scales is: converge configuration on all nodes, elect one seed, run galera_new_cluster only if no Primary Component exists yet, then start joiners in series.

An Ansible sketch that keeps bootstrap idempotent:

- name: Render identical wsrep config to every node
  ansible.builtin.template:
    src: wsrep.cnf.j2
    dest: /etc/mysql/mariadb.conf.d/60-galera.cnf
    mode: "0640"
  notify: none            # never auto-restart mid-play; sequencing is manual

- name: Detect an already-running Primary Component
  ansible.builtin.command: >
    mysql -N -B -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status'"
  register: pc_status
  changed_when: false
  failed_when: false

- name: Bootstrap the seed node only when no cluster exists
  ansible.builtin.command: galera_new_cluster
  when:
    - inventory_hostname == groups['galera'][0]
    - "'Primary' not in pc_status.stdout"

- name: Start joiners one at a time
  ansible.builtin.service:
    name: mariadb
    state: started
  throttle: 1             # serialize joins so SSTs never overlap
  when: inventory_hostname != groups['galera'][0]

When a bootstrap-time smoke test writes into the group (for example, seeding a schema), wrap the write with retry handling for the two transient wsrep conflict codes — 1213 (deadlock / certification conflict) and 1205 (lock wait timeout) — because a healthy multi-master group can legitimately return them under concurrency:

import time
import pymysql
from pymysql.err import OperationalError

RETRYABLE = {1205, 1213}  # lock wait timeout, deadlock/cert conflict


def write_with_retry(conn, sql, params=None, attempts=5):
    for attempt in range(1, attempts + 1):
        try:
            with conn.cursor() as cur:
                cur.execute(sql, params or ())
            conn.commit()
            return
        except OperationalError as exc:
            code = exc.args[0]
            if code in RETRYABLE and attempt < attempts:
                time.sleep(0.1 * attempt)  # linear backoff, then retry
                continue
            raise

The same converge-then-sequence approach maps cleanly onto Terraform provisioners or a CI/CD job that gates promotion on the readiness probe above.

Troubleshooting #

WSREP: failed to open gcomm backend connection: 110: failed to reach primary view The seed cannot bind or reach the group communication port. Confirm 4567/tcp+udp is open and not already held by a stale mysqld, and that wsrep_node_address is the node’s real, reachable IP rather than 127.0.0.1. On a first bootstrap this usually means the firewall was never opened — cross-check against Network Security & Firewall Rules for Galera.

Node refuses to start: It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster Galera found safe_to_bootstrap: 0 in grastate.dat. This is a guard against bootstrapping from a stale node. If — and only if — you have confirmed this node holds the most advanced data, set safe_to_bootstrap: 1 in /var/lib/mysql/grastate.dat and re-run galera_new_cluster. Never flip this flag blindly; it exists to prevent booting from a losing copy.

wsrep_cluster_status = non-Primary after adding nodes The group lost quorum — often two nodes see each other but not the third, or a network partition split the members. Query wsrep_cluster_size on each node; if two nodes report 1 independently, you have a split-brain from a second accidental bootstrap. Stop the rogue node, wipe its datadir if it diverged, and rejoin it as a plain start (never a second galera_new_cluster).

SST fails immediately: Process completed with error: wsrep_sst_mariabackup ... Access denied for user 'sst_user' The wsrep_sst_auth account was never created on the donor, or its grants are insufficient. Create it on the seed (Step 4) so it replicates, and verify RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT, BINLOG MONITOR are granted.

Joiner loops on full SST every restart The node is leaving uncleanly (seqno: -1 in grastate.dat), usually from SIGKILL/kill -9 instead of systemctl stop mariadb. A clean stop writes the last sequence number so the next start can use fast IST; a hard kill forces a full snapshot. Controlled shutdown and rejoin sequencing is detailed in Graceful Node Join and Leave Procedures. For decoding the raw startup output behind any of these symptoms, see Handling Galera Startup Errors & Logs.

Frequently Asked Questions #

Which node should I bootstrap first? On a brand-new deployment with no data anywhere, any node can be the seed — pick a deterministic one (e.g. the first host in your inventory). After an outage where nodes hold data, bootstrap from the node with the highest committed sequence number (seqno in grastate.dat); bootstrapping from a stale node discards the newer writes on the others.

What is the difference between bootstrapping and starting a node? Bootstrapping (galera_new_cluster) creates a brand-new Primary Component of size 1 by using the empty gcomm:// group. Starting a node normally (systemctl start mariadb) makes it join an existing component using the full member list. You bootstrap exactly one node exactly once per cluster lifetime; everything else is a normal start.

Why did my cluster split into two independent nodes? Almost always because galera_new_cluster was run on more than one node, or the empty gcomm:// address was left in a config file so a restart re-bootstrapped. Each bootstrap forms a separate Primary Component that accepts writes independently and diverges. Recover by choosing the authoritative node, then wiping and rejoining the others as plain starts.

Related #

• Galera Cluster Setup & Node Management — the parent guide covering the full node lifecycle
• How to Bootstrap MariaDB Galera on Ubuntu 22.04 — distribution-specific package paths and systemd hardening
• Graceful Node Join and Leave Procedures — controlled SST/IST sequencing and clean shutdown
• Choosing the Right SST Method for Large Datasets — mariabackup vs rsync transfer tuning
• wsrep.cnf Configuration Deep Dive — full parameter matrix and provider options
• Handling Galera Startup Errors & Logs — decoding wsrep log lines during bootstrap