Handling Galera Startup Errors & Logs

This procedure builds on the node lifecycle model described in Galera Cluster Setup & Node Management, and solves one specific operational problem: turning an opaque, failed MariaDB Galera start into a diagnosis you can act on within seconds. A synchronous multi-master group is unforgiving at startup — a single misaligned wsrep parameter, a stale grastate.dat state file, or a firewall rule that silently drops the state-transfer port can escalate from one node refusing to start into a group that will not form a writable Primary Component at all. This page treats the MariaDB error log as structured telemetry rather than a passive text stream, maps every common startup failure to an exact log signature, and gives database administrators, DevOps engineers, and platform teams a repeatable path from symptom to remediation.

The Galera Startup State Machine and Where It Breaks #

Galera does not run a separate cluster daemon. Every synchronization event, membership change, and fatal error is routed through the ordinary MariaDB server error log by the wsrep provider (libgalera_smm.so). A healthy start walks a strict, observable progression: the provider loads and reads wsrep_provider_options, the node opens the group communication channel and attempts to join an existing Primary Component, a donor is negotiated for state transfer, the transfer (IST or SST) runs, and the node finally reports wsrep_local_state_comment: Synced with wsrep_ready: ON. A startup failure is simply the point at which that progression stalls or aborts, and each stall point emits a distinct log line.

Understanding the two entry paths is what makes the log readable. A node that forms a new group takes the bootstrap path (the empty gcomm:// group), described in Bootstrapping Your First Galera Cluster; a node that joins an existing group takes the joiner path and requires a healthy donor. The join handshake and the state-transfer choices behind it are covered in Initial Data Synchronization Methods. Knowing which path a node should be on tells you immediately whether a “cannot find peer” line is expected (bootstrap) or fatal (join).

Prerequisites & Environment Requirements #

Before you can diagnose a start reliably, the observability surface itself must be in place. Confirm each of the following on every node:

• MariaDB 10.6 LTS or later (11.4 LTS for new builds) with the bundled galera-4 provider. Version skew between a donor and a joiner is itself a startup failure cause, so record the exact mariadb --version on every host.
• A readable, persistent error log. On systemd deployments the output is captured by journalctl -u mariadb and, when log_error is set, also written to /var/log/mariadb/mariadb.log (RHEL/Alma/Rocky) or /var/log/mysql/error.log (Debian/Ubuntu). If neither exists, fix logging before anything else — you are debugging blind.
• The Galera ports reachable between all nodes: 3306 (SQL), 4567 (group communication, TCP+UDP), 4568 (IST), and 4444 (SST). Opening and locking these to known peers is detailed in Network Security & Firewall Rules for Galera; a dropped 4444 is the single most common cause of a “stuck joiner”.
• Time synchronized via chronyd/ntpd. Skewed clocks produce misleading log timestamps and can break TLS-secured group communication.
• A local, non-NFS data directory. InnoDB file locks over NFS produce startup errors that look like corruption. Sizing and storage guidance lives in Galera Cluster Hardware Requirements.

Reading the Log: A Step-by-Step Diagnostic Procedure #

Work these steps in order. Each one narrows the failure class before you touch a config file, so you never “fix” the wrong layer.

1. Capture the full startup window, not the tail #

The provider prints its most important lines before it exits, so a naive tail often misses them. Pull the whole last start:

# Everything MariaDB logged since the most recent start attempt
journalctl -u mariadb --since "10 min ago" --no-pager

# Or, from the file, the last 500 lines cover a normal startup window
tail -n 500 /var/log/mariadb/mariadb.log

You are looking for the last successful state before the abort — the progression stops at exactly one transition, and that transition names the failure class.

2. Anchor on three signatures, not ad-hoc grep #

Ad-hoc grep in production produces false positives (many wsrep lines are informational). Target three deterministic anchors instead:

# 1) Provider / protocol failures
grep -E 'wsrep.*\[(ERROR|FATAL)\]' /var/log/mariadb/mariadb.log
# 2) State-transfer breakdowns
grep -E 'WSREP_SST: \[ERROR\]|SST failed|xtrabackup: error' /var/log/mariadb/mariadb.log
# 3) OS-level resource contention
grep -E "Bind on TCP/IP port|Address already in use|Unable to lock ./ibdata1" /var/log/mariadb/mariadb.log

3. Read the sequence number before acting #

If the failure involves group formation, the committed sequence number (seqno) in grastate.dat decides which node is authoritative. Read it on every node before you bootstrap or wipe anything:

grep -E 'seqno|safe_to_bootstrap' /var/lib/mysql/grastate.dat

The node with the highest non-negative seqno holds the newest committed writes. Bootstrapping from a lower seqno silently discards the newer history on the others — the single most damaging mistake in this whole workflow.

4. Classify the failure and jump to the matching fix #

With the anchor line and the seqno in hand, the failure falls into one of four classes below. Each has an exact signature and a distinct resolution path.

Configuration mismatch #

An invalid wsrep_cluster_address or mismatched wsrep_provider_options (documented in the wsrep.cnf Configuration Deep Dive) stalls the provider at wsrep: 0 (Initializing) and eventually logs failed to open gcomm backend connection. Confirm every node shares an identical wsrep_cluster_name and that wsrep_node_address resolves to a routable, non-loopback interface.

# Compare identity/address keys across the running config
grep -E 'wsrep_cluster_name|wsrep_cluster_address|wsrep_node_address' \
     /etc/my.cnf.d/*.cnf
# Confirm the Galera ports are actually listening once started
ss -tlnp | grep -E ':(3306|4567|4568|4444)\b'

Also confirm wsrep_sst_method names a method MariaDB actually ships (mariabackup, rsync, or mysqldump); a typo here aborts the join before any transfer begins.

State-file corruption and the bootstrap flag #

Galera writes safe_to_bootstrap: 1 to grastate.dat only on the last node to shut down cleanly, precisely so an unclean node cannot re-seed a divergent group. Forcing a bootstrap on a node holding safe_to_bootstrap: 0 logs WSREP: It may not be safe to bootstrap the cluster and refuses. The correct move is to identify the authoritative node (step 3), then:

# On the node with the highest seqno only — never on more than one
sudo sed -i 's/^safe_to_bootstrap: 0/safe_to_bootstrap: 1/' \
     /var/lib/mysql/grastate.dat
sudo galera_new_cluster           # supported wrapper; sets --wsrep-new-cluster

Note that systemctl start mariadb does not accept a --wsrep-new-cluster flag — galera_new_cluster is the only supported way to seed. Detailed bootstrap sequencing lives in Bootstrapping Your First Galera Cluster.

SST/IST failure and credential errors #

State Snapshot Transfer breaks when the donor cannot stream data — usually invalid wsrep_sst_auth credentials, insufficient disk on donor or joiner, or a firewall blocking 4444. The joiner log emits WSREP_SST: [ERROR] Error while getting data from donor node. Recreate the SST account identically on every node:

-- Run on every node; the account must exist cluster-wide
CREATE USER IF NOT EXISTS 'sst_user'@'localhost' IDENTIFIED BY 'secure_password';
GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT, BINLOG MONITOR
      ON *.* TO 'sst_user'@'localhost';
FLUSH PRIVILEGES;

Then confirm wsrep_sst_auth = "sst_user:secure_password" matches byte-for-byte across every node’s config, verify donor free space, and confirm firewalld/iptables allows inbound 4444/tcp. When SST keeps failing on a large dataset, the transfer method itself may be the bottleneck — see Choosing the Right SST Method for Large Datasets.

Port binding and resource contention #

Galera needs exclusive access to 3306, 4567, 4568, and 4444. A conflicting process aborts the start with Can't start server: Bind on TCP/IP port, while an InnoDB: Unable to lock ./ibdata1 line points to a zombie mysqld or an NFS-mounted data directory.

# Clear stale bindings from a crashed instance
sudo fuser -k 3306/tcp 4567/tcp 4568/tcp 4444/tcp
# Confirm the data directory is local, not NFS
df -hT /var/lib/mysql | grep -v nfs

If OOM kills strike during state transfer, size the gcache and flow control in wsrep_provider_options (for example gcache.size=2G; gcs.fc_limit=256) so a large IST does not exhaust RAM.

Parameter Deep-Dive: The Knobs That Decide a Start #

These are the parameters most often responsible for a failed or hung start. Tune them before you automate recovery, not after.

Full context for each of these — including their interaction with certification and flow control — is in the wsrep.cnf Configuration Deep Dive.

Verification & Health Checks #

A start “succeeds” only when the node is actually part of the Primary Component and writable, not merely when mysqld is running. Confirm all four indicators:

# Process/unit is active and did not exit non-zero
systemctl status mariadb --no-pager

-- The four variables that together mean "healthy member"
SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_ready',                 -- ON  = node accepts queries
  'wsrep_cluster_status',        -- Primary = part of the quorum
  'wsrep_local_state_comment',   -- Synced = caught up
  'wsrep_cluster_size'           -- matches the expected node count
);

The concept behind the Primary/Synced states is explained in Understanding Galera Synchronous Replication. For a scriptable probe, query the same variables with PyMySQL and handle the two write-conflict error codes Galera can surface during a still-settling start:

#!/usr/bin/env python3
"""Post-start Galera health probe (Python 3.9+, PyMySQL)."""
import sys
import pymysql

REQUIRED = {
    "wsrep_ready": "ON",
    "wsrep_cluster_status": "Primary",
    "wsrep_local_state_comment": "Synced",
}

def probe(host: str = "127.0.0.1") -> int:
    try:
        conn = pymysql.connect(host=host, user="monitor",
                               password="monitor_pw", connect_timeout=5)
    except pymysql.err.OperationalError as exc:
        print(f"CRITICAL: cannot connect to {host}: {exc}")
        return 2
    try:
        with conn.cursor() as cur:
            cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_%'")
            status = {row[0]: row[1] for row in cur.fetchall()}
    except pymysql.err.InternalError as exc:
        code = exc.args[0]
        if code in (1213, 1205):  # deadlock / lock-wait during settling
            print(f"WARN: transient wsrep contention (code {code}); retry")
            return 1
        raise
    finally:
        conn.close()

    for var, want in REQUIRED.items():
        got = status.get(var)
        if got != want:
            print(f"CRITICAL: {var}={got!r} (expected {want!r})")
            return 2
    print(f"OK: Primary/Synced, size={status.get('wsrep_cluster_size')}")
    return 0

if __name__ == "__main__":
    sys.exit(probe(*sys.argv[1:]))

Automation Integration #

Manual triage does not scale across a fleet, so fold the diagnostic into the deployment pipeline. The following validator is designed to run as a systemd ExecStartPost hook or a post-restart CI step: it tails the startup window, classifies the failure, prints an actionable next step, and only ever reads grastate.dat — it never auto-bootstraps, because seeding the wrong node is unrecoverable.

#!/usr/bin/env python3
"""Galera startup validator — classify a failed start (Python 3.9+)."""
import re
import sys
from datetime import datetime
from pathlib import Path

LOG_PATH = Path("/var/log/mariadb/mariadb.log")

PATTERNS = {
    "config_mismatch": re.compile(r"failed to open gcomm backend connection"),
    "sst_failure": re.compile(r"WSREP_SST: \[ERROR\]"),
    "port_conflict": re.compile(r"Can't start server: Bind on TCP/IP port"),
    "bootstrap_required": re.compile(r"not safe to bootstrap the cluster"),
}

ACTIONS = {
    "config_mismatch": "Verify wsrep_cluster_address and wsrep_node_address parity.",
    "sst_failure": "Validate wsrep_sst_auth and donor disk/port 4444.",
    "port_conflict": "Clear stale bindings with fuser/ss, then restart.",
    "bootstrap_required": "Confirm highest seqno, then galera_new_cluster on ONE node.",
}

def classify() -> int:
    if not LOG_PATH.exists():
        print("CRITICAL: error log missing; check log_error setting.")
        return 2
    window = LOG_PATH.read_text(errors="replace").splitlines()[-500:]
    hits = {name for line in window for name, rx in PATTERNS.items()
            if rx.search(line)}
    if not hits:
        print("OK: no startup failure signatures detected.")
        return 0
    stamp = datetime.now().isoformat(timespec="seconds")
    for name in hits:
        print(f"[{stamp}] {name}: {ACTIONS[name]}")
    return 2

if __name__ == "__main__":
    sys.exit(classify())

Wire it in idempotently with Ansible so every node reports the same way — this pairs naturally with the fleet-wide checks in Automated Node Health Monitoring:

- name: Install Galera startup validator
  ansible.builtin.copy:
    src: galera_startup_validator.py
    dest: /usr/local/bin/galera_startup_validator.py
    mode: "0755"

- name: Run validator after every mariadb start
  ansible.builtin.copy:
    dest: /etc/systemd/system/mariadb.service.d/validate.conf
    content: |
      [Service]
      ExecStartPost=/usr/bin/python3 /usr/local/bin/galera_startup_validator.py
  notify: reload systemd

Because the validator exits non-zero on a detected failure, a CI gate or an alerting rule can fail the deploy loudly instead of leaving a half-joined node silently out of the group.

Troubleshooting #

Specific log lines and their exact remediation:

• WSREP: failed to open gcomm backend connection: 110: failed to reach primary view — the node cannot reach any peer. On a joiner this is fatal: check wsrep_cluster_address, DNS resolution of wsrep_node_address, and that 4567/tcp+udp is open. On the first node of a cold cluster this is expected until you bootstrap.
• WSREP: It may not be safe to bootstrap the cluster from this node — safe_to_bootstrap: 0 in grastate.dat. Do not force-edit blindly. Read seqno on every node, bootstrap only the highest, then start the rest as plain joins. Controlled sequencing is in Graceful Node Join and Leave Procedures.
• WSREP_SST: [ERROR] Error while getting data from donor node — SST aborted. Confirm wsrep_sst_auth parity, donor free space, and inbound 4444/tcp; check the donor’s log for the reason it refused (often a missing SST-user grant).
• Can't start server: Bind on TCP/IP port: Address already in use — a previous mysqld or a foreign process holds the port. Run fuser -k on the four Galera ports and confirm no second unit template is enabled.
• InnoDB: Unable to lock ./ibdata1, error: 11 — a zombie mysqld still holds the data files, or the datadir is on NFS. Kill the stale PID (ps -ef | grep mysqld) and verify df -hT /var/lib/mysql is a local filesystem.

Related #

• Galera Cluster Setup & Node Management — the parent guide covering the full node lifecycle
• Bootstrapping Your First Galera Cluster — forming the initial Primary Component without split-brain
• wsrep.cnf Configuration Deep Dive — the full parameter matrix behind every startup line
• Initial Data Synchronization Methods — how SST/IST joins actually run
• Graceful Node Join and Leave Procedures — clean shutdown so safe_to_bootstrap stays correct
• Automated Node Health Monitoring — continuous wsrep_ scraping to catch degradation before the next restart