Validating Galera Config in CI Before Restart

A rolling restart is the point where a bad wsrep.cnf stops being a text file and becomes an outage, so the safest place to catch a broken option is a CI gate that parses the config, dry-runs it against the MariaDB binary, and asserts that the required keys are byte-identical across every node — all before a single service is bounced. This guide extends the wsrep.cnf Configuration Deep Dive with a runnable pipeline gate that fails the build instead of the Galera cluster.

Context: Why Validation Must Precede the Restart #

In a synchronous multi-master group the restart is unforgiving because several parameters are read only once, at provider load, and must agree across members. A node that boots with a subtly different wsrep_cluster_name, a gcomm:// prefix stripped by a templating bug, or a wsrep_provider_options string that a peer parses differently does not warn you — it either forms its own standalone group or trips a certification mismatch that only surfaces under write load. Once you have started a rolling restart, each node that comes back wrong reduces the healthy quorum, and by the time the symptom appears the window to abort cheaply has closed.

The defense is to move every check that can run without a live restart into CI, where a failure costs a red pipeline rather than a degraded group. Three classes of mistake are all catchable statically: syntax (a value the daemon cannot parse), completeness (a required identity key missing from a rendered file), and cross-node divergence (a key that must be identical drifting between hosts). Catching all three before the first systemctl restart is the difference between a boring deploy and a 2 a.m. page.

Solution: A Three-Stage Gate #

Stage 1 — Dry-run parse against the real binary #

The most authoritative syntax check is the daemon itself. mariadbd --validate-config parses the entire merged tree — every drop-in, in precedence order — without binding ports or starting replication, and returns non-zero if any directive is unparseable. Run it in the pipeline against the same merged path the node will use:

# Parse the full merged config; non-zero exit == the restart would fail
mariadbd --validate-config --defaults-file=/etc/mysql/mariadb.cnf
echo "validate-config exit: $?"

Because provider options are resolved by the library, not the server, you also want to surface the effective wsrep_ values the daemon would apply, which the same binary can print without starting:

mariadbd --wsrep-provider=/usr/lib/galera/libgalera_smm.so \
  --verbose --help 2>/dev/null | grep -A2 "wsrep-provider-options"

A container image pinned to the exact production MariaDB version is the ideal runner for this stage, so the parser that validates in CI is byte-for-byte the parser that will run on the node.

Stage 2 — Assert required keys and forbidden mistakes #

--validate-config proves the file parses; it does not prove the file is complete or sane. A wsrep.cnf with no wsrep_cluster_address parses perfectly and then boots a lonely standalone node. This structural gate parses the rendered file, asserts the mandatory identity keys are present, and rejects the specific mistakes that pass syntax but break clustering — a missing gcomm:// prefix and a provider-options string declared more than once:

import configparser
import subprocess
import sys

REQUIRED = {
    "wsrep_on", "wsrep_provider", "wsrep_cluster_name",
    "wsrep_cluster_address", "wsrep_node_name", "wsrep_node_address",
    "wsrep_sst_method",
}

def structural_checks(path: str) -> list[str]:
    errors: list[str] = []
    parser = configparser.ConfigParser(strict=False)
    # strict=False tolerates duplicate keys; we detect them ourselves below.
    parser.read(path)

    if not parser.has_section("mysqld"):
        return ["missing [mysqld] section"]

    present = set(parser.options("mysqld"))
    for missing in sorted(REQUIRED - present):
        errors.append(f"missing required key: {missing}")

    addr = parser.get("mysqld", "wsrep_cluster_address", fallback="")
    if addr and not addr.strip().strip('"').startswith("gcomm://"):
        errors.append("wsrep_cluster_address is missing the gcomm:// prefix")

    # A provider_options string declared twice silently replaces the first.
    with open(path, encoding="utf-8") as fh:
        decls = [ln for ln in fh if ln.strip().startswith("wsrep_provider_options")]
    if len(decls) > 1:
        errors.append(f"wsrep_provider_options declared {len(decls)} times "
                      "(a later declaration replaces the whole string)")
    return errors

def dry_run(defaults_file: str) -> list[str]:
    result = subprocess.run(
        ["mariadbd", "--validate-config", f"--defaults-file={defaults_file}"],
        capture_output=True, text=True,
    )
    return [] if result.returncode == 0 else [f"validate-config failed:\n{result.stderr.strip()}"]

if __name__ == "__main__":
    cnf = sys.argv[1] if len(sys.argv) > 1 else "/etc/mysql/wsrep.cnf"
    problems = structural_checks(cnf) + dry_run("/etc/mysql/mariadb.cnf")
    for p in problems:
        print(f"[FAIL] {p}", file=sys.stderr)
    if problems:
        sys.exit(1)
    print("[OK] structural and syntax checks passed")

Stage 3 — Prove the must-match keys are identical across nodes #

Some directives are per-node by design (wsrep_node_name, wsrep_node_address, ist.recv_addr), while others must be byte-identical or the group fractures: wsrep_cluster_name, wsrep_cluster_address, and the tuning inside wsrep_provider_options that governs GCache and flow control. This gate collects the rendered file from every node and asserts the must-match set agrees:

import configparser
import sys

MUST_MATCH = ("wsrep_cluster_name", "wsrep_cluster_address")

def read_key(path: str, key: str) -> str | None:
    parser = configparser.ConfigParser(strict=False)
    parser.read(path)
    return parser.get("mysqld", key, fallback=None)

def compare(node_files: dict[str, str]) -> bool:
    ok = True
    for key in MUST_MATCH:
        values = {node: read_key(path, key) for node, path in node_files.items()}
        distinct = set(values.values())
        if len(distinct) > 1:
            ok = False
            print(f"[DIVERGENCE] {key} differs across nodes:")
            for node, val in values.items():
                print(f"    {node}: {val!r}")
    if ok:
        print("[OK] all must-match keys are identical across nodes")
    return ok

if __name__ == "__main__":
    # e.g. node1=/tmp/render/node1.cnf node2=/tmp/render/node2.cnf ...
    files = dict(pair.split("=", 1) for pair in sys.argv[1:])
    sys.exit(0 if compare(files) else 1)

Parameter Reference: Which Keys Must Match Versus Vary #

The provider-options row is the subtle one: ist.recv_addr legitimately differs per node, but gcache.size and gcs.fc_limit must not, so the comparison should split the string and diff only the shared keys. When a divergence is flagged, the debugging path for a silently-dropped option is Debugging wsrep_provider_options Parse Errors.

Verification: Wire It Into the Pipeline #

A minimal GitLab CI job chains all three stages and blocks the deploy stage on their success:

validate-galera-config:
  stage: test
  image: mariadb:11.4          # pinned to the production version
  script:
    - python3 ci/structural_checks.py rendered/node1.cnf
    - python3 ci/cross_node_diff.py
        node1=rendered/node1.cnf
        node2=rendered/node2.cnf
        node3=rendered/node3.cnf
  rules:
    - changes:
        - "**/wsrep.cnf"
        - "templates/galera-*.cnf.j2"

rolling-restart:
  stage: deploy
  needs: ["validate-galera-config"]   # never runs unless validation passed
  script:
    - ansible-playbook -i inventory rolling_restart.yml
  when: manual

The needs dependency is what makes this a gate: the rolling-restart job is unreachable unless validate-galera-config was green. On the node side, the same serialized, wait-for-Synced restart is described in the wsrep.cnf Configuration Deep Dive, and the idempotent rendering that feeds these files is covered in Automating Node Provisioning with Ansible.

Edge Cases & Gotchas #

• Validate against the production major version, not latest. Provider option names and defaults shift between MariaDB releases, so a config that validates on 11.4 can be rejected on a node still running 10.6. Pin the CI image to the exact version the fleet runs, and bump both together.
• --validate-config does not catch semantic nonsense. A gcache.size=2GB (invalid unit) may parse as a string and slip through the daemon check while the provider later discards it. Pair the syntax gate with the loaded-value diff from Debugging wsrep_provider_options Parse Errors so unit and duration mistakes are caught too.
• A rendered file is not the merged file. CI usually validates the drop-in you rendered, but the node merges it with distribution defaults where the last value wins. Run --validate-config against the full --defaults-file chain, not just the isolated wsrep.cnf, or a stray base-layer override will still surprise you at boot.

Related #

• wsrep.cnf Configuration Deep Dive — the load-precedence model and the rolling-restart discipline this gate protects
• Debugging wsrep_provider_options Parse Errors — catch the unit and duration mistakes that pass a syntax check
• Automating Node Provisioning with Ansible — render the per-node files the cross-node gate compares
• Handling Galera Startup Errors & Logs — decode the boot failures a gate is meant to prevent
• Bootstrapping Your First Galera Cluster — the single-bootstrap rule a divergent cluster address can violate