Automating Node Provisioning with Ansible for MariaDB Galera Clusters

This procedure builds on the configuration model in the wsrep.cnf Configuration Deep Dive and answers one focused question: how do you provision a MariaDB Galera node with Ansible so that the same playbook is safe to re-run, forms quorum deterministically, and never accidentally bootstraps a second primary component? Manual provisioning invites configuration drift, State Snapshot Transfer (SST) race conditions, and inconsistent wsrep topology — an operator SSHes into three hosts, edits wsrep.cnf slightly differently on each, and the group either splits or refuses to form. This page shows the idempotent Ansible pattern that eliminates that drift: a deterministic inventory, validated templating, state-aware bootstrap sequencing, and bounded SST recovery.

Why Deterministic Sequencing Matters in a Multi-Master Group #

Galera has no leader election during initial formation. Exactly one node must start with wsrep_cluster_address=gcomm:// (an empty address that creates a new primary component), and every other node must join an existing address such as gcomm://10.0.1.10,10.0.1.11,10.0.1.12. If two nodes bootstrap empty — the classic failure of a naively parallel playbook — you get two independent primary components that both accept writes and can never reconcile, the split-brain condition described in Bootstrapping Your First Galera Cluster.

Ansible’s default free strategy runs hosts in parallel, so the automation itself becomes the race. The fix is not to serialize the whole play — it is to encode the topology as data (which host bootstraps, which join) and let each task gate on that data plus the on-disk cluster state. Done correctly, the playbook is fully idempotent: run it against a healthy three-node group and nothing changes; run it against a cold group and it forms exactly one primary component.

Designing a Deterministic Inventory #

Encode the role of every host explicitly. A static YAML inventory with galera_role, wsrep_node_name, and wsrep_node_address gives each task the data it needs to branch, and keeps a single bootstrap candidate per group lifecycle:

# inventory/galera_nodes.yml
all:
  children:
    galera_cluster:
      hosts:
        galera-node-01:
          ansible_host: 10.0.1.10
          galera_role: bootstrap
          wsrep_node_name: node01
          wsrep_node_address: 10.0.1.10
        galera-node-02:
          ansible_host: 10.0.1.11
          galera_role: joiner
          wsrep_node_name: node02
          wsrep_node_address: 10.0.1.11
        galera-node-03:
          ansible_host: 10.0.1.12
          galera_role: joiner
          wsrep_node_name: node03
          wsrep_node_address: 10.0.1.12

A dynamic inventory generator (Python parsing a CMDB or cloud provider API) can inject galera_role at runtime, but it must guarantee the invariant: no more than one host tagged bootstrap per cluster. Enforce it with an assertion at the top of the play rather than trusting the data source:

- name: Enforce single bootstrap candidate
  ansible.builtin.assert:
    that: >
      (groups['galera_cluster']
        | map('extract', hostvars, 'galera_role')
        | select('equalto', 'bootstrap') | list | length) == 1
    fail_msg: "Exactly one host must have galera_role=bootstrap"
  run_once: true

Pre-Flight Validation and Templating wsrep.cnf #

Structure the work as a role that isolates OS prep, package install, configuration templating, and service orchestration. The galera_node role should expose galera_cluster_name, wsrep_sst_method, and wsrep_provider_options as variables. Before touching any service, validate the environment. Galera needs TCP 3306 (client), 4567 (replication/gcomm), 4568 (IST), and 4444 (SST) reachable between all peers — the reasoning behind those ports is covered in Network Security & Firewall Rules for Galera:

- name: Validate Galera prerequisites
  block:
    - name: Ensure required ports are open
      ansible.builtin.iptables:
        chain: INPUT
        protocol: tcp
        destination_port: "{{ item }}"
        jump: ACCEPT
      loop: [3306, 4444, 4567, 4568]

    - name: Verify SST binaries are present
      ansible.builtin.command: "{{ item }} --version"
      loop: [mariabackup, socat, rsync]
      changed_when: false
      register: sst_binaries

    - name: Fail on mismatched mariabackup major version
      ansible.builtin.fail:
        msg: "mariabackup version incompatible across nodes"
      when: >
        (sst_binaries.results
          | selectattr('item', 'equalto', 'mariabackup')
          | first).stdout is not search('10\.11')

Template wsrep.cnf with the validate directive so a syntax error is caught before the file is ever written into place. validate runs the command against a temp copy (%s), and Ansible only moves the file if it exits zero:

- name: Deploy wsrep.cnf
  ansible.builtin.template:
    src: templates/wsrep.cnf.j2
    dest: /etc/my.cnf.d/wsrep.cnf
    owner: root
    group: root
    mode: '0644'
    validate: 'mariadbd --validate-config --defaults-file=%s'
  notify: Restart MariaDB

The template itself keeps every Galera directive under one [mysqld] section and treats wsrep_provider_options as a single semicolon-delimited string — a second declaration would replace the whole string rather than merge, so all tuning lives in one line. Tune those flow-control keys against the certification behavior described in configuring wsrep_provider_options for low latency:

# templates/wsrep.cnf.j2 -> /etc/my.cnf.d/wsrep.cnf
[mysqld]
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name={{ galera_cluster_name }}
wsrep_cluster_address={{ wsrep_cluster_address }}
wsrep_node_name={{ wsrep_node_name }}
wsrep_node_address={{ wsrep_node_address }}
wsrep_sst_method={{ wsrep_sst_method | default('mariabackup') }}
wsrep_provider_options="gcache.size=2G;gcs.fc_limit=256;gcs.fc_factor=0.8;socket.checksum=1"
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2

Bootstrap Sequencing and the safe_to_bootstrap Flag #

The bootstrap decision is data-driven, not hard-coded. On a node that has run before, /var/lib/mysql/grastate.dat carries safe_to_bootstrap: 1 on the node that shut down last and holds the most recent write-set — that is the only node safe to bootstrap from. The playbook reads that flag rather than assuming the bootstrap host is always correct:

- name: Detect safe bootstrap state
  ansible.builtin.command: "grep -q 'safe_to_bootstrap: 1' /var/lib/mysql/grastate.dat"
  register: grastate_check
  changed_when: false
  failed_when: false

- name: Bootstrap the primary component
  ansible.builtin.command: galera_new_cluster
  when:
    - galera_role == 'bootstrap'
    - grastate_check.rc != 0        # not already the safe node / fresh install
  notify: Wait for cluster sync

- name: Start joiners against the full cluster address
  ansible.builtin.systemd:
    name: mariadb
    state: started
    enabled: true
  vars:
    wsrep_cluster_address: "gcomm://10.0.1.10,10.0.1.11,10.0.1.12"
  when: galera_role == 'joiner'

galera_new_cluster is the safe wrapper — it launches mariadbd with --wsrep-new-cluster for a single start without permanently editing wsrep_cluster_address, so the node reverts to normal join behavior on its next restart. That matters for idempotency: nothing on disk is left in a “bootstrap forever” state. For the state transitions a joiner passes through as it pulls its first SST, see Graceful Node Join and Leave Procedures.

Intercepting SST Failures and Bounded Retries #

A freshly provisioned joiner pulls a full SST, and that is where automation most often stalls. WSREP: 113 (an ENETUNREACH / timeout surfacing during SST) usually traces to an MTU mismatch on an overlay network, donor throttling, or an undersized gcs.fc_limit. Wrap the join in a bounded retry with a fixed delay — retries/until apply a constant delay, not exponential backoff — and only swallow the error you actually expect to be transient:

- name: Join with bounded SST retry
  ansible.builtin.systemd:
    name: mariadb
    state: started
  register: join_result
  retries: 3
  delay: 20
  until: join_result is not failed
  failed_when: >
    join_result is failed and
    'WSREP: 113' not in (join_result.msg | default(''))

For richer diagnostics than a systemd exit code, drive the health check from a Python orchestrator. This snippet targets Python 3.9+, uses PyMySQL, and handles the write-conflict error codes 1213 (deadlock) and 1205 (lock wait timeout) explicitly so a transient certification conflict during warm-up is retried rather than treated as a fatal provisioning failure:

#!/usr/bin/env python3
"""Poll a freshly joined Galera node until it reports Synced."""
import sys
import time
import pymysql
from pymysql.err import OperationalError

def wait_for_synced(host: str, user: str, password: str,
                    attempts: int = 30, interval: int = 10) -> bool:
    for _ in range(attempts):
        try:
            conn = pymysql.connect(host=host, user=user, password=password,
                                   connect_timeout=5)
            with conn.cursor() as cur:
                cur.execute(
                    "SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'")
                state = cur.fetchone()[1]
            conn.close()
            if state == "Synced":
                return True
        except OperationalError as exc:
            code = exc.args[0]
            if code in (1213, 1205):      # deadlock / lock wait — retry
                pass
            elif code == 2003:            # node not accepting connections yet
                pass
            else:
                print(f"[FATAL] unexpected DB error {code}", file=sys.stderr)
                return False
        time.sleep(interval)
    return False

if __name__ == "__main__":
    ok = wait_for_synced(sys.argv[1], "monitor", "monitor_pw")
    sys.exit(0 if ok else 1)

Invoke it from Ansible with ansible.builtin.script and gate the run’s success on its exit code. The same polling logic underpins ongoing observability in Monitoring Galera Cluster State with Python.

Parameter Reference #

Verifying the Provisioning Run #

Never mark a run successful on a systemctl start alone — confirm the group actually reached quorum. Query wsrep_cluster_size and wsrep_cluster_status and fail the play if either is wrong:

- name: Verify quorum before declaring success
  ansible.builtin.command: >
    mysql -N -B -e
    "SELECT VARIABLE_VALUE FROM information_schema.GLOBAL_STATUS
     WHERE VARIABLE_NAME='wsrep_cluster_size';"
  register: cluster_size
  changed_when: false
  failed_when: (cluster_size.stdout | trim | int) < 3
  run_once: true

From the shell, the two commands that confirm a healthy formation are:

mysql -N -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';"      # expect 3
mysql -N -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';"    # expect Primary

Every node must report the same size and Primary; a node reporting non-Primary or size=1 has partitioned and needs the desync recovery in Troubleshooting Node Desync During Join.

Edge Cases and Gotchas #

• Cloud images with a baked-in datadir. Golden AMIs or snapshots that ship a populated /var/lib/mysql carry a stale grastate.dat with a UUID from the build environment. Every joiner then thinks it belongs to a different group and refuses to join. Wipe the datadir (or at least remove grastate.dat and gvwstate.dat) in the image build, or add a provisioning task that clears it when galera_role == 'joiner' and the node has never synced.
• A second empty bootstrap on re-run. If the grastate_check guard is dropped, re-running the playbook after a full outage can bootstrap the wrong node — one that does not hold the latest write-set — silently discarding committed transactions. Always gate galera_new_cluster on both galera_role and the safe_to_bootstrap flag, and never bootstrap from a node you have not verified.
• systemd unit name and drop-in conflicts. On Ubuntu the service is mariadb, but a leftover mysql alias or a MYSQLD_OPTS value in a systemd drop-in can override the templated wsrep_cluster_address at boot with no log warning. Reserve runtime injection for values that genuinely vary per boot, and confirm the effective value with SHOW GLOBAL VARIABLES LIKE 'wsrep_cluster_address' after start. Boot-time parse and provider-load failures are decoded in Handling Galera Startup Errors & Logs.

Related #

• wsrep.cnf Configuration Deep Dive — the parent guide to load precedence, parameter domains, and validation this playbook templates
• Bootstrapping Your First Galera Cluster — the single-bootstrap rule the inventory encodes
• Choosing the Right SST Method for Large Datasets — picking mariabackup vs rsync for the joiner transfer
• Graceful Node Join and Leave Procedures — the state transitions a provisioned joiner passes through
• Galera Cluster Setup & Node Management — the parent framework for node lifecycle and operations