Configuring a Galera Arbitrator with garbd: Quorum Without Data

A Galera arbitrator (garbd) is a lightweight daemon that joins the group communication layer as a full voting member but stores no data — it exists to break ties, so two full data nodes in two availability zones can survive the loss of either zone without going read-only. This guide, part of Automated Failover & Quorum Recovery in Galera, shows exactly how to deploy garbd as a systemd service in a third failure domain, wire it into the Galera cluster on port 4567, weight its vote correctly, and verify it is counting toward quorum — plus the honest trade-off of when a garbd tie-breaker is the right call versus when you should just run a third full node.

Context: Why a Dataless Vote Solves the Two-Node Trap #

Galera keeps the group writable only while the connected members hold a strict majority of the last-agreed weight. A two-node cluster has a total weight of 2, so a majority requires strictly more than 1 — meaning both nodes. Lose either one and the survivor is a minority of one, transitions to non-Primary, and refuses writes. That is the two-node trap: a Galera cluster that cannot tolerate a single failure, which is the exact opposite of why you clustered. The quorum arithmetic behind it is developed in the failover and quorum-recovery guide.

The classic fix is a third full node, giving a total weight of 3 where any two nodes form a majority. But a third full node means a third full copy of the data, a third machine sized for the workload, and a third SST donor to manage. garbd gives you the vote without the data: it participates in the group communication system and certification protocol, receives every write-set so it can vote on the primary component, but discards them instead of applying — it never runs InnoDB, never serves a query, and never acts as an SST donor. Placed in a third availability zone, it turns a symmetric two-AZ split into a decidable one: whichever data node can still see the arbitrator holds two votes of three and stays writable.

Solution: Deploy garbd as a Managed Service #

Step 1 — Install the arbitrator package #

garbd ships separately from the server. On the third-zone host (which needs no MariaDB server at all):

# Debian / Ubuntu
apt-get install -y galera-arbitrator-4
# RHEL / Rocky / Alma
dnf install -y galera-4-garbd

The host only needs outbound and inbound reachability to both data nodes on port 4567; it needs no data volume, no InnoDB tuning, and a fraction of the memory of a full node.

Step 2 — Write the arbitrator configuration #

garbd reads a small INI-style file. The group must match the data nodes’ wsrep_cluster_name exactly, and address is the gcomm:// seed list of the real nodes:

# /etc/default/garb  (Debian) or /etc/sysconfig/garb (RHEL)
# The cluster this arbitrator votes in — MUST equal wsrep_cluster_name.
GALERA_GROUP="prod-galera-primary"

# gcomm:// seed list of the DATA nodes (not the arbitrator itself).
GALERA_NODES="10.0.1.11:4567 10.0.2.12:4567"

# Provider options: give the arbitrator a normal weight and log to a file.
GALERA_OPTIONS="pc.weight=1; gmcast.listen_addr=tcp://0.0.0.0:4567"

# Send garbd's own log somewhere durable.
LOG_FILE="/var/log/garbd.log"

Keep pc.weight=1 for a symmetric two-data-node design so the total weight is 3. Only raise the arbitrator’s weight (for example to 2) if you deliberately want the zone-C vote to dominate — useful when one data zone is your designated survivor, but a choice that makes the arbitrator a single point of quorum failure, so weight it up only with intent.

Step 3 — Run it under systemd #

The packages ship a garb unit that sources the file above. Enable and start it, then confirm it stays up:

systemctl enable --now garb
systemctl status garb --no-pager

If you manage units directly, the effective service is simply the garbd binary invoked with the group and address, run as an unprivileged user and restarted on failure:

# /etc/systemd/system/garb.service (illustrative — the package unit is preferred)
[Unit]
Description=Galera Arbitrator (garbd)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=/etc/default/garb
ExecStart=/usr/bin/garbd --group ${GALERA_GROUP} --address gcomm://10.0.1.11:4567,10.0.2.12:4567 --options ${GALERA_OPTIONS} --log ${LOG_FILE}
Restart=on-failure
RestartSec=5
User=nobody

[Install]
WantedBy=multi-user.target

Step 4 — Open the firewall between all three zones #

garbd speaks the same group-communication protocol as the data nodes, so port 4567 (TCP, and UDP for the multicast path if you use it) must be open bidirectionally between the arbitrator and each data node. It never needs 4568 (IST) or 4444 (SST) because it never transfers state. The full port map and hardening rules are in Network Security & Firewall Rules for Galera.

Parameter Reference #

Verification #

After starting garbd, confirm the data nodes actually count it. On either data node, the Galera cluster size should include the arbitrator:

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';   -- 2 data nodes + garbd → 3
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status'; -- Primary

wsrep_cluster_size counting 3 while you run only two data nodes is the proof the arbitrator joined and is voting. Inspect the members to see the arbitrator explicitly:

SELECT NODE_NAME, NODE_INCOMING_ADDRESS
FROM information_schema.wsrep_membership;   -- one row shows the garbd node

Then confirm the daemon is healthy on the arbitrator host and logging view changes:

systemctl is-active garb
grep -E 'view\(|PRIM|NON_PRIM' /var/log/garbd.log | tail

The real test is a failure drill: stop MariaDB on one data node and confirm the other stays Primary with wsrep_cluster_size=2 (surviving data node + arbitrator) and continues accepting writes. Wire that assertion into the same health checks described in Automated Node Health Monitoring.

Edge Cases & Gotchas #

• The arbitrator must not share a failure domain with a data node. Running garbd in the same AZ, rack, or hypervisor as data node 1 means a single-zone outage takes down both a data node and the tie-breaker at once — you are back to a survivor that is a minority of one. The whole value is topological: it only breaks ties if it fails independently, which is why it belongs in a distinct third zone. Placement rules live in Designing Multi-Master Topologies.
• garbd cannot be an SST donor or joiner. Because it holds no data, a joining node can never SST from the arbitrator, and the arbitrator never needs state itself. If your monitoring flags the garbd “node” as never reaching Synced, that is expected — it has no data state to sync; judge it by whether it appears in wsrep_cluster_size, not by a state comment.
• A weighted arbitrator is a single point of quorum failure. Setting the arbitrator’s pc.weight above the data nodes’ concentrates quorum in the one process that stores nothing and is easiest to forget to monitor. If the weighted arbitrator dies, both data nodes can lose quorum simultaneously. Prefer equal weights unless you have a specific designated-survivor requirement, and if you do weight it, monitor the arbitrator as critically as a data node.

Related #

• Automated Failover & Quorum Recovery in Galera — the quorum arithmetic and failover procedure the arbitrator feeds into
• Recovering from a Non-Primary Component — what to do when a vote still leaves a node stuck non-Primary
• Designing Multi-Master Topologies — where to place data nodes and an arbitrator across zones
• Network Security & Firewall Rules for Galera — the port-4567 ingress rules the arbitrator depends on