Network Security & Firewall Rules for MariaDB Galera Cluster

This procedure builds on the communication model described in MariaDB Galera Core Architecture & Fundamentals, and solves one specific operational problem: how to lock the four Galera network channels down to a least-privilege ruleset without breaking the deterministic, low-jitter packet delivery that synchronous replication depends on. Galera’s multi-master design removes the primary-replica bottleneck, but it replaces it with a hard networking contract — every member must reach every other member on a fixed set of ports, symmetrically, with no stateful asymmetry in the path. A firewall that is too permissive exposes the group communication layer to unauthorized state injection; a firewall that is subtly wrong (one direction blocked, an ephemeral SST port closed, connection tracking exhausted mid-transfer) does not fail at rule-apply time. It fails later as a node eviction, a certification timeout, or a silent partition. This page is the step-by-step guide for database administrators, DevOps engineers, and platform teams who need a production nftables ruleset, the kernel tuning that keeps it stable under State Snapshot Transfer load, and the verification commands that prove the group can still form quorum.

Concept: The Galera Communication Plane and Its Trust Boundary #

A Galera group is a full mesh. Unlike asynchronous replication — where a relay log absorbs latency and a blocked replica simply lags — the Galera synchronous replication model requires that a write-set be delivered to, and certified by, every member before the originating transaction commits. That means the firewall is not guarding a client edge; it is sitting inside the consensus path. Any packet loss the firewall introduces is directly observable as flow-control back-pressure and, past the eviction timeout, as a membership change.

The plane splits into four channels, each with a distinct trust boundary. Client SQL on 3306 faces applications and bastion hosts. The other three — group communication on 4567, Incremental State Transfer on 4568, and State Snapshot Transfer on 4444 — are strictly node-to-node and must never be reachable from outside the defined cluster subnet. The group communication channel is special: 4567 carries both TCP (bulk replication, flow control, and the write-set certification voting traffic) and UDP (lightweight Extended Virtual Synchrony heartbeats and membership discovery). Permit the TCP half and drop the UDP half and the group will still form, then thrash on missed heartbeats.

The rule that governs everything below: firewall state must be symmetric on every node. Galera’s Extended Virtual Synchrony protocol treats a one-directional drop the same as a dead peer. If node A can reach node B on 4567 but return traffic takes a different interface that filters it, the group experiences an asymmetric partition — the hardest failure mode to diagnose because each node’s log blames the other.

Prerequisites & Environment Requirements #

The firewall change touches the consensus path, so confirm the surrounding environment before you apply a single rule:

Software versions

• MariaDB 10.6 LTS or later (11.4 LTS for new builds) with the Galera 4 provider libgalera_smm.so.
• nftables 0.9.3+ with a kernel that ships the nf_tables backend (any current Debian, Ubuntu 20.04+, RHEL 8+). Legacy iptables works but loses atomic rule replacement, which matters on a live node.
• mariabackup installed on every node — the default SST method opens the 4444 listener, and a missing binary makes the port test misleading.

Network ports — these are the channels the ruleset must permit between members and nothing else:

System settings

• A defined cluster subnet in CIDR form (for example 10.0.50.0/24) and a separate management subnet for SSH — never fold administrative access into the node-to-node scope.
• Consistent MTU across every path between nodes; a mismatched MTU with DF set silently drops the large frames SST uses.
• Headroom in net.netfilter.nf_conntrack_max for the connection churn a full SST generates. Sizing that is covered under the parameter deep-dive.
• If nodes span availability zones or a routed fabric, confirm return paths are symmetric — the topology constraints are laid out in Designing Multi-Master Topologies.

Step-by-Step: Building a Least-Privilege Galera Firewall #

The workflow is: name the group as a set, compose a default-drop ruleset that permits only the four channels from that set, apply it atomically so a live node never loses connectivity mid-swap, then tune the kernel for SST bursts.

Step 1 — Model the node group as an address set, not a list of rules #

Hardcoding one accept rule per peer does not scale and drifts the moment a node is added. nftables sets let you name the group once and match every port against it. Define the subnet and the node’s own address as inputs so the same script runs unmodified on every member:

CLUSTER_CIDR="${GALERA_CLUSTER_CIDR:-10.0.50.0/24}"   # every member lives here
MGMT_CIDR="${GALERA_MGMT_CIDR:-10.0.100.0/24}"         # SSH / admin origin only
NODE_IP="${GALERA_NODE_IP:-}"                          # this node's replication address

Keeping the subnet in one variable is what makes the rule least-privilege: the four node-to-node ports are matched against @cluster_nodes and are unreachable from anywhere else. A 0.0.0.0/0 source on 4567 or 4568 would let an outsider inject state into the group and bypass certification — the single most damaging firewall mistake on a Galera deployment.

Step 2 — Compose a default-drop ruleset with validated inputs #

A Galera firewall must default to drop and open only what the group needs. Validate the CIDR before it reaches the kernel so a malformed value fails the deployment instead of producing a wide-open or empty ruleset:

#!/usr/bin/env bash
set -euo pipefail

validate_cidr() {
  local cidr="$1"
  if [[ ! "$cidr" =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}$ ]]; then
    echo "ERROR: Invalid CIDR format: $cidr" >&2
    exit 1
  fi
}

CLUSTER_CIDR="${GALERA_CLUSTER_CIDR:-10.0.50.0/24}"
MGMT_CIDR="${GALERA_MGMT_CIDR:-10.0.100.0/24}"
NODE_IP="${GALERA_NODE_IP:-}"

validate_cidr "$CLUSTER_CIDR"
validate_cidr "$MGMT_CIDR"
if [[ -z "$NODE_IP" ]]; then
  echo "ERROR: GALERA_NODE_IP environment variable is required." >&2
  exit 1
fi

The set -euo pipefail line matters more than it looks: without it, a failed validate_cidr in a pipeline could still leave the script running and apply a partial ruleset. Fail fast, before the kernel netfilter stack is touched.

Step 3 — Apply the ruleset atomically with nft -f #

Replacing a firewall on a running database node is where connectivity is most at risk. nft -f - swaps the entire table in a single transaction — there is no window where the old rules are gone but the new ones are not yet loaded, so an in-flight write-set is never dropped:

cat <<EOF | nft -f -
table inet galera_firewall {
  set cluster_nodes {
    type ipv4_addr
    flags interval
    elements = { $CLUSTER_CIDR }
  }

  chain input {
    type filter hook input priority 0; policy drop;

    # Keep established flows alive across the swap
    ct state established,related accept
    ct state invalid drop

    # Loopback is always trusted
    iif lo accept

    # ICMP for path-MTU discovery (do not blackhole this — SST relies on it)
    ip protocol icmp accept
    ip6 nexthdr icmpv6 accept

    # Galera node-to-node channels: TCP for all four, UDP only for GCS
    ip saddr @cluster_nodes tcp dport { 3306, 4567, 4568, 4444 } accept
    ip saddr @cluster_nodes udp dport 4567 accept

    # Administrative SSH from the management subnet only
    ip saddr $MGMT_CIDR tcp dport 22 accept
  }
}
EOF

echo "nftables ruleset applied atomically."

Two lines are load-bearing. ct state established,related accept placed first means the transactional swap never severs a live GCS connection. The explicit ip protocol icmp accept keeps path-MTU discovery working — a common outage is an operator “hardening” the firewall by dropping all ICMP, which black-holes the ICMP fragmentation needed messages and hangs SST on the first oversized frame. TLS on these channels does not change the port map; when you layer encryption per Setting Up Secure TLS for Galera Cluster Communication, the same 4567/4568/4444 rules apply and certificate failures surface as resets rather than firewall drops.

Step 4 — Size connection tracking for SST churn #

A full SST opens and tears down a burst of connections while streaming an entire dataset. On a busy group the default nf_conntrack_max is too small, and once the table fills the kernel drops new flows — including the very GCS packets that keep the node in the group. Raise the ceiling and persist it:

sysctl -w net.netfilter.nf_conntrack_max=262144
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=1200
echo "net.netfilter.nf_conntrack_max = 262144" > /etc/sysctl.d/60-galera-conntrack.conf
sysctl --system

Parameter Deep-Dive: The Knobs That Decide Stability #

Firewall ports are only half the story; the kernel and provider settings around them determine whether the ruleset survives real traffic. These are the settings worth tuning explicitly:

The interplay that matters: nf_conntrack_tcp_timeout_established must always sit above evs.inactive_timeout. If the kernel prunes a GCS flow before Galera would declare the peer dead, the firewall — not the network — becomes the cause of the eviction, and the log will point everywhere except the conntrack table.

Verification & Health Checks #

Applying rules is not proof the group can communicate. Verify at three layers: the ruleset itself, the wire, and the group’s own view of membership.

Confirm the ruleset loaded and the ports are open on the wire:

# The active table — every port and the cluster set should be present
nft list table inet galera_firewall

# Prove the four channels are actually listening
ss -tulpn | grep -E ':(3306|4567|4568|4444)\b'

# From a peer, confirm reachability in BOTH directions (symmetry check)
nc -zv 10.0.50.11 4567 && nc -zv 10.0.50.11 4568 && nc -zv 10.0.50.11 4444

Then confirm the group itself agrees it is whole. A firewall that quietly drops 4567 UDP will still let TCP connect, so the authoritative test is the group’s membership view:

SHOW GLOBAL STATUS WHERE Variable_name IN
  ('wsrep_cluster_size', 'wsrep_cluster_status',
   'wsrep_local_state_comment', 'wsrep_evs_delayed');

A healthy node reports wsrep_cluster_size equal to the node count, wsrep_cluster_status = Primary, and wsrep_local_state_comment = Synced. A non-empty wsrep_evs_delayed is the earliest firewall-induced symptom — it lists peers the EVS layer is struggling to reach before eviction happens.

For continuous checks, a small probe folds these into a health signal. Using PyMySQL with explicit exception handling so a probe failure is never mistaken for a healthy node:

import sys
import pymysql

def check_membership(host: str, expected: int) -> int:
    try:
        conn = pymysql.connect(host=host, user="monitor",
                               password="secret", connect_timeout=5)
    except pymysql.err.OperationalError as exc:
        # 2003: cannot connect — port blocked or node down
        print(f"[CRIT] {host} unreachable: {exc}", file=sys.stderr)
        return 2
    try:
        with conn.cursor() as cur:
            cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'")
            size = int(cur.fetchone()[1])
            cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_evs_delayed'")
            delayed = cur.fetchone()[1]
    except (pymysql.err.OperationalError, pymysql.err.InternalError) as exc:
        # 1205 lock wait / 1213 deadlock are transient; surface, do not crash
        print(f"[WARN] {host} status query failed: {exc}", file=sys.stderr)
        return 1
    finally:
        conn.close()
    if size < expected or delayed:
        print(f"[WARN] {host} size={size} delayed={delayed!r}")
        return 1
    print(f"[OK] {host} size={size}")
    return 0

if __name__ == "__main__":
    sys.exit(check_membership("10.0.50.10", expected=3))

Automation Integration #

Firewall provisioning belongs in the same pipeline as the rest of the node build so a new member is never live before its rules are. Two patterns cover most deployments.

Infrastructure-as-code validation. When a configuration-management step renders the ruleset, validate the network before it touches the kernel. Python’s standard ipaddress module removes the regex edge cases and fails fast on a malformed or implausibly wide cluster subnet:

import ipaddress
import subprocess
import sys

def validate_cluster_network(cidr: str) -> ipaddress.IPv4Network:
    try:
        network = ipaddress.IPv4Network(cidr, strict=False)
    except ValueError as exc:
        sys.exit(f"Invalid cluster network: {exc}")
    if not (16 <= network.prefixlen <= 28):
        sys.exit("Cluster CIDR prefix must be between /16 and /28")
    return network

def render_and_apply(cidr: str) -> None:
    payload = f"""
    table inet galera_firewall {{
      set cluster_nodes {{
        type ipv4_addr
        flags interval
        elements = {{ {cidr} }}
      }}
      chain input {{
        type filter hook input priority 0; policy drop;
        ct state established,related accept
        iif lo accept
        ip saddr @cluster_nodes tcp dport {{ 3306, 4567, 4568, 4444 }} accept
        ip saddr @cluster_nodes udp dport 4567 accept
      }}
    }}
    """
    subprocess.run(["nft", "-f", "-"], input=payload.encode(), check=True)

if __name__ == "__main__":
    net = validate_cluster_network("10.0.50.0/24")
    render_and_apply(str(net))

See the official Python ipaddress documentation for subnet manipulation beyond this validation. The subprocess.run(..., check=True) call raises on a non-zero nft exit, so a rejected ruleset stops the play instead of leaving the node half-configured.

Ansible ordering. In a role, render the ruleset from inventory (ansible.builtin.template into /etc/nftables.conf), apply it with a handler, and gate it behind the CIDR check above — but sequence the firewall task before MariaDB starts on a fresh node and after it on an existing one, so you never open ports on a group member that has not yet loaded the provider. The same rolling discipline that governs a wsrep.cnf change applies here: change one node, confirm wsrep_cluster_size is unchanged, then proceed.

Troubleshooting #

wsrep_evs_delayed lists a peer that is clearly up. The EVS layer is losing packets to that node in at least one direction. This is almost always an asymmetric rule or a stateful firewall on the return path. Confirm with nc -zv <peer> 4567 from both ends and compare nft list ruleset between the two nodes — a rule present on one and missing on the other produces exactly this signature.

Node connects on TCP but never reaches Synced, log shows repeated evs::proto view changes. The 4567 UDP half is blocked while TCP is open — a classic mistake when a rule permits tcp dport 4567 but omits udp dport 4567. Re-check Step 3; both protocol lines must be present.

SST hangs at a fixed byte count, then the joiner logs Process completed with error: wsrep_sst_mariabackup ... failed. Either 4444 is filtered or ICMP is fully dropped and a path-MTU black hole is stalling large frames. Verify 4444 with nc -zv, then confirm ip protocol icmp accept is present — the SST-stalls-at-N-bytes symptom is the fingerprint of dropped fragmentation needed messages.

Random evictions under write load with nf_conntrack: table full, dropping packet in dmesg. The connection-tracking table overflowed during an SST burst and started dropping GCS packets. Raise net.netfilter.nf_conntrack_max per Step 4 and confirm nf_conntrack_tcp_timeout_established sits above evs.inactive_timeout.

A read-routed application node loses connectivity but the write group is healthy. If you steer reads to specific members, a firewall change that scopes 3306 too tightly can strand the read path; reconcile the client-facing scope against your fallback routing and read-only node design rather than widening the node-to-node ports.

Related #

• MariaDB Galera Core Architecture & Fundamentals — the parent guide to the communication model these rules protect
• Setting Up Secure TLS for Galera Cluster Communication — encrypting the same channels this firewall scopes
• Understanding Galera Synchronous Replication — why packet loss in the path becomes flow-control back-pressure
• Write-Set Certification Process Explained — the voting traffic that rides 4567
• Designing Multi-Master Topologies — node placement and symmetric-path requirements across zones