Building a Prometheus Exporter for wsrep Status

This walkthrough builds on the collector design in Python Monitoring & Alerting Patterns for Galera and answers one precise question: how do you expose a MariaDB Galera node’s wsrep_ status variables as Prometheus gauges from a small, self-contained Python exporter you fully control? Rather than pushing snapshots on a loop, an exporter holds a long-lived HTTP endpoint that Prometheus scrapes on its own schedule, translating SHOW GLOBAL STATUS LIKE 'wsrep_%' into named gauges at request time. This page gives you a runnable prometheus_client exporter, a systemd unit to run it, the scrape config to wire it into Prometheus, and a precise list of which wsrep_ variables are worth exporting and why.

Context: Why a Custom Exporter, Not Just mysqld-exporter

The standard mysqld-exporter already surfaces a wsrep collector, and for many teams that is enough. A purpose-built exporter earns its place when you need control the general-purpose tool does not give you: a curated metric set instead of hundreds of server gauges, cluster-specific derived values (a boolean “is this node in the primary component” gauge, for instance), custom labels drawn from your inventory, or an exporter that runs where a full mysqld-exporter deployment is inconvenient. Because Galera exposes its entire runtime through SHOW GLOBAL STATUS, a focused exporter is a few dozen lines and stays trivially auditable — you can read exactly what it scrapes and what it emits.

The pull model matters for correctness, too. Prometheus owns the scrape interval, so every gauge is timestamped consistently and the rate() and increase() functions work as intended over the counters. That is important for one metric in particular: wsrep_flow_control_paused is an interval average reset on read, whereas wsrep_flow_control_paused_ns is a monotonic nanosecond counter — the exporter should expose the counter so PromQL computes the paused fraction correctly regardless of scrape timing. The alerting rules that consume these gauges are developed in Alerting on Flow Control & Receive-Queue Depth.

Prometheus pull path from a Galera node through the Python exporter A MariaDB Galera node exposes wsrep status on port 3306. A Python exporter using prometheus_client runs a custom collector that, on each scrape, queries SHOW GLOBAL STATUS LIKE wsrep percent and maps the values to named gauges, serving them on an HTTP endpoint at port 9105 slash metrics. The Prometheus server pulls that endpoint on its own interval, storing the samples and evaluating alert rules. Galera node wsrep_% :3306 SHOW STATUS Python exporter prometheus_client custom collector maps values → gauges :9105/metrics Prometheus pull · store · alert scrape on Prometheus' own interval
Prometheus owns the scrape schedule and pulls the exporter's /metrics endpoint; the exporter queries the node only when scraped, so gauges are always fresh at read time.

Solution: A prometheus_client Exporter for wsrep

The cleanest design registers a custom collector whose collect() method runs at scrape time, so the query happens on demand rather than on a background timer that could drift out of sync with Prometheus. The exporter targets Python 3.9+, uses PyMySQL, and handles the Galera write-conflict codes 1213 (deadlock / certification conflict) and 1205 (lock wait timeout) as transient — a contended node must still export a fresh scrape rather than a gap.

#!/usr/bin/env python3
"""wsrep_exporter.py — a focused Prometheus exporter for MariaDB Galera.
Targets Python 3.9+, PyMySQL, and prometheus_client."""
import os
import sys
import time
import pymysql
from prometheus_client import start_http_server
from prometheus_client.core import GaugeMetricFamily, REGISTRY

# Numeric wsrep_ variables exported as gauges (metric name -> help text).
GAUGES = {
    "wsrep_cluster_size":            "Members in the primary component",
    "wsrep_local_recv_queue_avg":    "Mean received write-set apply backlog since last read",
    "wsrep_local_send_queue_avg":    "Mean send-queue depth since last read",
    "wsrep_flow_control_paused_ns":  "Cumulative nanoseconds this node was flow-control paused",
    "wsrep_flow_control_sent":       "Count of flow-control pause events this node sent",
    "wsrep_flow_control_recv":       "Count of flow-control pause events this node received",
    "wsrep_local_cert_failures":     "Cumulative local certification (write) conflicts",
    "wsrep_cert_deps_distance":      "Average apply parallelism available",
    "wsrep_last_committed":          "Highest committed global sequence number (seqno)",
}
# Enum wsrep_ variables exported as 1/0 booleans (metric -> the 'healthy' value).
BOOLS = {
    "wsrep_ready":          "ON",
    "wsrep_cluster_status": "Primary",
}


class WsrepCollector:
    def __init__(self, dsn: dict, node_label: str):
        self.dsn = dsn
        self.node = node_label

    def _scrape(self) -> dict:
        conn = pymysql.connect(connect_timeout=4, read_timeout=4, **self.dsn)
        try:
            with conn.cursor() as cur:
                cur.execute("SHOW GLOBAL STATUS LIKE 'wsrep_%'")
                return {name: val for name, val in cur.fetchall()}
        finally:
            conn.close()

    def collect(self):
        up = 1
        try:
            status = self._scrape()
        except pymysql.err.OperationalError as exc:
            code = exc.args[0] if exc.args else None
            # 1213/1205 mean the node is up but contended; retry once inline.
            if code in (1213, 1205):
                try:
                    status = self._scrape()
                except pymysql.err.MySQLError:
                    status, up = {}, 0
            else:
                status, up = {}, 0
        except pymysql.err.MySQLError:
            status, up = {}, 0

        yield GaugeMetricFamily("galera_up", "1 if the exporter scraped the node",
                                value=up)

        for metric, help_text in GAUGES.items():
            raw = status.get(metric)
            if raw is None:
                continue
            g = GaugeMetricFamily(f"galera_{metric}", help_text, labels=["node"])
            try:
                g.add_metric([self.node], float(raw))
            except (TypeError, ValueError):
                continue
            yield g

        for metric, healthy in BOOLS.items():
            raw = status.get(metric)
            if raw is None:
                continue
            g = GaugeMetricFamily(f"galera_{metric}",
                                  f"1 when {metric} == {healthy}", labels=["node"])
            g.add_metric([self.node], 1.0 if raw == healthy else 0.0)
            yield g


def main() -> None:
    dsn = {
        "host": os.environ["WSREP_HOST"],
        "user": os.environ["WSREP_USER"],
        "password": os.environ["WSREP_PASSWORD"],
        "port": int(os.environ.get("WSREP_PORT", "3306")),
    }
    node = os.environ.get("WSREP_NODE_LABEL", dsn["host"])
    REGISTRY.register(WsrepCollector(dsn, node))
    start_http_server(int(os.environ.get("WSREP_EXPORTER_PORT", "9105")))
    print(f"wsrep exporter serving /metrics on :9105 for node {node}", file=sys.stderr)
    while True:
        time.sleep(3600)


if __name__ == "__main__":
    main()

Two decisions carry the design. Exporting wsrep_flow_control_paused_ns (a monotonic counter) rather than wsrep_flow_control_paused (a reset-on-read average) means Prometheus can compute the true paused fraction with rate(galera_wsrep_flow_control_paused_ns[5m]) / 1e9, which is scrape-interval-independent. Emitting the enum variables wsrep_ready and wsrep_cluster_status as 1/0 gauges lets you alert with plain numeric comparisons instead of string matching, which Prometheus cannot express.

Which Gauges to Expose

Export a deliberately small set. Every metric below maps to a distinct failure mode; the general-purpose collectors ship many more, but this is the diagnostic core for write availability.

Exported gauge Source variable Type What it detects
galera_wsrep_cluster_size wsrep_cluster_size gauge A member dropped from the primary component
galera_wsrep_cluster_status wsrep_cluster_status bool (1=Primary) Node fell into a non-primary partition; rejects writes
galera_wsrep_ready wsrep_ready bool (1=ON) Provider refusing queries during sync or after a fault
galera_wsrep_flow_control_paused_ns wsrep_flow_control_paused_ns counter Cumulative pause time; rate gives the paused fraction
galera_wsrep_flow_control_sent wsrep_flow_control_sent counter This node is the one asking the group to throttle
galera_wsrep_local_recv_queue_avg wsrep_local_recv_queue_avg gauge Apply backlog; a leading indicator of a stall
galera_wsrep_local_cert_failures wsrep_local_cert_failures counter Certification-conflict rate on hot rows
galera_wsrep_cert_deps_distance wsrep_cert_deps_distance gauge Available parallelism; guides wsrep_slave_threads
galera_wsrep_last_committed wsrep_last_committed gauge Compare across nodes to find a stalled applier

The wsrep_cert_deps_distance gauge is what you use to size parallel apply — it estimates how many write-sets can be applied concurrently, feeding the wsrep_slave_threads decision documented in the wsrep.cnf Configuration Deep Dive. Comparing galera_wsrep_last_committed across nodes reveals a member falling behind before flow control even engages.

Run It as a systemd Service

One exporter instance runs per node, scraping the local instance over 127.0.0.1, with credentials injected from an EnvironmentFile (mode 0400) rather than baked into the unit:

[Unit]
Description=MariaDB Galera wsrep Prometheus exporter
After=mariadb.service network-online.target

[Service]
Type=simple
User=wsrep-exporter
EnvironmentFile=/etc/wsrep-exporter/env
ExecStart=/opt/wsrep-exporter/.venv/bin/python /opt/wsrep-exporter/wsrep_exporter.py
Restart=on-failure
RestartSec=10
# Least privilege: no write access, private tmp.
ProtectSystem=strict
PrivateTmp=true

[Install]
WantedBy=multi-user.target

The EnvironmentFile supplies WSREP_HOST=127.0.0.1, WSREP_USER, WSREP_PASSWORD, and a WSREP_NODE_LABEL that matches your inventory name so the node label is stable across restarts. The monitoring account needs only USAGE, exactly as provisioned in the monitoring patterns guide.

Wire It Into Prometheus

Point Prometheus at every node’s exporter. Relabel so the target host and your node label are both preserved:

scrape_configs:
  - job_name: galera_wsrep
    scrape_interval: 15s
    static_configs:
      - targets:
          - 10.0.2.11:9105
          - 10.0.2.12:9105
          - 10.0.2.13:9105
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

Verification

Confirm the exporter serves parseable metrics before Prometheus ever scrapes it:

# The endpoint must return text with your galera_ gauges.
curl -s http://127.0.0.1:9105/metrics | grep '^galera_'

# Validate the exposition format is well-formed for Prometheus.
curl -s http://127.0.0.1:9105/metrics | promtool check metrics

Cross-check a gauge against the source of truth so a wrong value is attributable to the mapping, not the server:

SHOW GLOBAL STATUS WHERE Variable_name IN
  ('wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_flow_control_paused_ns');

Then confirm Prometheus is actually storing samples: query galera_up in the Prometheus expression browser and expect 1 for every node target, and galera_wsrep_cluster_status should read 1 on every healthy member.

Edge Cases & Gotchas

  • Never export the reset-on-read average as a counter. wsrep_flow_control_paused is reset each time status is read, so exporting it as a gauge means its value depends on how recently something else read status — and applying rate() to it is meaningless. Export wsrep_flow_control_paused_ns instead; it is monotonic and rate-safe. Mixing the two is the most common wsrep-exporter mistake.
  • A node mid-SST refuses connections; export galera_up 0, not a stale scrape. During a State Snapshot Transfer a joiner often rejects new connections until it reaches Synced. The exporter above yields galera_up 0 and omits the other gauges, so a Joiner reads as “scrape failed” rather than silently reporting the last-known good values. Alert on galera_up == 0 separately from cluster-health alerts, and correlate with the transfer flow in Initial Data Synchronization Methods.
  • wsrep_cluster_status is a string Prometheus cannot compare. Prometheus stores only float samples, so exporting the raw Primary/non-Primary string is impossible — that is why the exporter maps it to a 1/0 boolean gauge. Alert on galera_wsrep_cluster_status == 0, never on a label value.