Debugging wsrep_provider_options Parse Errors

wsrep_provider_options is a single semicolon-delimited string handed to the Galera provider as one opaque blob, and a single mistyped key, wrong unit, or malformed PT duration silently disables an entire set of tuning without a hard error. This reference — an extension of the wsrep.cnf Configuration Deep Dive — shows how to read the log signatures that expose a bad option, validate each key against the value the running provider actually loaded, and catch the parse mistakes that leave a node running on defaults it never intended.

Context: Why a Bad Provider Option Fails Quietly #

Unlike top-level [mysqld] directives, which the server validates individually at startup, everything inside wsrep_provider_options is parsed by the provider library (libgalera_smm.so) after the server has already accepted the line. The server sees a syntactically valid quoted string and moves on; the provider then splits it on semicolons, trims each key = value pair, and applies the ones it recognizes. The consequence is a two-tier failure model: a malformed string (an unbalanced quote, a stray newline) can abort provider load loudly, but a malformed pair — a misspelled key, a bad unit suffix, an out-of-range PT period — is frequently ignored with only an INFO-level note, leaving the option at its compiled-in default.

That is why a provider-options typo behaves so differently from a normal config error. You set gcs.fc_limit=256, the node starts cleanly, wsrep_cluster_status reports Primary, and everything looks healthy — but you actually typed gcs.fc_limt=256, the provider never matched the key, and flow control is still throttling at the default of 16. The Galera cluster stalls under write load hours later, and nothing in the config file looks wrong. Because the whole string is replaced (not merged) if it is declared twice — a subtlety detailed in the wsrep.cnf Configuration Deep Dive — a second declaration that drops a key produces the same silent regression.

Solution: Read the Log, Then Diff Loaded Against Intended #

The reliable debugging loop has two halves. First, read what the provider logged when it parsed the string at startup. Second, compare the loaded option set against what you wrote — never trust the file alone, because the file is what lied to you.

Read the parse-time log signatures #

Every provider load prints the full effective option set to the error log at startup. This is the single most useful artifact, because it shows the values as the provider resolved them, defaults included. Filter for it right after a restart:

# The provider dumps its complete resolved option set on load
grep -E "WSREP: Provider options|wsrep_load|gcache\.|gcs\.|evs\." \
  /var/log/mysql/error.log | tail -40

The signatures that matter fall into three buckets. A malformed string aborts the load outright and is loud:

[ERROR] WSREP: Failed to parse provider options: 
[ERROR] WSREP: wsrep::init() failed: 7, must shutdown

A type or range error on a recognized key is logged at ERROR or WARN and usually refuses the value:

[ERROR] WSREP: Invalid value for 'evs.suspect_timeout': '10'
[Note]  WSREP: Ignoring invalid duration, expected ISO8601 period (PTnS)

The dangerous case — an unrecognized key — often surfaces only as a Note, or not at all, because the parser has no schema to reject against. That is precisely why grepping the resolved dump beats grepping for errors: a discarded gcs.fc_limt leaves no ERROR line, but the resolved dump will show gcs.fc_limit = 16, contradicting the 256 you set. Reading these boot-time byte sequences in context is catalogued in Handling Galera Startup Errors & Logs.

Diff the loaded string against the intended one #

The authoritative source of truth is the runtime variable, which the provider exports back as a fully-resolved, comma-or-semicolon-delimited string:

SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options'\G

The output includes every option the provider knows about, defaults filled in — so a key you set that does not appear at your value is a key that was discarded. Automate the comparison so it becomes a gate rather than an eyeball exercise. This probe parses the loaded string, compares it to the set of keys you intended, and flags any that reverted to a default. It uses PyMySQL and handles the wsrep contention codes 1213 (deadlock / certification conflict) and 1205 (lock wait timeout) so a busy node does not read as a config fault:

import sys
import pymysql
from pymysql.err import OperationalError, MySQLError

# What you believe you configured, key -> expected value (as strings).
INTENDED = {
    "gcache.size": "2147483648",     # 2G in bytes, as the provider reports it
    "gcs.fc_limit": "256",
    "evs.suspect_timeout": "PT10S",
}

def loaded_options(host: str, user: str, password: str) -> dict:
    conn = pymysql.connect(host=host, user=user, password=password,
                           connect_timeout=5, read_timeout=5)
    try:
        with conn.cursor() as cur:
            cur.execute("SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options'")
            raw = cur.fetchone()[1]
    except MySQLError as exc:
        if exc.args and exc.args[0] in (1213, 1205):
            print(f"[warn] {host} contended ({exc.args[0]}); retry", file=sys.stderr)
            raise
        raise
    finally:
        conn.close()
    opts = {}
    for pair in raw.split(";"):
        if "=" in pair:
            k, v = pair.split("=", 1)
            opts[k.strip()] = v.strip()
    return opts

def audit(host: str) -> bool:
    try:
        loaded = loaded_options(host, "monitor", "secret")
    except OperationalError as exc:
        print(f"[FATAL] {host} unreachable: {exc}", file=sys.stderr)
        return False
    ok = True
    for key, want in INTENDED.items():
        got = loaded.get(key)
        if got != want:
            print(f"[DRIFT] {host} {key}: loaded={got!r} intended={want!r}")
            ok = False
    if ok:
        print(f"[OK] {host}: all intended provider options loaded")
    return ok

if __name__ == "__main__":
    sys.exit(0 if all(audit(h) for h in sys.argv[1:]) else 1)

A [DRIFT] line where loaded is a round default value (16, PT5S, 134217728) is the fingerprint of a discarded pair — a typo or a bad unit that the provider quietly refused.

Provider-Option Syntax Reference #

Most parse failures come from getting the value grammar wrong, not the key. These are the rules the provider enforces per value type.

The PT time format is the single richest source of silent failures: evs.suspect_timeout, evs.inactive_timeout, evs.keepalive_period, and pc.announce_timeout all take ISO-8601 durations, so PT5S is five seconds and PT1M is one minute, but a bare 5 is either rejected or misread. Byte sizes are the second: only single-letter K, M, G suffixes are honored, so 2GB truncates or fails to parse depending on version. Latency-sensitive tuning of these same EVS and GCS knobs is worked through in Configuring wsrep_provider_options for Low Latency.

Verification #

After any change, prove the provider loaded what you meant with three commands, in order:

# 1. Did provider load abort or warn? (loud failures)
journalctl -u mariadb --since "5 min ago" | grep -iE "WSREP.*(parse|invalid|ignor)"

# 2. What did the provider actually resolve? (silent failures)
mysql -N -e "SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options'" | tr ';' '\n'

# 3. Spot-check one high-stakes key end to end
mysql -N -e "SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options'" \
  | grep -o 'gcs.fc_limit = [0-9]*'

Piping the resolved string through tr ';' '\n' turns the one-line blob into a readable per-option list, which makes a discarded key obvious at a glance. Wire the same check into the validation gate that runs before every restart, described in Validating Galera Config in CI Before Restart, so a parse regression fails the pipeline instead of the Galera cluster.

Edge Cases & Gotchas #

• Whitespace around the equals sign. gcache.size = 2G parses, but a value with a trailing space inside the quotes (gcache.size=2G ; ) can leave the trailing token attached to the next key on some versions. Keep the string tight and let the provider’s own resolved dump be the arbiter.
• A second declaration silently truncates your tuning. Because wsrep_provider_options is one value and the last declaration wins entirely, a drop-in that re-declares it with only gcache.size erases your gcs and evs keys — they revert to defaults with no error. Own the whole string in exactly one layer, as the precedence model in the wsrep.cnf Configuration Deep Dive requires.
• Read-only options rejected at runtime look like a parse error. Some keys (gcache.size, gcache.page_size) are read only at provider load, so trying to change them with SET GLOBAL wsrep_provider_options='...' fails or is ignored even though the syntax is perfect. Change those in the file and restart; only the dynamically settable subset takes a live SET GLOBAL.

Related #

• wsrep.cnf Configuration Deep Dive — load precedence, parameter domains, and why the whole string is replaced not merged
• Validating Galera Config in CI Before Restart — gate a provider-options change in a pipeline before it reaches a node
• Handling Galera Startup Errors & Logs — reading the provider-load and parse byte sequences at boot
• Configuring wsrep_provider_options for Low Latency — correct EVS and GCS values to set once the syntax is proven
• Automating Node Provisioning with Ansible — render the provider string from one source of truth to stop drift