Recovering from a Non-Primary Component

When wsrep_cluster_status reads non-Primary, a MariaDB Galera node has lost quorum and is refusing every write — and the recovery you choose in the next few minutes either restores service cleanly or splits your data forever. This guide, part of Automated Failover & Quorum Recovery in Galera, answers one precise question: exactly when and how do you promote a stuck non-Primary component back to Primary with pc.bootstrap, and when must you refuse to. It covers the symptoms you will actually see, the survivor-selection rule that prevents data loss, the runtime SET GLOBAL wsrep_provider_options='pc.bootstrap=YES' command, and the specific danger of reaching for pc.ignore_sb instead.

Context: Why a non-Primary Node Refuses Writes

A Galera group stays writable only while the members that can see each other hold a majority of the last-agreed weight — the surviving majority is the primary component, and any partition that finds itself in the minority transitions to non-Primary to make split-brain arithmetically impossible. That quorum arithmetic is developed in full in Understanding Galera Synchronous Replication; the operational consequence is that a non-Primary node fails every write with WSREP has not yet prepared node for application use and serves reads only if you explicitly allow them.

There are two distinct ways to arrive at non-Primary, and they demand opposite responses. In the first, a majority partition still exists somewhere — you are simply looking at the losing minority, which will rejoin on its own once connectivity heals. In the second, no partition holds a majority (a symmetric split, or a total outage where every survivor is a minority of the original weight), so no primary component exists at all and one must be reconstructed by hand. Bootstrapping in the first case is a catastrophe; bootstrapping the correct node in the second is the fix. The entire recovery hinges on telling them apart before you type anything.

Solution: Diagnose, Select the Survivor, then Bootstrap

Step 1 — Confirm you are actually non-Primary, and that nobody else is Primary

Check the status on every node you can still reach:

SHOW GLOBAL STATUS WHERE Variable_name IN
  ('wsrep_cluster_status', 'wsrep_cluster_size',
   'wsrep_local_state_comment', 'wsrep_last_committed', 'wsrep_cluster_conf_id');

If any reachable node reports wsrep_cluster_status = Primary, stop — a writable component already exists, and your non-Primary node will rejoin it automatically once the network heals. Forcing a bootstrap now creates a second primary component and diverges the dataset. Only proceed when every reachable node reports non-Primary.

Step 2 — Select the survivor with the most complete state

The node you promote must be the one holding the highest committed sequence number, because its history is a superset of the others’ and promoting a laggard silently discards the transactions it never saw. Read each survivor’s true position from the running server or, if stopped, from the recovery path:

-- On each running non-Primary node
SHOW GLOBAL STATUS LIKE 'wsrep_last_committed';
# On a stopped node, derive the authoritative seqno (replays the InnoDB redo log)
mariadbd --wsrep-recover 2>&1 | grep 'Recovered position'
# → WSREP: Recovered position: 6f2a4b...:184230

Compare the sequence numbers across all survivors. The largest one is the node you bootstrap; every other node will discard its divergent tail and resync from it. Selecting the highest-seqno survivor is the same rule the full failover procedure applies after a total loss.

Step 3 — Promote the survivor with pc.bootstrap

Once you have confirmed no Primary exists and identified the highest-seqno node, promote that node only back to Primary at runtime — no restart required:

-- On the chosen highest-seqno survivor ONLY
SET GLOBAL wsrep_provider_options = 'pc.bootstrap=YES';

This tells the Galera provider to declare its current component the primary component. The node immediately becomes writable, its wsrep_cluster_status flips to Primary, and any other survivors that can see it re-evaluate quorum and rejoin — pulling the delta by Incremental State Transfer if gcache still holds it, or a full State Snapshot Transfer otherwise. Verify before routing traffic:

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';   -- → Primary
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';     -- climbs as peers rejoin

pc.bootstrap is the runtime counterpart to galera_new_cluster: use pc.bootstrap when the node is already running and stuck non-Primary, and galera_new_cluster when you are cold-starting a stopped survivor into a fresh component. Never issue pc.bootstrap on more than one node.

Parameter & Command Reference

Item Type Default Use it to Danger
pc.bootstrap=YES dynamic provider option — Promote a running non-Primary component to Primary in place Promoting two components at once forks the data
galera_new_cluster CLI wrapper (--wsrep-new-cluster) — Cold-start a stopped survivor as a new primary component Bootstrapping a non-highest-seqno node loses committed writes
wsrep_last_committed status (integer) — Rank survivors to pick the one to promote A non-Primary node still reports it; use it to compare, not to write
wsrep_cluster_conf_id status (integer) — Detect two components: differing values = split view Equal values do not by themselves prove quorum
pc.recovery boolean true Auto-restore the primary component after a full power loss via gvwstate.dat If false, a total outage always needs a manual bootstrap
pc.ignore_sb boolean false (Two-node designs only) keep a minority writable Standing true on 3+ nodes permits real split-brain
pc.ignore_quorum boolean false Emergency: ignore quorum to force writability Leaves no split-brain guard whatsoever; never a standing value

Verification

After the bootstrap, prove the group genuinely reconverged into a single view rather than two:

-- Run on every node; all three checks must agree cluster-wide
SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_cluster_status',      -- 'Primary' everywhere
  'wsrep_cluster_size',        -- equals the expected member count
  'wsrep_cluster_conf_id',     -- IDENTICAL on all nodes = one shared view
  'wsrep_local_state_comment', -- 'Synced' everywhere
  'wsrep_evs_state'            -- 'OPERATIONAL'
);

The critical check is wsrep_cluster_conf_id: if two nodes report Primary but with different conf-ids, you have created exactly the split you were trying to avoid. A quick end-to-end proof is to write a sentinel row on the bootstrapped node and confirm it appears on every rejoined peer within a second. For a scripted gate that reads this surface and handles the 1213/1205 contention codes, reuse the controller in the failover and quorum-recovery guide or the Python probes in Monitoring Galera Cluster State with Python.

Edge Cases & Gotchas

  • pc.ignore_sb=true is not a recovery tool — it is a split-brain switch. It is tempting when a two-node cluster wedges non-Primary after losing one node, because it makes the survivor writable instantly. But on any cluster of three or more it lets a partitioned minority keep accepting writes at the same time as the majority, and the two histories diverge with no automatic reconciliation. If you need a two-node deployment to survive one loss, add a Galera arbitrator with garbd as a third vote instead of disabling split-brain protection.
  • Bootstrapping the wrong survivor is silent. pc.bootstrap will happily promote a node whose seqno is behind a peer’s; there is no warning, and the higher-seqno peer’s extra transactions are discarded when it resyncs downward. Always compare wsrep_last_committed (or --wsrep-recover output) across every survivor first — the comparison is the safety mechanism, not the command.
  • Two accidental bootstraps fork the Galera cluster. If two operators each promote a different node, or automation fires pc.bootstrap on more than one, you get two Primary components with different wsrep_cluster_conf_id values and divergent data. Recovery then means picking one component as truth, dumping any writes that landed on the other, and re-SST-ing its nodes. Serialize bootstrap authority to a single actor.