Diagnosing IST Failures & gcache Eviction

This diagnostic guide extends gcache Sizing for IST & Fast Recovery and answers the question every operator asks after a rejoin drags on for an hour: why did this node fall back to a full State Snapshot Transfer when it should have caught up with a quick Incremental State Transfer? The answer is almost always that the sequence number the joiner needed was no longer in the donor’s write-set ring — but “the ring was too small” is only one of four distinct causes, and they demand different fixes. This page shows you the exact log lines to read on both the donor and the joiner, how to confirm eviction from the ring, how page-store eviction and gcache.keep_pages_size interact, and a decision tree that maps each symptom to its remedy.

Context: Why IST Silently Degrades to SST in Multi-Master #

A rejoining node in a synchronous multi-master cluster asks a donor for every write-set it missed, identified by seqno. IST succeeds only if the donor still holds an unbroken chain of write-sets from the joiner’s last applied seqno forward. The failure is quiet: nothing errors at the moment of eviction: write-sets simply age out of the ring in the normal course of operation, and the shortfall becomes visible only later, when a node happens to rejoin and its required seqno has already scrolled off the bottom. Because the provider cannot replay a gap, it abandons IST and starts SST — the expensive full-datadir path described in Initial Data Synchronization Methods. Diagnosing this well means proving which of the several failure modes occurred, because raising gcache.size fixes eviction but does nothing for an unclean-shutdown seqno of -1 or a page-size mismatch.

Solution: Read the Logs and Prove the Cause #

Every IST failure leaves a paired trail — one line on the joiner explaining what it asked for, one on the donor explaining why it could not serve it. Read both.

Step 1 — Confirm which path the rejoin actually took #

On the joiner, the two outcomes are unmistakable in the error log:

# IST success looks like this:
sudo grep -E "Receiving IST|IST received" /var/log/mysql/error.log
#   WSREP: Receiving IST: 40217 writesets, seqnos 918342-958559

# Fallback to SST looks like this:
sudo grep -E "IST failed|Failed to prepare for incremental|Running: .*sst" /var/log/mysql/error.log
#   WSREP: Failed to prepare for incremental state transfer ... falling back to SST
#   WSREP: Running: 'wsrep_sst_mariabackup --role joiner ...'

The phrase Failed to prepare for incremental state transfer followed by a fallback to wsrep_sst_* is the definitive signature that IST was attempted and rejected.

Step 2 — Prove or rule out ring eviction #

Eviction is the first and most common cause. Compare the seqno the joiner needed against the oldest seqno the donor still holds. On the donor, read the low-water mark:

-- On the donor: the oldest seqno still replayable from gcache
SHOW GLOBAL STATUS LIKE 'wsrep_local_cached_downto';

On the joiner, its last applied seqno is in grastate.dat (or the recovered value from Step 3). If the joiner’s seqno is less than the donor’s wsrep_local_cached_downto, the write-set chain the joiner needs has been overwritten in the ring — this is textbook eviction, and the fix is to enlarge the ring per the gcache sizing worksheet. The donor log states it plainly:

WSREP: IST first seqno 918342 not found from cache, falling back to SST

Step 3 — Rule out an unclean-shutdown seqno of -1 #

If eviction is not the cause, inspect the joiner’s saved position. A node that crashed or was killed rather than shut down cleanly writes seqno: -1 and safe_to_bootstrap: 0 into grastate.dat, meaning it does not know how far it had applied — and a node with an unknown position cannot request IST, so it is forced to SST regardless of ring size:

sudo cat /var/lib/mysql/grastate.dat
# seqno:   -1        <- position unknown; IST impossible
# safe_to_bootstrap: 0

# Recover the real seqno from the InnoDB redo log without joining:
sudo -u mysql mariadbd --wsrep-recover
#   WSREP: Recovered position: 5f3a...:958517

mariadbd --wsrep-recover reads the committed position out of InnoDB and writes it back, restoring the node’s ability to request IST. This mechanism and the surrounding startup diagnostics are covered in Handling Galera Startup Errors & Logs.

Step 4 — Understand page-store eviction and keep_pages_size #

The gcache is not purely in-memory. When the ring or an individual write-set exceeds the RAM allocation, the provider spills to on-disk page-store files sized by gcache.page_size. As write-sets are consumed, the provider frees drained page files — this is page eviction, and it is normal. The problem arises under spiky write loads: the provider repeatedly allocates and deletes page files, and in the churn it can free pages that still held write-sets a slow joiner needed. gcache.keep_pages_size counters this by instructing the provider to keep at least that many bytes of page-store files allocated rather than deleting them the instant they drain:

[mysqld]
# Keep 4G of drained page files allocated so spiky writes don't evict
# write-sets a lagging joiner still needs.
wsrep_provider_options = "gcache.size=14G; gcache.page_size=1G; gcache.keep_pages_size=4G"

Setting gcache.keep_pages_size to a few gigabytes stabilizes the effective retained window on bursty workloads, where the nominal gcache.size alone can be misleading because page churn evicts history faster than the byte budget implies.

Parameter Reference #

Verification #

After applying a fix, force a controlled rejoin and confirm it now takes IST. Stop one node cleanly, wait a couple of minutes so it genuinely lags, then restart it and watch the log:

# On the rejoining node, in one terminal:
sudo tail -f /var/log/mysql/error.log | grep -E "IST|SST|Recovered position"
# Expect: 'Receiving IST: N writesets, seqnos A-B'  (NOT 'Running: wsrep_sst_...')

Then confirm from status that the join completed cleanly and the node is back in the primary component:

SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_local_state_comment',  -- expect 'Synced'
  'wsrep_cluster_status'        -- expect 'Primary'
);

If it still fell back to SST, re-walk the decision tree — the cause you fixed was not the operative one.

Edge Cases & Gotchas #

• A donor under load can evict mid-IST. If the donor keeps taking heavy writes while serving a slow joiner, its ring can advance past the seqno it is still streaming, aborting the IST partway through. On write-heavy clusters, prefer a less-loaded donor via wsrep_sst_donor, or raise gcache.keep_pages_size so in-flight history is protected.
• gcache.recover=no turns every restart-then-rejoin into SST. Even a correctly sized ring is discarded at restart unless recovery is enabled, so a node that reboots and immediately needs to donate has an empty cache. Confirm gcache.recover=yes is set, as detailed in gcache Sizing for IST & Fast Recovery.
• A page-size mismatch masquerades as eviction. If gcache.page_size differs between donor and joiner, the log may show an IST failure that looks like a cache miss but is really a provider-level incompatibility. Confirm the value is identical across the group with SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options' before enlarging the ring, since more space will not fix a mismatch.

Related #

• gcache Sizing for IST & Fast Recovery — the configuration reference for the ring you are diagnosing
• Sizing gcache to Avoid a Full SST on Rejoin — the arithmetic that prevents eviction in the first place
• Handling Galera Startup Errors & Logs — reading provider-load and recovery output, including --wsrep-recover
• Initial Data Synchronization Methods — the SST fallback whose cost makes these diagnostics worth doing