After a network split, a node can make a write progress and end-up with a diverged local seqno

[Bogdan Dobrelya]

How to reproduce:

Given:
* A cluster of 5 Galera nodes (percona packages)
* running in the multi-master writes/reads layout,
* auto-recovery decisions made by the custom Pacemaker OCF RA, it can only monitor/stop/start the mysqld_safe, no more interference, it removes nothing in the data path.

With that, do multiple runs of custom jepsen tests with Nemesis in the random network-split partitions mode (all links and details about test cases I described here https://goo.gl/VHyIIE , although that's tl;dr).

Results:
Expected: nodes will always recover after partitions with a merged state, isolated nodes can't make write progress w/o quorum.

Actual: a node (the n5 here) have done write progress, diverged, and refuses to start with the errors like:
[ERROR] WSREP: Local state seqno (189675) is greater than group seq no (188050): states diverged.

Logs, package versions, configs and wsrep status/vars are attached.

[Krunal Bauskar (krunal-bauskar)]

https://groups.google.com/forum/#!topic/codership-team/MOuSg_tiIOI

[Bogdan Dobrelya (bogdando)]

https://github.com/codership/galera/issues/401

[Krunal Bauskar (krunal-bauskar)]

upstream is closed will close this issue too.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1903