Mysql doesn't start on controllers in case nodes are being started in different order

[Eugene Nikanorov]

Consider the following case:
controllers are getting shut down in the order 1, 2, 3
When 1 is shut down, 2 and 3 are still active and receiving writes to the db.
Then 2 and 3 are shut down, in order.

When 1 is then started, mysql there becomes primary node.
When 2 and 3 start, they become joiners. Mysql on nodes 2 and 3 complains that it has more recent information than node1 and refuses to start.

OCF scripts never manage to start mysql on those nodes, until mysql is started with different set of parameters making the mysql instance to sync its state from the very beginning.

While mysql behavior is justified here - the user should make a decision, which information should be discarded and which should not be - in fact user can't do anything about it, also they are not notified about the problem in any meaningfull way.

All the user sees is 2 out of 3 nodes down in 'pcs status' output.

Expected result:
Some sort of notification for the user about what is the problem and what are the steps to fix it.

[Bogdan Dobrelya (bogdando)]

Full cluster shutdown is a rare case, so I believe it has a medium prio

[Bogdan Dobrelya (bogdando)]

Expected result:
Some sort of notification for the user about what is the problem and what are the steps to fix it.

Mysql logs should contain required info and steps for this corner case (look for a state diverged)