mysqld allows client connections even when in NON_PRIMARY view

[Alex Yurchenko]

Currently clients are allowed to connect to mysqld server even when it is in non-primary view. This is ugly as client gets aware of inability to replicate (and therefore, commit a transaction) only in the commit phase and gets the following error:

./sqlgen: Query failed: 1180: Got error 1 during COMMIT
Query was: 'COMMIT'

When mysqld has a non-primary view it should be largely unoperational. The best situation would be that it is not accessible via network, so that load balancer can easily detect a failed node. If this is too restrictive, server may demand that any connection that wants to do anything should first explicitly disable replication for session:

SET @@wsrep_on=0;

Such flexible behaviour might be complex to implement. An easy solution (now that we store state persistently) would be graceful restart and not returning from wsrep_init() until a primary view is received. From the log file it looks like server does not go to listen for connections until wsrep_init() completes.

[Alex Yurchenko (ayurchen)]

Well, currently restarting means the node will loose all the information about cluster topology and won't know which nodes to reconnect to.

So for now the trick with global status variable wsrep_ready and thd->variables.wsrep_on seems to be the only solution.

[Seppo Jaakola (seppo-jaakola)]

Plan is to fix this issue by introducing new 'wsrep_ready' status variable.
 wsrep_ready flag will tell, if node is ready to operate in the cluster.
Flag will be initially 0, and is set to 1, when node state has reached the
same level as other cluster nodes. In practice, we set the flag after snapshot
state transfer is complete (or we know that SST is not needed).

When wsrep_ready is 0, the session processing will allow only non-cluster connections
to process (wsrep_on==0). Also SET commands are allowed to make it possible to
declare a connection as non-cluster connection. This SET loophole is needed to let
mysqldump to be loaded from the donor node.

[Seppo Jaakola (seppo-jaakola)]

The wsrep_sst_mysqldump script needs to be modified to specify the connection to joiner as wsrep_on==OFF connection. This makes it possible to load the dump, even that wsrep_ready==OFF
in the joiner side.

Assumption is that the donor node is joined to cluster already and mysqldump command from
donor node works with default wsrep_on==ON connection

[Alex Yurchenko (ayurchen)]

wsrep_sst_mysqldump already does this:

STOP_WSREP="SET @@wsrep_on=0;"
...
(echo $STOP_WSREP && $MYSQLDUMP && echo $SET_START_POSITION || exit 1) | $MYSQL

The whole wsrep_on thing was invented to prevent state dump from being replicated by joiner.

[Seppo Jaakola (seppo-jaakola)]

Not all mysql sessions to the joiner were declared with wsrep_on=OFF, and this caused the issues.
A fix for this has been committed, but in my testing, I have noticed a few cases, where the donor side was not wsrep_ready and SST failed for that.
However, many changes in startup/shutdown logic have happened since then, and I need to re-test with current implementation.

[Seppo Jaakola (seppo-jaakola)]

There has been many changes for SST manouvres. Testing with current trunk shows, that
it can happen that mysqld init phase can assume that SST is needed, but gcs layer
decides that node is in fact in sync. This will leave the node closed for external access
(wsrep_ready==OFF).

Here is the log for 2'nd node startup phase:

090812 10:05:12 [Warning] WSREP: Gap in state sequence. Need state transfer.
090812 10:05:12 [Note] /home/galera/mysql-5.1.36-2859/mysql/libexec/mysqld: read
y for connections.
Version: '5.1.36-debug' socket: '/home/galera/mysql-5.1.36-2859/mysql/var/mysql
d.sock' port: 3306 wsrep_0.6
090812 10:05:12 [Warning] WSREP: Group state: abe9d918-1dd1-11b2-0800-c025207f70
fc:0
090812 10:05:12 [Warning] WSREP: Local state: 00000000-0000-0000-0000-0000000000

[Seppo Jaakola (seppo-jaakola)]

Latest commit for Bug: #411814, changeset: #2862 propagates updates wsrep_last_committed and wsrep_group_uuid values from wsrep provider library to mysqld and makes sure all logic depending on these variables is now correct.

Testing with trunk version does not reveal problems with startup phase anymore.