mysqld cleanup segfaults when rsync SST fails

[Alex Yurchenko]

From the codership mailing list:

I use ./mysql-galera -g gcomm://0.0.0.0:4567 start to start first
node successfully,
But when I use ./mysql-galera -g gcomm://first_ip:4567 start in
second node, the mysqld coredump.

   The bt output of gdb libexec/mysqld var/core is

(gdb) bt
#0 0x00000033d660b9b2 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000005e2a56 in handle_segfault (sig=11) at mysqld.cc:2722
#2 <signal handler called>
#3 plugin_thdvar_init (thd=0x2e308f0) at sql_plugin.cc:664
#4 0x0000000002e32410 in ?? ()
#5 0x0000000002e308f0 in ?? ()
#6 0x0000000002e32410 in ?? ()
#7 0x0000000002e33328 in ?? ()
#8 0x00007fff02cf0e70 in ?? ()
#9 0x00000000005d1110 in THD::init (this=0x2e30de8) at sql_class.cc:984
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Error log:
111110 19:37:09 [ERROR] WSREP: Could not prepare state transfer request: failed to guess address to accept state transfer at. wsrep_sst_receive_address must be set manually.
111110 19:37:09 [ERROR] Aborting
...
111110 19:37:10 [Note] WSREP: rollbacker thread exiting
111110 19:37:11 [Note] WSREP: Some threads may fail to exit.
111110 19:37:11 InnoDB: Starting shutdown...
111110 19:37:11 - mysqld got signal 11 ;

[Alex Yurchenko (ayurchen)]

The cause:
> When the server run in the function "plugin_thdvar_init"
> (sql_plugin.cc:2540), the value of
> global_system_variables.table_plugin is "NULL", which leades to the
> crash.

> The same reason can lead to coredump in
> plugin_thdvar_init(sql_plugin.cc), acl_getroot(sql_acl.cc) and
> inc_host_errors(hostname.cc).

[Seppo Jaakola (seppo-jaakola)]

Trying to figure out how to reproduce this, here's the steps I used for this:
1. start first node
2. configure second node with:
    wsrep_sst_method= rsync
    #wsrep_sst_receive_address=
3. start the second node

This caused an assert for the second node running wsrep-5.5 build:

111204 11:26:16 [Note] WSREP: WSREP rollback thread wakes for signal
111204 11:26:16 [Note] WSREP: WSREP rollback thread has empty abort queue
111204 11:26:16 [Note] WSREP: rollbacker thread exiting
/home/seppo/work/testing/node2-5.5/mysql/sbin/mysqld: debugger aborting because missing DBUG_RETURN or DBUG_VOID
_RETURN macro in function "?func"

[Seppo Jaakola (seppo-jaakola)]

Pushed a fix for wsrep-5.5: http://bazaar.launchpad.net/~codership/codership-mysql/wsrep-5.5/revision/3637

With this, the "howto reproduce" scenario succeeds, but prints one spurious warning in the log:

111204 11:34:06 [Note] WSREP: rollbacker thread exiting
111204 11:34:07 [Note] WSREP: Some threads may fail to exit.
111204 11:34:07 [Note] /home/seppo/work/testing/node2-5.5/mysql/sbin/mysqld: Shutdown complete

Error in my_thread_global_end(): 2 threads didn't exit

[Seppo Jaakola (seppo-jaakola)]

pushed similar fix in wsrep-5.1: http://bazaar.launchpad.net/~codership/codership-mysql/wsrep-5.1/revision/3172

However, this vulnerability did not fire in reproduction test. Fix is a good for sanity, anyways.

The howto reproduce scnenario with wsrep-5.1 build, gives me same success as wsrep-5.5:

111204 12:12:25 [Note] WSREP: rollbacker thread exiting
111204 12:12:26 [Note] WSREP: Some threads may fail to exit.
111204 12:12:26 [Note] /home/seppo/work/testing/node2-5.1/mysql/sbin/mysqld: Shutdown complete

Error in my_thread_global_end(): 2 threads didn't exit