Signal handler thread kills the server instead of main thread causing segfault

[David Shrewsbury]

With revno 1733, the server is segfaulting on shutdown when PBXT is enabled:

$ sbin/drizzled --datadir=$PWD/var --plugin_add=pbxt
$ bin/drizzle --shutdown

100827 10:02:20 InnoDB: Starting shutdown...
100827 10:02:22 InnoDB: Shutdown completed; log sequence number 44254
100827 10:02:22 [Note] PrimeBase XT Engine shutdown...
100827 10:02:22 - drizzled got signal 11;
This could be because you hit a bug. It is also possible that this binary
 or one of the libraries it was linked against is corrupt, improperly built,
 or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

read_buffer_size=131072
max_used_connections=1
connection_count=0
It is possible that drizzled could use up to
(read_buffer_size + sort_buffer_size)*thread_count
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Number of stack frames obtained: 9
()
pthread_kill()
SignalHandler::~SignalHandler()
drizzled::module::Registry::~Registry()
drizzled::module::Registry::shutdown()
main()
__libc_start_main()
sbin/drizzled() [0x60b609]
Segmentation fault (core dumped)

Backtrace:

(gdb) bt
#0 0x00007f10f9337abc in pthread_kill () from /lib/libpthread.so.0
#1 0x0000000000794cfe in write_core (sig=11) at drizzled/signal_handler.cc:97
#2 0x00000000007950fd in drizzled_handle_segfault (sig=11)
    at drizzled/signal_handler.cc:214
#3 <signal handler called>
#4 0x00007f10f9337a90 in pthread_kill () from /lib/libpthread.so.0
#5 0x0000000000a57e19 in SignalHandler::~SignalHandler() ()
#6 0x0000000000714281 in ~Registry (this=0x14b3d20,
    __in_chrg=<value optimized out>) at drizzled/module/registry.cc:44
#7 0x00000000007143f4 in drizzled::module::Registry::shutdown ()
    at drizzled/module/registry.cc:71
#8 0x000000000060c0da in main (argc=3, argv=0x7fff8463a488)
    at drizzled/main.cc:333

[Andrew Hutchings (linuxjedi)]

Crash is with signal handler plugin:

(gdb) frame 2
#2 0x00000000005abe72 in drizzled::module::Registry::~Registry (
    this=0xc0c570, __in_chrg=<value optimized out>)
    at drizzled/module/registry.cc:44
44 delete (*plugin_iter).second;
(gdb) print plugin_iter
$1 = {first = "signal handler", second = }

[Andrew Hutchings (linuxjedi)]

Crash without signal handler plugin also crashing:

#0 0x000000364160c5f0 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000005f8765 in drizzled::kill_drizzle ()
    at drizzled/sql_base.cc:4633
#2 0x0000000000616639 in drizzled::dispatch_command (
    command=<value optimized out>, session=0x2688aa0, packet=0x266b951 "0",
    packet_length=1) at drizzled/sql_parse.cc:238
#3 0x00000000005e7dea in drizzled::Session::executeStatement (this=0x2688aa0)
    at drizzled/session.cc:666
#4 0x00000000005e9766 in drizzled::Session::run (this=0x2688aa0)
    at drizzled/session.cc:522
#5 0x00007fffef2b3234 in runSession (arg=<value optimized out>)
    at ./plugin/multi_thread/multi_thread.h:67
#6 (anonymous namespace)::session_thread (arg=<value optimized out>)
    at plugin/multi_thread/multi_thread.cc:52
#7 0x0000003641607761 in start_thread () from /lib64/libpthread.so.0
#8 0x00000036412e14ed in clone () from /lib64/libc.so.6

[Andrew Hutchings (linuxjedi)]

(doh! of course you would get that fail)

[Andrew Hutchings (linuxjedi)]

*somehow* signal_thread_in_use gets set to true after the thread ends. Haven't found where this happens yet, looks like a bad pointer or something?

[Andrew Hutchings (linuxjedi)]

ok ignore my last comment (never mess with GDB late at night), there are a few problems here. This is the sequence of events:

1. a SIGTERM or whatever is triggered
2. the signal handler thread catches this
3. the signal handler thread kills the server inside its own thread
4. this means the signal handler thread is now dead
5. the main thread (which should have been killed) now removes the signal handler plugin
6. the signal handler plugin destructor tries to kill the signal handler thread (which is already dead at this point)
7. pthread_kill segfaults because thread_id doesn't exist

OK, so the problems are:
1. a signal handler thread shouldn't do any processing/shutdown, just mark a flag (this is a _big_ mysqld problem too).
2. the main thread should do the shutdown, not the signal handler thread.

[Andrew Hutchings (linuxjedi)]

Fix causes valgrind warnings. Need to fix these too