Debug build shutdown may crash with void add_global_thread(THD*): Assertion `!shutdown_in_progress' failed.

[Laurynas Biveinis]

Seen very intermittently on 5.6 trunk:

rpl.rpl_get_master_version_and_clock 'mix' w2 [ fail ] Found warnings/errors in server log file!
        Test ended at 2017-02-14 00:00:10
line
mysqld-debug: /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.cc:892: void add_global_thread(THD*): Assertion `!shutdown_in_progress' failed.
...
Thread 1 (Thread 0x7f279af97700 (LWP 9381)):
#0 0x00007f27997d6611 in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1 0x0000000000af0d17 in my_write_core (sig=6) at /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/stacktrace.c:422
#2 0x000000000073e8cd in handle_fatal_signal (sig=6) at /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/signal_handler.cc:236
#3 <signal handler called>
#4 0x00007f2798c0bc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5 0x00007f2798c0f028 in __GI_abort () at abort.c:89
#6 0x00007f2798c04bf6 in __assert_fail_base (fmt=0x7f2798d553b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xed75d0 "!shutdown_in_progress", file=file@entry=0xed7470 "/mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.cc", line=line@entry=892, function=function@entry=0xee0170 <add_global_thread(THD*)::__PRETTY_FUNCTION__> "void add_global_thread(THD*)") at assert.c:92
#7 0x00007f2798c04ca2 in __GI___assert_fail (assertion=0xed75d0 "!shutdown_in_progress", file=0xed7470 "/mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.cc", line=892, function=0xee0170 <add_global_thread(THD*)::__PRETTY_FUNCTION__> "void add_global_thread(THD*)") at assert.c:101
#8 0x0000000000627e43 in add_global_thread (thd=0x3728cb0) at /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.cc:892
#9 0x000000000062c531 in block_until_new_connection () at /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.cc:3019
#10 0x000000000062c66d in one_thread_per_connection_end (thd=0x3728cb0, block_pthread=true) at /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.cc:3073
#11 0x00000000007bac89 in do_handle_one_connection (thd_arg=0x3728cb0) at /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/sql_connect.cc:1567
#12 0x00000000007ba648 in handle_one_connection (arg=0x3728cb0) at /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/sql_connect.cc:1456
#13 0x0000000000e6f9b3 in pfs_spawn_thread (arg=0x36493e0) at /mnt/workspace/percona-server-5.6-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/perfschema/pfs.cc:1860
#14 0x00007f27997d1184 in start_thread (arg=0x7f279af97700) at pthread_create.c:312
#15 0x00007f2798ccf37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

The cause is that shutdown_in_progress accesses are not synchronised in any way, and there exists a race condition between shutdown and e.g. cached connection handling thread waking up to handle a new connection. The easiest fix is to remove the assert, as adding synchronisation will not buy us much.

[Laurynas Biveinis (laurynas-biveinis)]

This was introduced by a fix for bug 1617198

[Laurynas Biveinis (laurynas-biveinis)]

     https://github.com/percona/percona-server/pull/1441
     https://github.com/percona/percona-server/pull/1442

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3646