mysqld crash in check_grant_all_columns

[Rick Pizzi]

We have a repeating crash since a few days on a very busy server.

5.6.24-72.2-log Percona Server (GPL), Release 72.2, Revision 8d0f85b on CentOS 6.7

Stacktrace:

stack_bottom = 7fcc245c7d30 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8cd37c]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x6555a1]
/lib64/libpthread.so.0(+0xf710)[0x7ff8cf305710]
/usr/sbin/mysqld(my_hash_first+0xb)[0x8b003b]
/usr/sbin/mysqld(my_hash_search+0x11)[0x8b00a1]
/usr/sbin/mysqld(_Z23check_grant_all_columnsP3THDmP24Field_iterator_table_ref+0x125)[0x66e8b5]
/usr/sbin/mysqld(_Z13insert_fieldsP3THDP23Name_resolution_contextPKcS4_P13List_iteratorI4ItemEb+0x709)[0x68f469]
/usr/sbin/mysqld(_Z10setup_wildP3THDP10TABLE_LISTR4ListI4ItemEPS5_j+0x23d)[0x68f87d]
/usr/sbin/mysqld(_ZN4JOIN7prepareEP10TABLE_LISTjP4ItemjP8st_orderS5_S3_P13st_select_lexP18st_select_lex_unit+0x291)[0x6f3e61]
/usr/sbin/mysqld(_ZN18st_select_lex_unit7prepareEP3THDP13select_resultm+0x88b)[0x740ceb]
/usr/sbin/mysqld(_Z21mysql_derived_prepareP3THDP3LEXP10TABLE_LIST+0x12f)[0x6b025f]
/usr/sbin/mysqld(_Z20mysql_handle_derivedP3LEXPFbP3THDS0_P10TABLE_LISTE+0x66)[0x6b00c6]
/usr/sbin/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x90)[0x692c60]
/usr/sbin/mysqld[0x55b91e]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x27dc)[0x6d792c]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0xc0fb)[0x7ff8c8e810fb]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5a8)[0x6dc228]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x108c)[0x6dda4c]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x162)[0x6ab002]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6ab0f0]
/lib64/libpthread.so.0(+0x79d1)[0x7ff8cf2fd9d1]
/lib64/libc.so.6(clone+0x6d)[0x7ff8cda178fd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fcbb6076010): is an invalid pointer
Connection ID (thread ID): 9532
Status: NOT_KILLED

[Rick Pizzi (pizzi)]

Just happened again today.
In the meantime we have:

- switched server (new hardware)
- upgraded to 5.6.29-76.2

So, even in latest release bug still bites.

Thread pointer: 0x7f390ff24000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f3a4e965d00 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8d2e0c]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x6580b1]
/lib64/libpthread.so.0(+0xf7e0)[0x7f66637b07e0]
/usr/sbin/mysqld(my_hash_first+0xb)[0x8b5abb]
/usr/sbin/mysqld(my_hash_search+0x11)[0x8b5b21]/usr/sbin/mysqld(_Z23check_grant_all_columnsP3THDmP24Field_iterator_table_ref+0x125)[0x6713d5]
/usr/sbin/mysqld(_Z13insert_fieldsP3THDP23Name_resolution_contextPKcS4_P13List_iteratorI4ItemEb+0x709)[0x691e39]
/usr/sbin/mysqld(_Z10setup_wildP3THDP10TABLE_LISTR4ListI4ItemEPS5_j+0x23d)[0x69224d]
/usr/sbin/mysqld(_ZN4JOIN7prepareEP10TABLE_LISTjP4ItemjP8st_orderS5_S3_P13st_select_lexP18st_select_lex_unit+0x291)[0x6f7921]
/usr/sbin/mysqld(_ZN18st_select_lex_unit7prepareEP3THDP13select_resultm+0x88b)[0x7451ab]
/usr/sbin/mysqld(_Z21mysql_derived_prepareP3THDP3LEXP10TABLE_LIST+0x12f)[0x6b312f]
/usr/sbin/mysqld(_Z20mysql_handle_derivedP3LEXPFbP3THDS0_P10TABLE_LISTE+0x66)[0x6b2f96]
/usr/sbin/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x90)[0x6956e0]
/usr/sbin/mysqld[0x55c704]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1a9c)[0x6da75c]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0xc0fb)[0x7f660de810fb]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5a8)[0x6dfe18]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x106f)[0x6e161f]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x162)[0x6adec2]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6adfb0]
/usr/sbin/mysqld(pfs_spawn_thread+0x143)[0xb39f23]
/lib64/libpthread.so.0(+0x7aa1)[0x7f66637a8aa1]
/lib64/libc.so.6(clone+0x6d)[0x7f6661cb693d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f38c6406010): is an invalid pointer
Connection ID (thread ID): 68287450
Status: NOT_KILLED

[Rick Pizzi (pizzi)]

Creating a core file to help with this bug is a problem as this is our production master, and server has 256 GB of RAM... core would be huge and it would take a really long time to dump all the memory.

We cannot afford to have master offline for so long nor is it feasible to switch master at that time right after a crash.

If there is another solution to get more info about this bug without keeping the master in the stopped state please let us know.

[Rick Pizzi (pizzi)]

We have been able to reproduce this finally. It is not a MySQL bug, we are using McAfee audit plugin and the bug is in there (even if it crashes inside MySQL code): https://github.com/mcafee/mysql-audit/issues/145

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3402