server crashes with userstats ON

[Marc Castrovinci]

With userstat=1, mysql attempts to start and dies immediately. MySQL had been restarted before with userstat=1 without a problem, but now it is having problems. Only thing different is the box has swap now. I've verified this on multiple servers. Commenting out userstat=1 causes the server to start fine. It looks like this bug, https://bugs.launchpad.net/percona-server/+bug/1008278, which was supposed to be fixed in 5.5.25. I'm not using any type of replication setup. My version is: Server version: 5.5.25a-27.1-log Percona Server (GPL), Release 27.1

Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: 20:46:43 UTC - mysqld got signal 11 ;
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: This could be because you hit a bug. It is also possible that this binary
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: We will try our best to scrape up some info that will hopefully help
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: diagnose the problem, but since we have already crashed,
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: something is definitely wrong and this may fail.
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: Please help us make Percona Server better by reporting any
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: bugs at http://bugs.percona.com/
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld:
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: key_buffer_size=26214400
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: read_buffer_size=131072
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: max_used_connections=0
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: max_threads=1200
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: thread_count=0
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: connection_count=0
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: It is possible that mysqld could use up to
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 793909 K bytes of memory
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: Hope that's ok; if not, decrease some variables in the equation.
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld:
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: Thread pointer: 0x7fff9932e600
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: Attempting backtrace. You can use the following information to find out
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: where mysqld died. If you see no messages after this, something went
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: terribly wrong...
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: stack_bottom = 7fff99331710 thread_stack 0x30000
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7d0b75]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x69d004]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /lib/libpthread.so.0(+0xf8f0)[0x7f3ba90ff8f0]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld(my_hash_insert+0x331)[0x7b84d1]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld(_ZN7handler25update_global_table_statsEv+0x2f5)[0x6a3755]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xad)[0x55d6dd]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld(_Z19close_thread_tablesP3THD+0x18b)[0x55dccb]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld(_Z18close_mysql_tablesP3THD+0x11)[0x55dd51]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld[0x5a43b5]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x9ca)[0x5a75da]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld[0x52127a]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x5ed)[0x524e3d]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /lib/libc.so.6(__libc_start_main+0xfd)[0x7f3ba82c1c4d]
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: /usr/sbin/mysqld[0x51a515]

Aug 22 16:46:43 DEVELOP-DB daemon err mysqld:
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: Trying to get some variables.
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: Some pointers may be invalid and cause the dump to abort.
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: Query (0): is an invalid pointer
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: Connection ID (thread ID): 0
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: Status: NOT_KILLED
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld:
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: You may download the Percona Server operations manual by visiting
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: http://www.percona.com/software/percona-server/. You may find information
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: in the manual which will help you identify the cause of the crash.
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld:
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: The "--memlock" argument, which was enabled, uses system calls that are
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: unreliable and unstable on some operating systems and operating-system versions (notably, some versions of Linux).
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: This crash could be due to use of those buggy OS calls.
Aug 22 16:46:43 DEVELOP-DB daemon err mysqld: You should consider whether you really need the "--memlock" parameter and/or consult the OS distributer about "mlockall" bugs.
Aug 22 16:46:43 DEVELOP-DB daemon notice mysqld_safe: mysqld from pid file /var/lib/mysql/DEVELOP-DB.pid ended

[yinfeng (yinfeng-zwx)]

I think this is a duplicate bug of https://bugs.launchpad.net/percona-server/5.5/+bug/1011047

[Marc Castrovinci (marc-castrovinci)]

It is very similar, however I didn't add any plugins.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@marc,
This looks quite similar.

Can you tell us if plugin.MYD is empty? It is hard to check it now without crashing the server. So

strings $datadir/mysql/plugin.MYD should show something.

In this case like:

[root@archie mysql]# strings plugin.MYD
daemon_example
libdaemon_example.so

[Marc Castrovinci (marc-castrovinci)]

You are correct. Guess I did add plugins.

root@DEVELOP-DB:/var/lib/mysql/mysql# strings plugin.MYD
handlersocket
handlersocket.so