Signal 6 on shutdown on FreeBSD

[Olivier - interfaSys]

I had installed 5.2.4 on FreeBSD 8.1-zfs, gcc 4.5.2

There is no problem with MySQL 5.1.54, but with MariaDB, I always got this when stopping mysqld.

101222 13:08:55 [Note] /usr/local/libexec/mysqld: Normal shutdown
101222 13:08:55 [Note] Event Scheduler: Purging the queue. 0 events
101222 13:08:55 InnoDB: Starting shutdown...
101222 13:08:55 InnoDB: Shutdown completed; log sequence number 132903848930
pure virtual method called
terminate called without an active exception
101222 13:08:55 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=0
read_buffer_size=134217728
max_used_connections=3
max_threads=502
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 131602382 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

101222 13:08:55 mysqld_safe mysqld restarted

[Kristian Nielsen (knielsen)]

To debug this, we need a stack trace. Can you obtain a core dump and run gcc on it with the command `bt` ?
(Or obtain a stack trace from the crash some other way?)

Also, it might be useful to get the exact commands used to build, since you mention gcc version I suppose you built mariadb yourself.

[Olivier - interfaSys (olivier-interfasys)]

I've checked the filesystem. There is no mysqld.core anywhere.

The current problem is that because of the crash, a new instance is immediately restarted and mysqld cannot be stopped without killing mysqld_safe.

MariaDB was built from ports:
http://www.freshports.org/databases/mariadb/
I don't really know how to quickly get the build options since there doesn't seem to be anything specific in the port.

I've also tried with 5.2.3, same problem.

# ldd /usr/local/libexec/mysqld
/usr/local/libexec/mysqld:
        libz.so.5 => /lib/libz.so.5 (0x800ba7000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x800cbc000)
        libthr.so.3 => /lib/libthr.so.3 (0x800dd5000)
        librt.so.1 => /usr/lib/librt.so.1 (0x800eed000)
        libstdc++.so.6 => /usr/local/lib/gcc45/libstdc++.so.6 (0x800ff2000)
        libm.so.5 => /lib/libm.so.5 (0x801125000)
        libgcc_s.so.1 => /usr/local/lib/gcc45/libgcc_s.so.1 (0x800a99000)
        libc.so.7 => /lib/libc.so.7 (0x801244000)

Info on GCC
Using built-in specs.
COLLECT_GCC=/usr/local/bin/gcc45
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc45/gcc/x86_64-portbld-freebsd8.1/4.5.2/lto-wrapper
Target: x86_64-portbld-freebsd8.1
Configured with: ./../gcc-4.5.2/configure --enable-lto=yes --enable-gold --enable-threads --with-ppl --with-cloog --disable-ppl-version-check --with-libelf=/usr/local --disable-nls --libdir=/usr/local/lib/gcc45 --libexecdir=/usr/local/libexec/gcc45 --program-suffix=45 --with-as=/usr/local/bin/as --with-gmp=/usr/local --with-gxx-include-dir=/usr/local/lib/gcc45/include/c++/ --with-ld=/usr/local/bin/ld --with-libiconv-prefix=/usr/local --with-system-zlib --disable-libgcj --prefix=/usr/local --mandir=/usr/local/man --infodir=/usr/local/info/gcc45 --build=x86_64-portbld-freebsd8.1
Thread model: posix
gcc version 4.5.2 (GCC)

[Olivier - interfaSys (olivier-interfasys)]

I've found this in log/messages

pid 42698 (factorial), uid 0: exited on signal 11 (core dumped)
Would that have an impact on the final binary or is it just for tests?

[Sergei Golubchik (sergii)]

Hi, Olivier!

On Dec 23, Olivier - interfaSys wrote:
> I've found this in log/messages
>
> pid 42698 (factorial), uid 0: exited on signal 11 (core dumped)
> Would that have an impact on the final binary or is it just for tests?

It's not even for tests.
It's for generating the manual of DBUG debugging library
(the output of factorial binary is included in the troff file).
This manual is not installed (as far as I know).

Regards,
Sergei

[Olivier - interfaSys (olivier-interfasys)]

Thank you Sergei :)

I compiled it again to get more info. Here is the generated config.log

[Kristian Nielsen (knielsen)]

I don't know how to proceed without a stack trace or similar info about where exactly the mysqld binary is crashing.

To get a core file you may need to add --core-file option to my.cnf and/or make sure `ulimit -c unlimited` is active for mysqld. You may also need to ensure that mysqld has write permissions to current directory (or wherever FreeBSD puts core files, this is somewhat system dependent AFAIK).

Alternatively, start mysqld by itself (without using mysqld_safe) from within gdb, giving the same arguments that mysqld_safe would pass. This should also provide a stack trace.

[Olivier - interfaSys (olivier-interfasys)]

The option in my.cnf did the trick, but there isn't much information.

(gdb) bt
#0 0x000000080129903c in ?? ()
#1 0x0000000800de2d72 in ?? ()
#2 0x0000000000000000 in ?? ()
#3 0x0000000000000006 in ?? ()
#4 0x0000000000000000 in ?? ()
#5 0x0000000801608d00 in ?? ()
#6 0x0000000000000000 in ?? ()
#7 0x00000000005b3ef0 in ?? ()
#8 0x000000000000002c in ?? ()
#9 0x0000000800000031 in ?? ()
#10 0x0000002c00000031 in ?? ()
#11 0x0000001800000003 in ?? ()
#12 0x0000006e0000000b in ?? ()
#13 0x0000016500000005 in ?? ()
#14 0x0000000000000000 in ?? ()
#15 0x0000000000000000 in ?? ()
#16 0x0000000801477df0 in ?? ()
#17 0x000000004d141731 in ?? ()
#18 0x00007ffffd7afecc in ?? ()
#19 <signal handler called>
#20 0x000000080129903c in ?? ()
#21 0x00000008013351cb in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0xffffffdf00000000 in ?? ()
#24 0xffffffffffffffff in ?? ()
#25 0x00000000ffffffff in ?? ()
#26 0x0000000801608d00 in ?? ()
#27 0x00000000012a0e60 in ?? ()
#28 0x0000000801608d00 in ?? ()
#29 0x00000008010cd915 in ?? ()
#30 0x0000000000019008 in ?? ()
#31 0x00000000012a0f4c in ?? ()
#32 0x0000000000000001 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000801608d00 in ?? ()
#35 0x00000008010ca4f6 in ?? ()
#36 0x000000000000001b in ?? ()
#37 0x0000000000000202 in ?? ()
#38 0x0000000801608d00 in ?? ()
#39 0x00000008010ca543 in ?? ()
#40 0x00000008010ca530 in ?? ()
#41 0x00000008010cbdef in ?? ()
#42 0x00000008010cbdd0 in ?? ()
#43 0x00000000005b33af in ?? ()
#44 0x0000000000000000 in ?? ()
#45 0x0000000000a8cc68 in ?? ()
#46 0x0000000000000000 in ?? ()
#47 0x00000000005b51b2 in ?? ()
#48 0x000000004d14172e in ?? ()
#49 0x000000001f7dc9f8 in ?? ()
#50 0x000000080e9f4e40 in ?? ()
#51 0x000000080e9f4e48 in ?? ()
#52 0x0000000801608d00 in ?? ()
#53 0x0000000800ddd511 in ?? ()
#54 0x0000000000000000 in ?? ()
#55 0x0000000000000000 in ?? ()
#56 0x0000000000000000 in ?? ()
#57 0x0000000000000000 in ?? ()
#58 0x0000000000000000 in ?? ()
#59 0x0000000000a8da10 in ?? ()
#60 0x0000000000000000 in ?? ()
#61 0x0000000000000000 in ?? ()
Cannot access memory at address 0x7ffffd7b0000

[Kristian Nielsen (knielsen)]

Right, looks like the binary is stripped. We would need the backtrace from an unstripped binary.

[Olivier - interfaSys (olivier-interfasys)]

Well, I can't seem to get anything else, even with "-g" in the cflags

[Sergei Golubchik (sergii)]

I haven't used FreeBSD for about 10 years, and I'm not sure the following will work, but still

# make CONFIGURE_ARGS=--with-debug=full install

would that help? Or you can try some other way to pass --with-debug=full down to mariadb's configure.

It doesn't look like port's Makefile strips the binaries, it must be our Makefile then.
But just in case, you can run make in script:

# script -c 'make CONFIGURE_ARGS=--with-debug=full install'

and attach the resulting typescript file to the bug report.

[Olivier - interfaSys (olivier-interfasys)]

I've got a feeling that it's the XtraDB plugin that is causing this since it's the only component that seems different to what comes with MySQL which has no problem.

Is there an easy way to apply there configure args only to the xtradb plugin?

[Sergei Golubchik (sergii)]

not really. you can build with debug, copy ha_xtradb.so someplace, build and install without debug and copy debug version of ha_xtradb.so over.

but most probably it won't work - in debug build certain structures have different sizes and offsets, and xtradb may use them.

anyway, you can build mariadb with innodb and without xtradb. it's --with-plugin-innodb_plugin --without-plugin-xtradb.

But perhaps you don't need to do even that. Looking at your attached config.log I see that both XtraDB and InnoDB are built as dynamic plugins. That is, all you need to do to make sure that XtraDB is out of the question is "SHOW PLUGINS" to make sure that ha_xtradb.so is not listed and "UNINSTALL PLUGIN innodb" if it is. No need to rebuild anything.

[Olivier - interfaSys (olivier-interfasys)]

OK, so I disabled Innodb in my.cnf and there is no more crashes, so it's definitely xtradb that crashes.
Maybe it chokes on some data :S

[Olivier - interfaSys (olivier-interfasys)]

Some additional information: I'm using the gold linker.
I'm trying to see if I can compile it with a different linker to see if it helps.

[Olivier - interfaSys (olivier-interfasys)]

This seems to have been fixed in 5.2.5

[Stewart Smith (stewart)]

All development of XtraDB has moved under the Percona Server project - https://launchpad.net/percona-server - If this bug can be reproduced against current Percona Server, please file this bug against percona-server (you can simply do so by using the "Also affects project" link above).

Thanks,
Stewart Smith
Director of Server Development
Percona.