segmentation fault against Percona MySQL 5.6.15-63.0

[Christoph Haas]

For a couple of days mydumper stopped working for us and threw segmentation faults seconds after the lock has been acquired and threads were launched. I'm no gdb guru but this is what I could find out:

root@db1: mydumper -V
mydumper 0.6.2, built against MySQL 5.6.15
root@db1:~# gdb /usr/bin/mydumper
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /usr/bin/mydumper...(no debugging symbols found)...done.
(gdb) run -B memoria2_aggregation -e -o /mnt/ssd-raid0-tmp/mydumper/data/ -c -L /mnt/ssd-raid0-tmp/mydumper/mydumper.log -t 10 -v 4
Starting program: /usr/bin/mydumper -B database1 -e -o /mnt/ssd-raid0-tmp/mydumper/data/ -c -L /mnt/ssd-raid0-tmp/mydumper/mydumper.log -t 10 -v 4
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff5c0c700 (LWP 47022)]
[Thread 0x7ffff5c0c700 (LWP 47022) exited]
[New Thread 0x7ffff5c0c700 (LWP 9632)]
[New Thread 0x7ffff4fe8700 (LWP 9637)]
[New Thread 0x7fffeffff700 (LWP 9640)]
[New Thread 0x7fffef7fe700 (LWP 9645)]
[New Thread 0x7fffeeffd700 (LWP 9648)]
[New Thread 0x7fffee7fc700 (LWP 9651)]
[New Thread 0x7fffedffb700 (LWP 9654)]
[New Thread 0x7fffed7fa700 (LWP 9657)]
[New Thread 0x7fffecff9700 (LWP 9660)]
[New Thread 0x7fffcbfff700 (LWP 9663)]

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff668f399 in strtoull_l () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

This problem occurs in both 0.6.1 and 0.6.2. If I remember correctly we could not use version 0.5.1 in Ubuntu Precise right from the start because it failed to backup Percona MySQL servers.

[Max Bubenick (max-bubenick)]

Confirmed, when you have broken/invalid views, mydumper is not able to detect them by SHOW TABLE STATUS as the comment change and then try to get the data_size.

[Christoph Haas (chaas)]

How can I determine which views are broken/invalid? Those where the Data_length of the views is NULL?

[Max Bubenick (max-bubenick)]

with show table status you will see this

| v | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | View 'a.v' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them |

[Hubertus Krogmann (hubertus-krogmann)]

Hello

I got a static link version of 0.9.0 at a workshop at Percona Live 2015 which I used, but it segfaulted.

I did a bzr branch lp:mydumper (Branched 183 revision(s).) and compiled it.

mydumper -V
mydumper 0.9.0, built against MySQL 5.6.26-74.0
(Red Hat Enterprise Linux Server release 6.6 (Santiago))

But it also segfaulted

mydumper -v 3 -u mydumperuser -p "..." -h ... -P 3306 --compress-protocol -t 16 -B xxx -o /data2/mydumper/test3 -c -e -k -L /data2/mydumper/test3.log
last 2 lines in the log are "Thread xy dumping view"
show tables status for select TABLE_NAME from VIEWS where TABLE_SCHEMA = 'xxx' showed no invalid views

the source DB is 5.0.77 and a slave (-k is wanted, I don't want to interrupt replication)
Anything I could use for a workaround, or do to help finding the issue?
(I assume bzr branch ... gives the latest code?)

I tried -t 1 --ignore-engines MyISAM,InnoDB to see what is left and at which object it breaks.
It died on first view, so I tried without -c and run again, segfaulted again.

it generates 2 files (beside metadata.partial and xxx-schema-create.sql )
db.view1-schema.sql contains a create table ... ENGINE=MyISAM; statement
db.view1-schema-view.sql contains DROP TABLE ... ; DROP VIEW ...; statements

as we have severeal hundreds of tables -T is not really an option, is there a --no-views option to use.

last run with gdb:
gdb mydumper
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/bin/mydumper...done.
(gdb) run -v 3 -u mydumperuser -p "" -h -P 3306 --compress-protocol -t 1 --ignore-engines MyISAM,InnoDB -B -o /data2/mydumper/test6 -e -k -L /data2/mydumper/test6.log
...
[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff5f62700 (LWP 5755)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5f62700 (LWP 5755)]
0x000000360d047e2c in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install Percona-Server-shared-56-5.6.24-rel72.2.el6.x86_64 glib2-2.28.8-4.el6.x86_64 glibc-2.12-1.149.el6_6.9.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libgcc-4.4.7-16.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libstdc++-4.4.7-16.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 pcre-7.8-7.el6.x86_64 zlib-1.2.3-29.el6.x86_64

Hubertus

[Hubertus Krogmann (hubertus-krogmann)]

Hello
I overlooked the use: debuginfo-install .... line, did what it said, new run in gdb

run -v 3 -u mydumperuser -p "" -h -P 3306 --compress-protocol -t 1 --ignore-engines MyISAM,InnoDB -B xxx -o /data2/mydumper/test7 -e -k -L /data2/mydumper/test7.log

[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff7860700 (LWP 1191)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7860700 (LWP 1191)]
0x000000360d047e2c in _IO_vfprintf_internal (s=<value optimized out>, format=<value optimized out>, ap=<value optimized out>)
    at vfprintf.c:1641
1641 process_string_arg (((struct printf_spec *) NULL));

Hubertus

[Max Bubenick (max-bubenick)]

Hi, that seems not to be a mydumper bug.

And also you are running mysql version released almost 7 years ago, and what about your OS? I would recommend you to upgrade or use mysqldump for that version.

[Christoph Haas (chaas)]

@Max: MySQL 5.6.15 is not even two years old. And why do you think it's not a mydumper bug? Or which post are you referring to?

[Max Bubenick (max-bubenick)]

Sorry guys, but we are mixing things here.

Christoph your bug is already fixed on trunk code since March.

Hubertus lets move yours to a different place.

[Christoph Haas (chaas)]

@Max: Thanks for the clarification. I'll try the trunk version.