Tokubackup initialisation fails with SIGFPE intermittently

[Laurynas Biveinis]

On 5.6 trunk, intermittently:

./mtr --force --max-test-fail=0 --suite-timeout=9999 --testcase-timeout=450 --parallel=auto --big-test --unit-tests-report --debug-server '--mysqld-env=LD_PRELOAD=/mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/build/plugin/tokudb-backup-plugin/Percona-TokuBackup/backup/libHotBackup.so /usr/lib64/libeatmydata.so' --suite tokudb.backup
...
worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 30046, winpid: 30046, exit: 256] died after mysql-test-run waited 0.2 seconds for /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/build/mysql-test/var/1/mysqld.1/data/auto.cnf to be created.
...
Core was generated by `/mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/build/'.
Program terminated with signal 8, Arithmetic exception.
#0 0x00002b5fc7797f4b in file_hash_table::hash (this=0x2b5fc79b2b40,
    file=0x12a705e0 "/selinux/mls")
    at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/plugin/tokudb-backup-plugin/Percona-TokuBackup/backup/file_hash_table.cc:136
136 return (the_hash[0]+the_hash[1])%m_size;
#0 0x00002b5fc7797f4b in file_hash_table::hash (this=0x2b5fc79b2b40,
    file=0x12a705e0 "/selinux/mls")
    at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/plugin/tokudb-backup-plugin/Percona-TokuBackup/backup/file_hash_table.cc:136
#1 0x00002b5fc7797e45 in file_hash_table::get (this=0x2b5fc79b2b40,
    full_file_path=0x12a705e0 "/selinux/mls")
    at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/plugin/tokudb-backup-plugin/Percona-TokuBackup/backup/file_hash_table.cc:109
#2 0x00002b5fc7797dcc in file_hash_table::get_or_create (
    this=0x2b5fc79b2b40, file_name=0x12a705e0 "/selinux/mls")
    at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/plugin/tokudb-backup-plugin/Percona-TokuBackup/backup/file_hash_table.cc:95
#3 0x00002b5fc7797cfa in file_hash_table::get_or_create_locked (
    this=0x2b5fc79b2b40, file_name=0x12a705e0 "/selinux/mls",
    file=0x7fff9ee47f28, flags=0)
    at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/plugin/tokudb-backup-plugin/Percona-TokuBackup/backup/file_hash_table.cc:77
#4 0x00002b5fc77a0af2 in manager::setup_description_and_source_file (
    this=0x2b5fc79b2b20, fd=3, file=0x7fff9ee48240 "/selinux/mls", flags=0)
    at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/plugin/tokudb-backup-plugin/Percona-TokuBackup/backup/manager.cc:1038
#5 0x00002b5fc779ef9e in manager::open (this=0x2b5fc79b2b20, fd=3,
    file=0x7fff9ee48240 "/selinux/mls", flags=0)
    at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/plugin/tokudb-backup-plugin/Percona-TokuBackup/backup/manager.cc:346
#6 0x00002b5fc77a5f36 in open (file=0x7fff9ee48240 "/selinux/mls", oflag=0)
    at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/centos5-64/plugin/tokudb-backup-plugin/Percona-TokuBackup/backup/backup.cc:108
#7 0x0000003d99a09cde in is_selinux_mls_enabled ()
   from /lib64/libselinux.so.1
#8 0x0000003d99a0fe0e in ?? () from /lib64/libselinux.so.1
#9 0x0000003d99a10306 in ?? () from /lib64/libselinux.so.1
#10 0x0000003d9894f798 in __CTOR_LIST__ () from /lib64/libc.so.6
#11 0x00002b5fc7dd5078 in ?? ()
#12 0x00002b5fc778d000 in ?? ()
#13 0x0000003d99a03f6b in _init () from /lib64/libselinux.so.1
#14 0x00002b5fc7bc44a8 in ?? ()
#15 0x0000003d97e0d4ab in call_init () from /lib64/ld-linux-x86-64.so.2
#16 0x0000003d97e0d5b5 in _dl_init_internal ()
   from /lib64/ld-linux-x86-64.so.2
#17 0x0000003d97e00aaa in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#18 0x0000000000000010 in ?? ()
    at /opt/percona-devtoolset/root/usr/include/c++/4.8.2/bits/stl_tree.h:549
#19 0x00007fff9ee4b6f4 in ?? ()
#20 0x00007fff9ee4b754 in ?? ()
#21 0x00007fff9ee4b76f in ?? ()
#22 0x00007fff9ee4b7e6 in ?? ()
#23 0x00007fff9ee4b7f8 in ?? ()
#24 0x00007fff9ee4b817 in ?? ()
#25 0x00007fff9ee4b882 in ?? ()
#26 0x00007fff9ee4bb7d in ?? ()
#27 0x00007fff9ee4bbf5 in ?? ()
#28 0x00007fff9ee4bc91 in ?? ()
#29 0x00007fff9ee4bcb3 in ?? ()
#30 0x00007fff9ee4bcd3 in ?? ()
#31 0x00007fff9ee4bcf2 in ?? ()
#32 0x00007fff9ee4bd0f in ?? ()
#33 0x00007fff9ee4bd24 in ?? ()
#34 0x00007fff9ee4bd3f in ?? ()
#35 0x0000000000000000 in ?? ()

Looks like division by zero in % m_size?

[Vlad Lesin (vlad-lesin)]

Can't repeat the crash.

file_hash_table::m_size is initialized explicitly in file_hash_table ctor with 1 and changed only during rehashing in file_hash_table::maybe_resize().

So I don't have assumptions about how m_size can become zero. The only theoretical assumption is overflow in file_hash_table::maybe_resize() in this line:

m_size = m_size + m_count;

m_count is a number of files in the hash table. I don't think there are so much files on test host to be the cause of overflow.

But there is one more ability to make m_count big enough for overflow. This is m_count decrement in file_hash_table::remove().

At this moment the only thing I can do is to insert asserts in the code in a weak hope to catch the moment when m_size becomes zero or m_count becomes size_t(-1).

[Vlad Lesin (vlad-lesin)]

The corresponding asserts are added https://github.com/percona/Percona-TokuBackup/pull/82 .

[George Ormond Lorch III (gl-az)]

Tracking in https://jira.percona.com/browse/TDB-45, please login there to Percona JIRA and 'watch' for future updates. Marking it as opinion here as there is no other accurate matching state.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-1072