toku_pthread.h:114 toku_mutex_init: Assertion `!mutex->valid ' failed (errno=2)

Bug #1510908 reported by Shahriyar Rzayev
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Status tracked in 5.7
5.6
Triaged
High
Unassigned
5.7
Triaged
High
Unassigned

Bug Description

After compiling/installing Percona Server 5.6.26 from source in Ubuntu 14.04 tried to enable TokuDB engine , but failed with assert + abandoned tokudb files in datadir.
Here is description:

1. Unpacked source archive to:

/home/sh/Sandboxes/percona-server-5.6.26-74.0

2. CMAKE command:

cmake -DCMAKE_INSTALL_PREFIX=/opt/percona-5.6.26 -DMYSQL_DATADIR=/opt/percona-5.6.26/datadir -DSYSCONFDIR=/opt/ -DWITH_SSL=system -DMYSQL_TCP_PORT=3307 -DMYSQL_UNIX_ADDR=/opt/percona-5.6.26/mysqld.sock -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci -DWITH_DEBUG=1 -DCOMPILATION_COMMENT="Shahriyar Rzayev's Percona Server" -DOPTIMIZER_TRACE=1 -DWITH_ZLIB=system -DWITH_VALGRIND=1 -DCMAKE_C_FLAGS=-DHAVE_purify -DCMAKE_CXX_FLAGS=-DHAVE_purify

3. make + make install as well.

4. Installing system tables:
cd /opt/percona-5.6.26/scripts

sh@shrzayev:/opt/percona-5.6.26/scripts$ sudo ./mysql_install_db --no-defaults --basedir=/opt/percona-5.6.26 --user=root --datadir=/opt/percona-5.6.26/datadir

5. Starting Percona Server as:

sh@shrzayev:/opt/percona-5.6.26$ sudo /opt/percona-5.6.26/bin/mysqld_safe --defaults-file=/opt/percona-5.6.26/my.cnf --user=mysql --datadir=/opt/percona-5.6.26/datadir --socket=/opt/percona-5.6.26/datadir/mysqld-new.sock --pid-file=/opt/percona-5.6.26/datadir/mysqld-new.pid --port=3307 --log-error=/opt/percona-5.6.26/datadir/error.err

6. Successfully connected:

sh@shrzayev:~$ /opt/percona-5.6.26/bin/mysql -u root -p12345 -S /opt/percona-5.6.26/datadir/mysqld-new.sock
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.26-74.0-debug Shahriyar Rzayev's Percona Server

Copyright (c) 2009-2015 Percona LLC and/or its affiliates
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

7. Trying to enable TokuDB engine using ps_tokudb_admin:

sh@shrzayev:~$ sudo /opt/percona-5.6.26/bin/ps_tokudb_admin --enable --user=root -p12345 -S /opt/percona-5.6.26/datadir/mysqld-new.sock
[sudo] password for sh:
Checking if Percona Server is running with jemalloc enabled...
INFO: Percona Server is running with jemalloc enabled.

Checking transparent huge pages status on the system...
INFO: Transparent huge pages are currently disabled on the system.

Checking if thp-setting=never option is already set in config file...
INFO: Option thp-setting=never is not set in the config file.
      (needed only if THP is not disabled permanently on the system)

Checking TokuDB engine plugin status...
INFO: TokuDB engine plugin is not installed.

Adding thp-setting=never option into /etc/my.cnf
INFO: Successfully added thp-setting=never option into /etc/my.cnf

Installing TokuDB engine...
ERROR: Failed to install TokuDB engine plugin. Please check error log.

Output from error log:

Version: '5.6.26-74.0-debug' socket: '/opt/percona-5.6.26/datadir/mysqld-new.sock' port: 3307 Shahriyar Rzayev's Percona Server
/home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/tokudb/PerconaFT/portability/toku_pthread.h:114 toku_mutex_init: Assertion `!mutex->valid
' failed (errno=2)
: No such file or directory
Backtrace: (Note: toku_do_assert=0x0x7efdf256c720)
/opt/percona-5.6.26/lib/mysql/plugin/ha_tokudb.so(db_env_do_backtrace(_IO_FILE*)+0x26)[0x7efdf256c28a]
/opt/percona-5.6.26/lib/mysql/plugin/ha_tokudb.so(+0x19e467)[0x7efdf256c467]
/opt/percona-5.6.26/lib/mysql/plugin/ha_tokudb.so(toku_do_assert_zero_fail(unsigned long, char const*, char const*, char const*, int, int)+0x0)
[0x7efdf256c56a]
/opt/percona-5.6.26/lib/mysql/plugin/ha_tokudb.so(+0x115e5c)[0x7efdf24e3e5c]
/opt/percona-5.6.26/lib/mysql/plugin/ha_tokudb.so(+0x117322)[0x7efdf24e5322]
/opt/percona-5.6.26/lib/mysql/plugin/ha_tokudb.so(toku_txn_begin_with_xid(tokutxn*, tokutxn**, tokulogger*, txnid_pair_s, __TXN_SNAPSHOT_TYPE,
__toku_db_txn*, bool, bool)+0xf2)[0x7efdf24e4e1c]
/opt/percona-5.6.26/lib/mysql/plugin/ha_tokudb.so(toku_txn_begin(__toku_db_env*, __toku_db_txn*, __toku_db_txn**, unsigned int)+0x4a0)[0x7efdf2
52f92b]
/opt/percona-5.6.26/lib/mysql/plugin/ha_tokudb.so(+0x9fc8d)[0x7efdf246dc8d]
/opt/percona-5.6.26/lib/mysql/plugin/ha_tokudb.so(+0x917f9)[0x7efdf245f7f9]
/opt/percona-5.6.26/bin/mysqld(ha_initialize_handlerton(st_plugin_int*)+0xea)[0x6464b2]
/opt/percona-5.6.26/bin/mysqld[0x8129bb]
/opt/percona-5.6.26/bin/mysqld(mysql_install_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*)+0x3a3)[0x814bb6]
/opt/percona-5.6.26/bin/mysqld(mysql_execute_command(THD*)+0x7e28)[0x80607a]
/opt/percona-5.6.26/bin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x5a5)[0x809ca2]
/opt/percona-5.6.26/bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd54)[0x7fb611]
/opt/percona-5.6.26/bin/mysqld(do_command(THD*)+0x3bb)[0x7fa448]
/opt/percona-5.6.26/bin/mysqld(do_handle_one_connection(THD*)+0x25b)[0x7bdf06]
/opt/percona-5.6.26/bin/mysqld(handle_one_connection+0x33)[0x7bd977]
/opt/percona-5.6.26/bin/mysqld(pfs_spawn_thread+0x159)[0xe5032b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7efe29229182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7efe2873647d]

Output from gdb:

(gdb) bt
#0 0x00007efe2922e621 in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1 0x0000000000b0bfed in my_write_core (sig=6) at /home/sh/Sandboxes/percona-server-5.6.26-74.0/mysys/stacktrace.c:422
#2 0x000000000073ff27 in handle_fatal_signal (sig=6) at /home/sh/Sandboxes/percona-server-5.6.26-74.0/sql/signal_handler.cc:236
#3 <signal handler called>
#4 0x00007efe28672cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5 0x00007efe286760d8 in __GI_abort () at abort.c:89
#6 0x00007efdf256c46c in toku_do_backtrace_abort ()
    at /home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/tokudb/PerconaFT/portability/toku_assert.cc:147
#7 0x00007efdf256c56a in toku_do_assert_fail (expr_as_string=0x7efdf2585484 "!mutex->valid",
    function=0x7efdf2585d00 <_ZZL15toku_mutex_initP10toku_mutexPK19pthread_mutexattr_tE12__FUNCTION__> "toku_mutex_init",
    file=0x7efdf2585420 "/home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/tokudb/PerconaFT/portability/toku_pthread.h", line=114,
    caller_errno=2) at /home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/tokudb/PerconaFT/portability/toku_assert.cc:168
#8 0x00007efdf24e3e5c in toku_mutex_init (mutex=0x7efdf2caf6e8, attr=0x0)
    at /home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/tokudb/PerconaFT/portability/toku_pthread.h:114
#9 0x00007efdf24e5322 in toku_txn_create_txn (tokutxn=0x7efe2a6b60b8, parent_tokutxn=0x0, logger=0x7efdf2cc21c0,
    snapshot_type=TXN_SNAPSHOT_NONE, container_db_txn=0x7efdf2ce0000, for_recovery=false, read_only=false)
    at /home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/tokudb/PerconaFT/ft/txn/txn.cc:273
#10 0x00007efdf24e4e1c in toku_txn_begin_with_xid (parent=0x0, txnp=0x7efdf2ce0090, logger=0x7efdf2cc21c0, xid=...,
    snapshot_type=TXN_SNAPSHOT_NONE, container_db_txn=0x7efdf2ce0000, for_recovery=false, read_only=false)
    at /home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/tokudb/PerconaFT/ft/txn/txn.cc:141
#11 0x00007efdf252f92b in toku_txn_begin (env=0x7efdf2caf080, stxn=0x0, txn=0x7efe2a6b61a0, flags=0)
    at /home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/tokudb/PerconaFT/src/ydb_txn.cc:577
#12 0x00007efdf246dc8d in env_open (env=0x7efdf2caf080, home=0x18738c0 <mysql_real_data_home> "/opt/percona-5.6.26/datadir/", flags=11403457,
    mode=438) at /home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/tokudb/PerconaFT/src/ydb.cc:951
#13 0x00007efdf245f7f9 in tokudb_init_func (p=0x7efdf2c22180)
    at /home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/tokudb/hatoku_hton.cc:501
#14 0x00000000006464b2 in ha_initialize_handlerton (plugin=0x7efdf2c203f0) at /home/sh/Sandboxes/percona-server-5.6.26-74.0/sql/handler.cc:696
#15 0x00000000008129bb in plugin_initialize (plugin=0x7efdf2c203f0) at /home/sh/Sandboxes/percona-server-5.6.26-74.0/sql/sql_plugin.cc:1137
#16 0x0000000000814bb6 in mysql_install_plugin (thd=0x7efdfb751000, name=0x7efdfb753db0, dl=0x7efdfb753dc0)
    at /home/sh/Sandboxes/percona-server-5.6.26-74.0/sql/sql_plugin.cc:1909
#17 0x000000000080607a in mysql_execute_command (thd=0x7efdfb751000) at /home/sh/Sandboxes/percona-server-5.6.26-74.0/sql/sql_parse.cc:5423
#18 0x0000000000809ca2 in mysql_parse (thd=0x7efdfb751000, rawbuf=0x7efdf2c20030 "INSTALL PLUGIN tokudb SONAME 'ha_tokudb.so'", length=43,
    parser_state=0x7efe2a6b8170) at /home/sh/Sandboxes/percona-server-5.6.26-74.0/sql/sql_parse.cc:6972
#19 0x00000000007fb611 in dispatch_command (command=COM_QUERY, thd=0x7efdfb751000,
    packet=0x7efdfb755001 "INSTALL PLUGIN tokudb SONAME 'ha_tokudb.so'", packet_length=43)
    at /home/sh/Sandboxes/percona-server-5.6.26-74.0/sql/sql_parse.cc:1442
#20 0x00000000007fa448 in do_command (thd=0x7efdfb751000) at /home/sh/Sandboxes/percona-server-5.6.26-74.0/sql/sql_parse.cc:1054
#21 0x00000000007bdf06 in do_handle_one_connection (thd_arg=0x7efe027e8000)
---Type <return> to continue, or q <return> to quit---
    at /home/sh/Sandboxes/percona-server-5.6.26-74.0/sql/sql_connect.cc:1541
#22 0x00000000007bd977 in handle_one_connection (arg=0x7efe027e8000) at /home/sh/Sandboxes/percona-server-5.6.26-74.0/sql/sql_connect.cc:1444
#23 0x0000000000e5032b in pfs_spawn_thread (arg=0x7efdfb711b60)
    at /home/sh/Sandboxes/percona-server-5.6.26-74.0/storage/perfschema/pfs.cc:1860
#24 0x00007efe29229182 in start_thread (arg=0x7efe2a6b9700) at pthread_create.c:312
#25 0x00007efe2873647d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Also if you list datadir there will be tokudb related files:

sh@shrzayev:~$ ls /opt/percona-5.6.26/datadir/ | grep toku
log000000000000.tokulog28
__tokudb_lock_dont_delete_me_data
__tokudb_lock_dont_delete_me_environment
__tokudb_lock_dont_delete_me_logs
__tokudb_lock_dont_delete_me_recovery
__tokudb_lock_dont_delete_me_temp
tokudb.rollback

But plugin is not activated:

mysql> select plugin_name, plugin_status from information_schema.plugins where plugin_name like '%toku%';
Empty set (0.00 sec)

Tags: tokudb
Revision history for this message
George Ormond Lorch III (gl-az) wrote :
no longer affects: percona-server/5.5
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-936

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.