Percona-XtraDB Cluster server-57 5.7.19 mysqld failed to stop on "systemctl stop mysql"

Bug #1738910 reported by Burnley on 2017-12-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
New
Undecided
Unassigned

Bug Description

It happened when trying to recover the cluster from a deadlock.

(gdb) bt full
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1 0x00007f0a7039be01 in _L_lock_1022 () from /lib64/libpthread.so.0
No symbol table info available.
#2 0x00007f0a7039bda2 in __GI___pthread_mutex_lock (mutex=0x342a8f0) at ../nptl/pthread_mutex_lock.c:134
        cnt = 11
        max_cnt = 1882850349
        type = 3
        id = <optimized out>
#3 0x00000000007b7b7c in native_mutex_lock (mutex=0x342a8f0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/include/thr_mutex.h:84
No locals.
#4 my_mutex_lock (mp=0x342a8f0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/include/thr_mutex.h:182
No locals.
#5 inline_mysql_mutex_lock (src_line=347, src_file=0x1384630 "/mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos7-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.19-29.22/sql/wsrep_mysqld.cc", that=0x342a8f0)
    at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/include/mysql/psi/mysql_thread.h:715
        locker = 0x0
        state = {m_flags = 3108082433, m_operation = (PSI_MUTEX_TRYLOCK | unknown: 32764), m_mutex = 0x3463160, m_thread = 0x7ffdb9419360, m_timer_start = 12843033556212825344, m_timer = 0x3463178, m_wait = 0x7f0a6327f5ee}
#6 wsrep_pfs_instr_cb (type=<optimized out>, ops=<optimized out>, tag=<optimized out>, value=<optimized out>, alliedvalue=<optimized out>, ts=<optimized out>)
    at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/wsrep_mysqld.cc:347
        mutex = 0x342a8f0
        __PRETTY_FUNCTION__ = "void wsrep_pfs_instr_cb(wsrep_pfs_instr_type_t, wsrep_pfs_instr_ops_t, wsrep_pfs_instr_tag_t, void**, void**, const void*)"
#7 0x00007f0a6325d1f8 in lock (this=0x3423c90) at galerautils/src/gu_mutex.hpp:92
No locals.
#8 Lock (pfs_mtx=..., this=0x7ffdb9419330) at galerautils/src/gu_lock.hpp:106
No locals.
#9 galera::ReplicatorSMM::sst_received (this=0x3423ab0, state_id=..., state=<optimized out>, state_len=<optimized out>, rcode=-125) at galera/src/replicator_str.cpp:74
        __FUNCTION__ = "sst_received"
        lock = {_vptr.Lock = 0x7f0a634ba490 <vtable for gu::Lock+16>, mtx_ = 0x0, pfs_mtx_ = 0x3423c90}
#10 0x00000000007c1591 in wsrep_sst_cancel (call_wsrep_cb=call_wsrep_cb@entry=true) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/wsrep_sst.cc:469
        state_id = {uuid = {data = '\000' <repeats 15 times>}, seqno = -1}
#11 0x00000000007a1bdf in unireg_abort (exit_code=1) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/mysqld.cc:1431
No locals.
#12 0x00000000007be433 in wsrep_init_startup (first=first@entry=true) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/wsrep_mysqld.cc:1189
No locals.
#13 0x00000000007a593f in init_server_components () at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/mysqld.cc:4758
        mysql_home_len = 5
        tmp_path = 0x7ffdb9419ec0 "/usr/bin"
        tmp_str = <optimized out>
        len = <optimized out>
        log_errors_to_file = <optimized out>
        session_track_system_variables_check = {m_trackers = {0x100000000, 0x7f0a705b74f9 <check_match+89>, 0x8000000100000001, 0x7ffdb941a0b8, 0x0}}
#14 0x00000000007a72ee in mysqld_main (argc=53, argv=0x32a8a48) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/mysqld.cc:5659
        ho_error = <optimized out>
        retval = <optimized out>
        gtid_ret = <optimized out>
        ret = <optimized out>
        requested_open_files = 5000
        guardize = 4096
#15 0x00007f0a6e496c05 in __libc_start_main (main=0x77a5e0 <main(int, char**)>, argc=15, ubp_av=0x7ffdb941ac88, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdb941ac78) at ../csu/libc-start.c:274
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -260077893095286491, 7972763, 140727711542400, 0, 0, 261057095569217829, 175877992040732965}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x7f0a705bd4c3 <_dl_init+275>,
              0x7f0a707d1150}, data = {prev = 0x0, cleanup = 0x0, canceltype = 1885066435}}}
        not_first_call = <optimized out>
#16 0x000000000079a7c4 in _start ()
No symbol table info available.

Krunal Bauskar (krunal-bauskar) wrote :

wsrep_init that is responsible for galera library initialization has failed. Can you share the log file. It should have a possible reason for failure.

Burnley (burnley) wrote :

Thanks for looking into it. See the mysqld error logs from both nodes for 2017-12-20 to 2017-12-29.

Krunal Bauskar (krunal-bauskar) wrote :

As per the original comment server startup failed. Log files doesn't hint any server startup failure.

* Did you captured the log file during the same time-frame.
* Can you reproduce the issue. If yes, please share the test-case.

Please user following link henceforth.
https://jira.percona.com/browse/PXC-2022

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-2022

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers