Percona-XtraDB Cluster server-57 5.7.19 mysqld failed to stop on "systemctl stop mysql"

Bug #1738910 reported by Burnley
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
New
Undecided
Unassigned

Bug Description

It happened when trying to recover the cluster from a deadlock.

(gdb) bt full
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1 0x00007f0a7039be01 in _L_lock_1022 () from /lib64/libpthread.so.0
No symbol table info available.
#2 0x00007f0a7039bda2 in __GI___pthread_mutex_lock (mutex=0x342a8f0) at ../nptl/pthread_mutex_lock.c:134
        cnt = 11
        max_cnt = 1882850349
        type = 3
        id = <optimized out>
#3 0x00000000007b7b7c in native_mutex_lock (mutex=0x342a8f0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/include/thr_mutex.h:84
No locals.
#4 my_mutex_lock (mp=0x342a8f0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/include/thr_mutex.h:182
No locals.
#5 inline_mysql_mutex_lock (src_line=347, src_file=0x1384630 "/mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos7-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.19-29.22/sql/wsrep_mysqld.cc", that=0x342a8f0)
    at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/include/mysql/psi/mysql_thread.h:715
        locker = 0x0
        state = {m_flags = 3108082433, m_operation = (PSI_MUTEX_TRYLOCK | unknown: 32764), m_mutex = 0x3463160, m_thread = 0x7ffdb9419360, m_timer_start = 12843033556212825344, m_timer = 0x3463178, m_wait = 0x7f0a6327f5ee}
#6 wsrep_pfs_instr_cb (type=<optimized out>, ops=<optimized out>, tag=<optimized out>, value=<optimized out>, alliedvalue=<optimized out>, ts=<optimized out>)
    at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/wsrep_mysqld.cc:347
        mutex = 0x342a8f0
        __PRETTY_FUNCTION__ = "void wsrep_pfs_instr_cb(wsrep_pfs_instr_type_t, wsrep_pfs_instr_ops_t, wsrep_pfs_instr_tag_t, void**, void**, const void*)"
#7 0x00007f0a6325d1f8 in lock (this=0x3423c90) at galerautils/src/gu_mutex.hpp:92
No locals.
#8 Lock (pfs_mtx=..., this=0x7ffdb9419330) at galerautils/src/gu_lock.hpp:106
No locals.
#9 galera::ReplicatorSMM::sst_received (this=0x3423ab0, state_id=..., state=<optimized out>, state_len=<optimized out>, rcode=-125) at galera/src/replicator_str.cpp:74
        __FUNCTION__ = "sst_received"
        lock = {_vptr.Lock = 0x7f0a634ba490 <vtable for gu::Lock+16>, mtx_ = 0x0, pfs_mtx_ = 0x3423c90}
#10 0x00000000007c1591 in wsrep_sst_cancel (call_wsrep_cb=call_wsrep_cb@entry=true) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/wsrep_sst.cc:469
        state_id = {uuid = {data = '\000' <repeats 15 times>}, seqno = -1}
#11 0x00000000007a1bdf in unireg_abort (exit_code=1) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/mysqld.cc:1431
No locals.
#12 0x00000000007be433 in wsrep_init_startup (first=first@entry=true) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/wsrep_mysqld.cc:1189
No locals.
#13 0x00000000007a593f in init_server_components () at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/mysqld.cc:4758
        mysql_home_len = 5
        tmp_path = 0x7ffdb9419ec0 "/usr/bin"
        tmp_str = <optimized out>
        len = <optimized out>
        log_errors_to_file = <optimized out>
        session_track_system_variables_check = {m_trackers = {0x100000000, 0x7f0a705b74f9 <check_match+89>, 0x8000000100000001, 0x7ffdb941a0b8, 0x0}}
#14 0x00000000007a72ee in mysqld_main (argc=53, argv=0x32a8a48) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.19-29.22/sql/mysqld.cc:5659
        ho_error = <optimized out>
        retval = <optimized out>
        gtid_ret = <optimized out>
        ret = <optimized out>
        requested_open_files = 5000
        guardize = 4096
#15 0x00007f0a6e496c05 in __libc_start_main (main=0x77a5e0 <main(int, char**)>, argc=15, ubp_av=0x7ffdb941ac88, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdb941ac78) at ../csu/libc-start.c:274
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -260077893095286491, 7972763, 140727711542400, 0, 0, 261057095569217829, 175877992040732965}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x7f0a705bd4c3 <_dl_init+275>,
              0x7f0a707d1150}, data = {prev = 0x0, cleanup = 0x0, canceltype = 1885066435}}}
        not_first_call = <optimized out>
#16 0x000000000079a7c4 in _start ()
No symbol table info available.

Revision history for this message
Krunal Bauskar (krunal-bauskar) wrote :

wsrep_init that is responsible for galera library initialization has failed. Can you share the log file. It should have a possible reason for failure.

Revision history for this message
Burnley (burnley) wrote :

Thanks for looking into it. See the mysqld error logs from both nodes for 2017-12-20 to 2017-12-29.

Revision history for this message
Krunal Bauskar (krunal-bauskar) wrote :

As per the original comment server startup failed. Log files doesn't hint any server startup failure.

* Did you captured the log file during the same time-frame.
* Can you reproduce the issue. If yes, please share the test-case.

Please user following link henceforth.
https://jira.percona.com/browse/PXC-2022

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-2022

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.