Active Cluster node crashes sometimes on ALTER TABLE

[Jochen]

Updating our bitbocket isntallation, the active node (the node bitbucket connected to) crash with the following stacktrace. The problem is that it finished on the second run and I found that it has happened before:

first stacktrace:
stack_bottom = 7ef069dc1e58 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x91767b]
/usr/sbin/mysqld(handle_fatal_signal+0x378)[0x6633b8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7f0a1dfc23d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f0a1d37c418]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f0a1d37e01a]
/usr/sbin/mysqld[0xa7b7b1]
/usr/sbin/mysqld[0x9d66de]
/usr/sbin/mysqld(_Z15ha_delete_tableP3THDP10handlertonPKcS4_S4_b+0x15f)[0x5a0eef]
/usr/sbin/mysqld(_Z14quick_rm_tableP3THDP10handlertonPKcS4_j+0x128)[0x7483e8]
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P24st_ha_create_informationP10TABLE_LISTP10Alter_infojP8st_orderb+0x2daf)[0x7529ef]
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x54c)[0x844a4c]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2ea1)[0x6f45d1]
/usr/sbin/mysqld[0x6fa7ac]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x172f)[0x6fc3df]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x2aa)[0x6fdf9a]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x32d)[0x6be3ad]
/usr/sbin/mysqld(handle_one_connection+0x39)[0x6be419]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0x9b0c30]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa)[0x7f0a1dfb86fa]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f0a1d44db5d]

second stacktrace:
stack_bottom = 7ef069dc1e58 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x91767b]
/usr/sbin/mysqld(handle_fatal_signal+0x378)[0x6633b8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7f0a1dfc23d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f0a1d37c418]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f0a1d37e01a]
/usr/sbin/mysqld[0xa7b7b1]
/usr/sbin/mysqld[0x9d66de]
/usr/sbin/mysqld(_Z15ha_delete_tableP3THDP10handlertonPKcS4_S4_b+0x15f)[0x5a0eef]
/usr/sbin/mysqld(_Z14quick_rm_tableP3THDP10handlertonPKcS4_j+0x128)[0x7483e8]
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P24st_ha_create_informationP10TABLE_LISTP10Alter_infojP8st_orderb+0x2daf)[0x7529ef]
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x54c)[0x844a4c]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2ea1)[0x6f45d1]
/usr/sbin/mysqld[0x6fa7ac]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x172f)[0x6fc3df]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x2aa)[0x6fdf9a]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x32d)[0x6be3ad]
/usr/sbin/mysqld(handle_one_connection+0x39)[0x6be419]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0x9b0c30]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa)[0x7f0a1dfb86fa]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f0a1d44db5d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7eed7406da60): is an invalid pointer
Connection ID (thread ID): 316025
Status: NOT_KILLED

We are running:
percona-xtradb-cluster-server-5.6: 5.6.21-25.8-0ubuntu3
- Percona XtraDB (http://www.percona.com) 5.6.21-70.1
percona-galera-3: 3.8-3390-0ubuntu6
- wsrep_25.8.rXXXX

[Krunal Bauskar (krunal-bauskar)]

It is bit difficult to say what is going wrong from given stack trace.
It would be good if you can attach complete log file.
(Is ALTER running as RSU or TOI ?)

Also, it seems like you are using quite-old version of PXC. Latest one is 5.6.30 and in
meantime quite some important bugs has been fixed.
You may want to try upgrade.

[Jochen (jradmacher)]

Full mysql error.log

[Krunal Bauskar (krunal-bauskar)]

I think you are hitting "https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1282707".

Which is fixed as per the record in following revision "Percona XtraDB Cluster 5.6.22-25.8"

I strongly recommend you upgrade to PXC-5.6.30.

[Jochen (jradmacher)]

>I strongly recommend you upgrade to PXC-5.6.30.
I'll try this.

The Version I'm running is from the default ubuntu repository. I'm not sure, if the patch should be backported.

# apt-cache policy percona-xtradb-cluster-server-5.6
percona-xtradb-cluster-server-5.6:
  Installed: 5.6.21-25.8-0ubuntu3
  Candidate: 5.6.21-25.8-0ubuntu3
  Version table:
 *** 5.6.21-25.8-0ubuntu3 500
        500 http://de.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        100 /var/lib/dpkg/status