Comment 0 for bug 1549704

Description:

After some period of stable work Percona XtraDB Cluster nodes starts crashing and can't join cluster (no matter IST/SST).
The only way to restore cluster is to stop all nodes and rebootstrap.

Reproducibility: 100% (takes about 15-20 days to first crash).

Steps to reproduce (reproducibility - 100%):

1. Bootstrap cluster
2. Wait for 15-20 days
3. Some XtraDB node will crash
4. Try to join cluster (i.e. systemctl start mysql).

Actual results:

Crash and inability to join cluster.

2016-02-05 10:42:27 6383 [Warning] WSREP: 1.0 (mysql-rw0): State transfer to 0.0 (mysql-rw1) failed: -12 (Cannot allocate memory)
2016-02-05 10:42:27 6383 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():731: Will never receive state. Need to abort.

Expected results:

JOINER->SYNCED

Additional info:

I've tried to enable core dumping but it seems that galera disables it in sources so I've tried to run mysqld manually (i.e. without mysqld_safe wrapper) with gdb attached to catch fault and get bt.

# rpm -qa | grep Percona
Percona-XtraDB-Cluster-server-56-5.6.28-25.14.1.el7.x86_64
Percona-XtraDB-Cluster-devel-56-5.6.28-25.14.1.el7.x86_64
Percona-XtraDB-Cluster-client-56-5.6.28-25.14.1.el7.x86_64
Percona-XtraDB-Cluster-test-56-5.6.28-25.14.1.el7.x86_64
Percona-XtraDB-Cluster-shared-56-5.6.28-25.14.1.el7.x86_64
Percona-XtraDB-Cluster-galera-3-debuginfo-3.14-1.rhel7.x86_64
Percona-XtraDB-Cluster-full-56-5.6.28-25.14.1.el7.x86_64
Percona-XtraDB-Cluster-galera-3-3.14-1.rhel7.x86_64
Percona-XtraDB-Cluster-garbd-3-3.14-1.rhel7.x86_64
Percona-XtraDB-Cluster-56-debuginfo-5.6.28-25.14.1.el7.x86_64

# cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)

# hostnamectl
   Static hostname: mysql-rw2
           Chassis: container
    Virtualization: lxc-libvirt
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-229.20.1.el7.x86_64
      Architecture: x86-64