percona crashes about once per day

[malte]

Here is the contents of mysql/error.log when this happens:

----------------------------------------------------------------------------------------------

04:47:00 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=81
max_threads=402
thread_count=63
connection_count=63
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 176835 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f80b6a83000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f8094d60dc0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x7f8bbc4fa04c]
/usr/sbin/mysqld(handle_fatal_signal+0x3cb)[0x7f8bbc240a2b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f8bba4b7340]
/usr/sbin/mysqld(+0x6e4cd7)[0x7f8bbc4a4cd7]
/usr/sbin/mysqld(+0x3a8a5a)[0x7f8bbc168a5a]
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x1dd)[0x7f8bbc2e4b8d]
/usr/sbin/mysqld(_Z28ha_start_consistent_snapshotP3THD+0x281)[0x7f8bbc16da51]
/usr/sbin/mysqld(_Z11trans_beginP3THDj+0x100)[0x7f8bbc36b910]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2b9b)[0x7f8bbc2d40fb]
/usr/sbin/mysqld(+0x519ee8)[0x7f8bbc2d9ee8]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1280)[0x7f8bbc2db680]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x16c)[0x7f8bbc2dd8fc]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x31d)[0x7f8bbc29ed3d]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x7f8bbc29ede0]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0x7f8bbc772fb0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f8bba4af182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f8bb99bc38d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f80b902e010): is an invalid pointer
Connection ID (thread ID): 157893
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
140827 22:47:02 mysqld_safe Number of processes running now: 0
140827 22:47:02 mysqld_safe mysqld restarted
140827 22:47:02 mysqld_safe Skipping wsrep-recover for empty datadir: /var/lib/mysql
140827 22:47:02 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
2014-08-27 22:47:03 0 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
2014-08-27 22:47:03 0 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
2014-08-27 22:47:03 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-08-27 22:47:03 39649 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
2014-08-27 22:47:03 39649 [Note] Plugin 'FEDERATED' is disabled.
2014-08-27 22:47:03 39649 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-08-27 22:47:03 39649 [Note] InnoDB: The InnoDB memory heap is disabled
2014-08-27 22:47:03 39649 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-08-27 22:47:03 39649 [Note] InnoDB: Compressed tables use zlib 1.2.8
2014-08-27 22:47:03 39649 [Note] InnoDB: Using Linux native AIO
2014-08-27 22:47:03 39649 [Note] InnoDB: Using CPU crc32 instructions
2014-08-27 22:47:03 39649 [Note] InnoDB: Initializing buffer pool, size = 40.0G
2014-08-27 22:47:04 39649 [Note] InnoDB: Completed initialization of buffer pool
2014-08-27 22:47:04 39649 [Note] InnoDB: Highest supported file format is Barracuda.
2014-08-27 22:47:04 39649 [Note] InnoDB: Log scan progressed past the checkpoint lsn 1595515217651
2014-08-27 22:47:04 39649 [Note] InnoDB: Database was not shutdown normally!
2014-08-27 22:47:04 39649 [Note] InnoDB: Starting crash recovery.
2014-08-27 22:47:04 39649 [Note] InnoDB: Reading tablespace information from the .ibd files...
2014-08-27 22:47:05 39649 [Note] InnoDB: Restoring possible half-written data pages
2014-08-27 22:47:05 39649 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 1595520460288
InnoDB: Doing recovery: scanned up to log sequence number 1595525703168
InnoDB: Doing recovery: scanned up to log sequence number 1595530946048
InnoDB: Doing recovery: scanned up to log sequence number 1595536188928
InnoDB: Doing recovery: scanned up to log sequence number 1595541431808
InnoDB: Doing recovery: scanned up to log sequence number 1595546674688
InnoDB: Doing recovery: scanned up to log sequence number 1595551917568
InnoDB: Doing recovery: scanned up to log sequence number 1595557160448
InnoDB: Doing recovery: scanned up to log sequence number 1595562403328
InnoDB: Doing recovery: scanned up to log sequence number 1595567646208
InnoDB: Doing recovery: scanned up to log sequence number 1595572889088
InnoDB: Doing recovery: scanned up to log sequence number 1595578131968
InnoDB: Doing recovery: scanned up to log sequence number 1595583374848
InnoDB: Doing recovery: scanned up to log sequence number 1595588617728
InnoDB: Doing recovery: scanned up to log sequence number 1595593860608
InnoDB: Doing recovery: scanned up to log sequence number 1595599103488
InnoDB: Doing recovery: scanned up to log sequence number 1595604346368
InnoDB: Doing recovery: scanned up to log sequence number 1595609589248
InnoDB: Doing recovery: scanned up to log sequence number 1595614832128
InnoDB: Doing recovery: scanned up to log sequence number 1595620075008
InnoDB: Doing recovery: scanned up to log sequence number 1595625317888
InnoDB: Doing recovery: scanned up to log sequence number 1595630560768
InnoDB: Doing recovery: scanned up to log sequence number 1595635803648
InnoDB: Doing recovery: scanned up to log sequence number 1595641046528
InnoDB: Doing recovery: scanned up to log sequence number 1595646289408
InnoDB: Doing recovery: scanned up to log sequence number 1595651532288
InnoDB: Doing recovery: scanned up to log sequence number 1595656775168
InnoDB: Doing recovery: scanned up to log sequence number 1595662018048
InnoDB: Doing recovery: scanned up to log sequence number 1595667260928
InnoDB: Doing recovery: scanned up to log sequence number 1595672503808
InnoDB: Doing recovery: scanned up to log sequence number 1595677746688
InnoDB: Doing recovery: scanned up to log sequence number 1595682989568
InnoDB: Doing recovery: scanned up to log sequence number 1595684086129
2014-08-27 22:47:08 39649 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2014-08-27 22:47:11 39649 [Note] InnoDB: 128 rollback segment(s) are active.
2014-08-27 22:47:12 39649 [Note] InnoDB: Waiting for purge to start
2014-08-27 22:47:12 39649 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.19-67.0 started; log sequence number 1595684086129
2014-08-27 22:47:12 39649 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2014-08-27 22:47:12 39649 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2014-08-27 22:47:12 39649 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
2014-08-27 22:47:12 39649 [Note] - '0.0.0.0' resolves to '0.0.0.0';
2014-08-27 22:47:12 39649 [Note] Server socket created on IP: '0.0.0.0'.
2014-08-27 22:47:12 39649 [Note] Event Scheduler: Loaded 0 events
2014-08-27 22:47:12 39649 [Note] WSREP: Read nil XID from storage engines, skipping position init
2014-08-27 22:47:12 39649 [Note] WSREP: wsrep_load(): loading provider library 'none'
2014-08-27 22:47:12 39649 [Note] [Debug] WSREP: dummy_init
2014-08-27 22:47:12 39649 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.19-67.0-56' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), Release 25.6, wsrep_25.6.r4111

----------------------------------------------------------------------------------------------

I don't know what causes this crash. It appears to happen randomly throughout the day.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@malte,

The fix is available in percona-testing repository.

For CentOS: yum install http://repo.percona.com/testing/centos/7/os/noarch/percona-testing-0.0-1.noarch.rpm

[malte (alexander-polkabrothers)]

Thank you for pointing that out for me. I'm on Ubuntu however, but I see all I have to do is enable the binlog to work around this problem, so I'll do that for now.