mysqld got signal 11

[Brian Leishman]

Something about our server is constantly getting crashes.

Ubuntu 16.04 on AWS EC2
Galera 3.20(r7e383f7)
MySQL 5.7.18-15-57-log

I have the log here

-------------------------

14:15:12 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=23
max_threads=501
thread_count=18
connection_count=16
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 207515 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7ff23c01daa0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff4f824ea70 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xeca47b]
/usr/sbin/mysqld(handle_fatal_signal+0x499)[0x782ab9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7ff51ed9c390]
/usr/sbin/mysqld(_ZNK10Item_field11used_tablesEv+0x19)[0x80da19]
/usr/sbin/mysqld(_ZNK4Item10const_itemEv+0x15)[0x7c2865]
/usr/sbin/mysqld(_ZN19Item_func_concat_ws7val_strEP6String+0x102)[0xb6cca2]
/usr/sbin/mysqld(_ZN14Item_func_trim7val_strEP6String+0x40)[0xb72d90]
/usr/sbin/mysqld(_ZN14Item_func_trim7val_strEP6String+0x40)[0xb72d90]
/usr/sbin/mysqld(_ZN14Item_func_trim7val_strEP6String+0x40)[0xb72d90]
/usr/sbin/mysqld(_ZN14Arg_comparator14compare_stringEv+0x20)[0x82deb0]
/usr/sbin/mysqld(_ZN16Item_func_nullif7val_strEP6String+0x30)[0x82bc20]
/usr/sbin/mysqld(_ZN4Item19save_in_field_innerEP5Fieldb+0x8d)[0x81020d]
/usr/sbin/mysqld(_ZN4Item13save_in_fieldEP5Fieldb+0x15)[0x820285]
/usr/sbin/mysqld(_Z29update_generated_write_fieldsPK9st_bitmapP5TABLE+0x8f)[0xd336ef]
/usr/sbin/mysqld(_Z11fill_recordP3THDP5TABLER4ListI4ItemES6_P9st_bitmapS8_+0x162)[0xc23542]
/usr/sbin/mysqld(_Z36fill_record_n_invoke_before_triggersP3THDR4ListI4ItemES4_P5TABLE23enum_trigger_event_typei+0x80)[0xc23670]
/usr/sbin/mysqld(_Z12mysql_updateP3THDR4ListI4ItemES4_y15enum_duplicatesPyS6_+0x114b)[0xd12f1b]
/usr/sbin/mysqld(_ZN14Sql_cmd_update23try_single_table_updateEP3THDPb+0x1cc)[0xd1573c]
/usr/sbin/mysqld(_ZN14Sql_cmd_update7executeEP3THD+0x36)[0xd15ab6]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x16e4)[0xc802c4]
/usr/sbin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x50)[0xbfcba0]
/usr/sbin/mysqld(_ZN12sp_lex_instr23reset_lex_and_exec_coreEP3THDPjb+0x46c)[0xbfea3c]
/usr/sbin/mysqld(_ZN12sp_lex_instr29validate_lex_and_execute_coreEP3THDPjb+0xba)[0xbff49a]
/usr/sbin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x330)[0xc005b0]
/usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0x53b)[0xbf851b]
/usr/sbin/mysqld(_ZN7sp_head15execute_triggerEP3THDRK25st_mysql_const_lex_stringS4_P10GRANT_INFO+0x25a)[0xbf8f0a]
/usr/sbin/mysqld(_ZN7Trigger7executeEP3THD+0x12f)[0xd3977f]
/usr/sbin/mysqld(_ZN13Trigger_chain16execute_triggersEP3THD+0x18)[0xd3c8b8]
/usr/sbin/mysqld(_ZN24Table_trigger_dispatcher16process_triggersEP3THD23enum_trigger_event_type29enum_trigger_action_time_typeb+0x52)[0xd35b62]
/usr/sbin/mysqld(_Z12mysql_updateP3THDR4ListI4ItemES4_y15enum_duplicatesPyS6_+0x1221)[0xd12ff1]
/usr/sbin/mysqld(_ZN14Sql_cmd_update23try_single_table_updateEP3THDPb+0x1cc)[0xd1573c]
/usr/sbin/mysqld(_ZN14Sql_cmd_update7executeEP3THD+0x36)[0xd15ab6]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x16e4)[0xc802c4]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x66d)[0xc882cd]
/usr/sbin/mysqld[0xc8842b]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1f85)[0xc8adf5]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x27e)[0xc8c05e]
/usr/sbin/mysqld(handle_connection+0x3b8)[0xd57278]
/usr/sbin/mysqld(handle_connection+0x3b8)[0xd57278]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0x126b7c4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7ff51ed926ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ff51e2273dd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7ff23cf141b8): is an invalid pointer
Connection ID (thread ID): 8728
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
Log of wsrep recovery (--wsrep-recover):
2017-07-27T14:15:28.000840Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-07-27T14:15:28.000905Z 0 [Warning] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2017-07-27T14:15:28.001925Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.18-15-57-log) starting as process 14618 ...
2017-07-27T14:15:28.003973Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=ip-172
-31-24-62-bin' to avoid this problem.
2017-07-27T14:15:28.005206Z 0 [Note] InnoDB: PUNCH HOLE support available
2017-07-27T14:15:28.005219Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-07-27T14:15:28.005225Z 0 [Note] InnoDB: Uses event mutexes
2017-07-27T14:15:28.005229Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-07-27T14:15:28.005233Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-07-27T14:15:28.005237Z 0 [Note] InnoDB: Using Linux native AIO
2017-07-27T14:15:28.005617Z 0 [Note] InnoDB: Number of pools: 1
2017-07-27T14:15:28.005733Z 0 [Note] InnoDB: Using CPU crc32 instructions
2017-07-27T14:15:28.008009Z 0 [Note] InnoDB: Initializing buffer pool, total size = 9G, instances = 8, chunk size = 128M
2017-07-27T14:15:28.257339Z 0 [Note] InnoDB: Completed initialization of buffer pool
2017-07-27T14:15:28.318763Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2017-07-27T14:15:28.339004Z 0 [Note] InnoDB: Recovering partial pages from the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
2017-07-27T14:15:28.391505Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2017-07-27T14:15:28.433258Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 19450410673
2017-07-27T14:15:28.495023Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19455653376
2017-07-27T14:15:28.561670Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19460896256
2017-07-27T14:15:28.620811Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19466139136
2017-07-27T14:15:28.689284Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19471382016
2017-07-27T14:15:28.748074Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19476624896
2017-07-27T14:15:28.797094Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19481867776
2017-07-27T14:15:28.839447Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19487110656
2017-07-27T14:15:28.896844Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19492353536
2017-07-27T14:15:28.939504Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19497596416
2017-07-27T14:15:28.988672Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19502839296
2017-07-27T14:15:29.032153Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19508082176
2017-07-27T14:15:29.142124Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19454998016
2017-07-27T14:15:29.214843Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19460240896
2017-07-27T14:15:29.286710Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19465483776
2017-07-27T14:15:29.365828Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19470726656
2017-07-27T14:15:29.439394Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19475969536
2017-07-27T14:15:29.499052Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19481212416
2017-07-27T14:15:29.560015Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19486455296
2017-07-27T14:15:29.637035Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19491698176
2017-07-27T14:15:29.696519Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19496941056
2017-07-27T14:15:29.765000Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19502183936
2017-07-27T14:15:29.825511Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19507426816
2017-07-27T14:15:29.844417Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19508869150
2017-07-27T14:15:29.844449Z 0 [Note] InnoDB: Database was not shutdown normally!
2017-07-27T14:15:29.844456Z 0 [Note] InnoDB: Starting crash recovery.
2017-07-27T14:15:30.914759Z 0 [Note] InnoDB: Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 31457280 bytes
2017-07-27T14:15:31.512861Z 0 [Note] InnoDB: Transaction 19972105 was in the XA prepared state.
2017-07-27T14:15:31.525591Z 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo
2017-07-27T14:15:31.525603Z 0 [Note] InnoDB: Trx id counter is 19972608
2017-07-27T14:15:31.525685Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
2017-07-27T14:15:33.920699Z 0 [Note] InnoDB: Apply batch completed
2017-07-27T14:15:33.920747Z 0 [Note] InnoDB: Last MySQL binlog file position 0 37858379, file name ip-172-31-24-62-bin.000010
2017-07-27T14:15:33.955536Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2017-07-27T14:15:33.955563Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2017-07-27T14:15:33.955537Z 0 [Note] InnoDB: Starting in background the rollback of uncommitted transactions
2017-07-27T14:15:33.955584Z 0 [Note] InnoDB: Rollback of non-prepared transactions completed
2017-07-27T14:15:33.955601Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2017-07-27T14:15:33.995268Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2017-07-27T14:15:33.995923Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2017-07-27T14:15:33.995933Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2017-07-27T14:15:33.996019Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 5678ms. The settings might not be optimal. (flushed=0, during the time.)
2017-07-27T14:15:33.997102Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.18-15 started; log sequence number 19508869150
2017-07-27T14:15:33.997117Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2017-07-27T14:15:33.997369Z 0 [Note] Plugin 'FEDERATED' is disabled.
2017-07-27T14:15:34.002326Z 0 [Note] Recovering after a crash using ip-172-31-24-62-bin
2017-07-27T14:15:34.002345Z 0 [Note] WSREP: Binlog recovery, found wsrep position a148f201-7185-11e7-8dac-5232f5b724bd:9911775
2017-07-27T14:15:34.186975Z 0 [Note] WSREP: Binlog recovery scan stopped at Xid event 9911775
2017-07-27T14:15:34.187002Z 0 [Note] Starting crash recovery...
2017-07-27T14:15:34.187030Z 0 [Note] InnoDB: Starting recovery for XA transactions...
2017-07-27T14:15:34.187039Z 0 [Note] InnoDB: Transaction 19972105 in prepared state after recovery
2017-07-27T14:15:34.187044Z 0 [Note] InnoDB: Transaction contains changes to 1 rows
2017-07-27T14:15:34.187049Z 0 [Note] InnoDB: 1 transactions in prepared state after recovery
2017-07-27T14:15:34.187053Z 0 [Note] Found 1 prepared transaction(s) in InnoDB
2017-07-27T14:15:34.190019Z 0 [Note] Crash recovery finished.
2017-07-27T14:15:34.190090Z 0 [Note] Crashed binlog file ./ip-172-31-24-62-bin.000010 size is 37858949, but recovered up to 37858379. Binlog trimmed to 37858379 bytes.
2017-07-27T14:15:34.198806Z 0 [Note] WSREP: Recovered position: a148f201-7185-11e7-8dac-5232f5b724bd:9911775
2017-07-27T14:15:34.198827Z 0 [Note] Binlog end
2017-07-27T14:15:34.201281Z 0 [Note] Shutting down plugin 'ngram'
2017-07-27T14:15:34.201291Z 0 [Note] Shutting down plugin 'ARCHIVE'
2017-07-27T14:15:34.201294Z 0 [Note] Shutting down plugin 'partition'
2017-07-27T14:15:34.201297Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2017-07-27T14:15:34.201301Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2017-07-27T14:15:34.201347Z 0 [Note] Shutting down plugin 'MyISAM'
2017-07-27T14:15:34.201359Z 0 [Note] Shutting down plugin 'CSV'
2017-07-27T14:15:34.201364Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2017-07-27T14:15:34.201367Z 0 [Note] Shutting down plugin 'MEMORY'
2017-07-27T14:15:34.201371Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2017-07-27T14:15:34.201374Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2017-07-27T14:15:34.201377Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2017-07-27T14:15:34.201379Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2017-07-27T14:15:34.201382Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2017-07-27T14:15:34.201384Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2017-07-27T14:15:34.201387Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2017-07-27T14:15:34.201389Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2017-07-27T14:15:34.201392Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2017-07-27T14:15:34.201394Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2017-07-27T14:15:34.201397Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2017-07-27T14:15:34.201399Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2017-07-27T14:15:34.201402Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2017-07-27T14:15:34.201404Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2017-07-27T14:15:34.201407Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2017-07-27T14:15:34.201409Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2017-07-27T14:15:34.201412Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2017-07-27T14:15:34.201414Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2017-07-27T14:15:34.201417Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2017-07-27T14:15:34.201419Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2017-07-27T14:15:34.201422Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2017-07-27T14:15:34.201424Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2017-07-27T14:15:34.201427Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2017-07-27T14:15:34.201429Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2017-07-27T14:15:34.201432Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2017-07-27T14:15:34.201434Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2017-07-27T14:15:34.201437Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2017-07-27T14:15:34.201457Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2017-07-27T14:15:34.201460Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2017-07-27T14:15:34.201463Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2017-07-27T14:15:34.201465Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2017-07-27T14:15:34.201468Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS'
2017-07-27T14:15:34.201471Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT'
2017-07-27T14:15:34.201473Z 0 [Note] Shutting down plugin 'XTRADB_RSEG'
2017-07-27T14:15:34.201476Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2017-07-27T14:15:34.201478Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2017-07-27T14:15:34.201481Z 0 [Note] Shutting down plugin 'InnoDB'
2017-07-27T14:15:34.201518Z 0 [Note] InnoDB: FTS optimize thread exiting.
2017-07-27T14:15:34.201580Z 0 [Note] InnoDB: Starting shutdown...
2017-07-27T14:15:35.302997Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2017-07-27T14:15:37.922712Z 0 [Note] InnoDB: Shutdown completed; log sequence number 19508876239
2017-07-27T14:15:37.922801Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2017-07-27T14:15:37.922812Z 0 [Note] Shutting down plugin 'sha256_password'
2017-07-27T14:15:37.922826Z 0 [Note] Shutting down plugin 'mysql_native_password'
2017-07-27T14:15:37.922830Z 0 [Note] Shutting down plugin 'wsrep'
2017-07-27T14:15:37.922974Z 0 [Note] Shutting down plugin 'binlog'
2017-07-27T14:15:37.923161Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

-------------------------

And this is a fresh install, so all the default Percona config files are still there, except I removed the wsrep.cnf file in /etc/mysql/percona-xtradb-cluster.conf.d, and I've added the following to /etc/mysql/my.cnf

-------------------------

[client]
default-character-set = utf8mb4

[mysql]
default-character-set = utf8mb4

[mysqld]
sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

max_connections=1024

max-allowed-packet = 16M
max-connect-errors = 1000000

tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 500
thread-cache-size = 50
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048

innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 256M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 9G

log_bin_trust_function_creators=1

wsrep_provider=/usr/lib/libgalera_smm.so

wsrep_cluster_name=pxc-cluster
wsrep_cluster_address=gcomm://ip-1,ip-2,ip-3

wsrep_node_name=pxc1
wsrep_node_address=ip-1

wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=sstuser:passw0rd

pxc_strict_mode=ENFORCING

binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2

-------------------------

Most of the settings were either from the guide for setting up the cluster on Ubuntu, and from the Percona MySQL config wizard

[Brian Leishman (b2ian)]

The Percona package was installed from the source, package percona-xtradb-cluster-57, version 5.7.18-29.20-1.xenial

[Brian Leishman (b2ian)]

# dpkg -l | grep percona
ii percona-release 0.1-4.xenial all Package to install Percona gpg key and APT repo
ii percona-xtrabackup-24 2.4.8-1.xenial amd64 Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-57 5.7.18-29.20-1.xenial amd64 Percona XtraDB Cluster with Galera
ii percona-xtradb-cluster-client-5.7 5.7.18-29.20-1.xenial amd64 Percona XtraDB Cluster database client binaries
ii percona-xtradb-cluster-common-5.7 5.7.18-29.20-1.xenial amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-server-5.7 5.7.18-29.20-1.xenial amd64 Percona XtraDB Cluster database server binaries

[Brian Leishman (b2ian)]

That bin log shows the last query that was run as (not sure if this is the problem query)

------------------------------

### UPDATE `sterling`.`pagerequests`
### WHERE
### @1='077b287a-72de-11e7-afa1-1206bb36a2db' /* STRING(144) meta=65168 nullable=0 is_null=0 */
### @2='3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8' /* STRING(144) meta=65168 nullable=0 is_null=0 */
### @3='71.183.108.253' /* VARSTRING(256) meta=256 nullable=0 is_null=0 */
### @4=NULL /* STRING(144) meta=65168 nullable=1 is_null=1 */
### @5=NULL /* STRING(16) meta=65040 nullable=1 is_null=1 */
### @6=NULL /* STRING(16) meta=65040 nullable=1 is_null=1 */
### @7='pindepot.com' /* VARSTRING(1024) meta=1024 nullable=1 is_null=0 */
### @8='/forgotsent.html?email=drake%40mezco.net' /* BLOB/TEXT meta=2 nullable=1 is_null=0 */
### @9='https://pindepot.com/forgot.html' /* BLOB/TEXT meta=2 nullable=1 is_null=0 */
### @10='Mac' /* VARSTRING(256) meta=256 nullable=1 is_null=0 */
### @11='10.12' /* VARSTRING(256) meta=256 nullable=1 is_null=0 */
### @12='Chrome' /* VARSTRING(256) meta=256 nullable=1 is_null=0 */
### @13='59.0' /* VARSTRING(256) meta=256 nullable=1 is_null=0 */
### @14='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36' /* BLOB/TEXT meta=2 nullable=0 is_null=0 */
### @15=0 /* TINYINT meta=0 nullable=0 is_null=0 */
### @16=1920 /* INT meta=0 nullable=1 is_null=0 */
### @17=1080 /* INT meta=0 nullable=1 is_null=0 */
### @18=0 /* TINYINT meta=0 nullable=1 is_null=0 */
### @19='a26f5174-047e-11e5-8309-3417ebdfde80' /* STRING(144) meta=65168 nullable=1 is_null=0 */
### @20='4a7f318f-0480-11e5-8309-3417ebdfde80' /* STRING(144) meta=65168 nullable=1 is_null=0 */
### @21=1501172310.234800 /* TIMESTAMP(6) meta=6 nullable=0 is_null=0 */
### @22=1501172310.234800 /* TIMESTAMP(6) meta=6 nullable=1 is_null=0 */
### @23=NULL /* STRING(144) meta=65168 nullable=1 is_null=1 */
### @24='2017:07:27' /* DATE meta=0 nullable=1 is_null=0 */
### @25='˃�E\x0c��ᨺ|҅�ϱ��\x16o��Z�Wй�V:$' /* STRING(32) meta=65056 nullable=1 is_null=0 */
### @26=NULL /* VARSTRING(512) meta=512 nullable=1 is_null=1 */
### @27='pindepot.com' /* VARSTRING(1024) meta=1024 nullable=1 is_null=0 */
### SET
### @1='077b287a-72de-11e7-afa1-1206bb36a2db' /* STRING(144) meta=65168 nullable=0 is_null=0 */
### @2='3e55d1bb-d8b6-11e4-b38f-b8ca3a83b4c8' /* STRING(144) meta=65168 nullable=0 is_null=0 */
### @3='71.183.108.253' /* VARSTRING(256) meta=256 nullable=0 is_null=0 */
### @4=NULL /* STRING(144) meta=65168 nullable=1 is_null=1 */
### @5=NULL /* STRING(16) meta=65040 nullable=1 is_null=1 */
### @6=NULL /* STRING(16) meta=65040 nullable=1 is_null=1 */
### @7='pindepot.com' /* VARSTRING(1024) meta=1024 nullable=1 is_null=0 */
### @8='/forgotsent.html?email=drake%40mezco.net' /* BLOB/TEXT meta=2 nullable=1 is_null=0 */
### @9='https://pindepot.com/forgot.html' /* BLOB/TEXT meta=2 nullable=1 is_null=0 */
### @10='Mac' /* VARSTRING(256) meta=256 nullable=1 is_null=0 */
### @11='10.12' /* VARSTRING(256) meta=256 nullable=1 is_null=0 */
### @12='Chrome' /* VARSTRING(256) meta=256 nullable=1 is_...

[Krunal Bauskar (krunal-bauskar)]

There are couple of things to consider here:

a. I see following error message in your comment.

2017-07-27T14:15:28.000905Z 0 [Warning] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.

Which is probably coming from the said setting in my.cnf. I assume some of these setting in my.cnf are manually added at your end and not the default setting shipped by PXC install.
[mysqld]
sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

b. If this is fresh install wondering why is recovery kicking off.

[Brian Leishman (b2ian)]

Yes, I listed those in the initial post as the settings that I added. I have the settings that I've added added to my.cnf listed above, would those settings cause possible crashing? I migrated some rough data from a very old system, and strict mode was causing all sorts of issues, so I added that line manually as a work around.

And it's a completely fresh install with only those config changes and our imported data as of 2017-07-28

[Brian Leishman (b2ian)]

Actually I think I found the bug possibly. I followed the stack trace for every one of the crashes (was almost up to 10 times a day, today included so far) and I noticed that nullif was being called followed by three "trim"s, and I had a generated column on my users table with that exact same function order.

As a potential workaround, I've removed the generated column expression and created triggers to do the updating instead to see if the crashes continue.

That table's creation is as follows:

CREATE TABLE `users` (
  `UserID` char(36) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
  `CompanyID` char(36) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `FirstName` varchar(256) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `LastName` varchar(256) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `Email` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `Phone` varchar(64) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `CompanyName` varchar(256) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `Password` blob,
  `CountryID` char(36) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `StateID` char(36) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `DateTimeAdded` datetime DEFAULT NULL,
  `DateTimeSynced` datetime DEFAULT NULL,
  `DateTimeOnline` datetime DEFAULT NULL,
  `IP` varchar(256) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `LastAccessed` datetime DEFAULT NULL,
  `Deleted` int(1) NOT NULL DEFAULT '0',
  `ListID` varchar(256) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `Emails` int(1) NOT NULL DEFAULT '1',
  `Language` varchar(5) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `LinkID` char(36) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `FacebookAccessToken` varchar(4096) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `FacebookUserID` varchar(256) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `QuotesName` varchar(512) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `QuotesEmails` text COLLATE utf8mb4_unicode_ci,
  `QuotesCCs` text COLLATE utf8mb4_unicode_ci,
  `QuoteMessageID` binary(16) DEFAULT NULL,
  `QuotesMessage` mediumtext COLLATE utf8mb4_unicode_ci,
  `QuotesSubject` mediumtext COLLATE utf8mb4_unicode_ci,
  `Signature` mediumtext COLLATE utf8mb4_unicode_ci,
  `_LatestInvoiceDateTimeAdded` datetime DEFAULT NULL,
  `_InvoiceCount` int(11) NOT NULL DEFAULT '0',
  `_Customer` varchar(512) COLLATE utf8mb4_unicode_ci GENERATED ALWAYS AS (nullif(trim(both ' ' from trim(both '-' from trim(both ' ' from concat_ws('',`CompanyName`,' - ',`FirstName`,' ',`LastName`)))),'')) STORED,
  PRIMARY KEY (`UserID`),
  UNIQUE KEY `UserID_UNIQUE` (`UserID`),
  KEY `fk_users_1_idx` (`CompanyID`),
  KEY `fk_users_2_idx` (`CountryID`),
  KEY `fk_users_LinkID` (`LinkID`),
  KEY `users_getemail_INDEX` (`Email`(191),`_InvoiceCount`,`_LatestInvoiceDateTimeAdded`,`DateTimeAdded`),
  KEY `unify_email_INDEX` (`Email`(191),`UserID`),
  KEY `users_DateTimeOnline` (`DateTimeOnline`,`FirstName`(191),`LastName`(191)),
  KEY `users_StateID` (`StateID`),
  KEY `users_QuoteMessageID_idx` (`QuoteMessageID`),
  KEY `users_Customers` (`Deleted`,`DateTimeAdded`),
  CONSTRAINT `_mlat_fk_users_1` FOREIGN KEY (`CountryID`) REFERENCES `countries` (`CountryID`) ON DELETE NO ACTION ON UPDATE NO ACTION,
  CONSTRAINT `fk_users_...

[Brian Leishman (b2ian)]

Can almost confirm that was the issue. After removing the generated column expression and using triggers on the table to keep the column updated as the work around, we have had 0 crashes.

[Krunal Bauskar (krunal-bauskar)]

Brian,

Good to know you can crack the issue.

It sounds like the issues happens with generated column and likely to be a problem with upstream (either PS or MySQL). Do you have a reduced test-case that we can try on PS or MySQL and accordingly report the issue in upstream bug repo.

[Glafir (glafir)]

I got this error too unexpectedly!

06:00:01 UTC - mysqld got signal 11:
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=4194304
read_buffer_size=131072
max_used_connections=142
max_threads=501
thread_count=90
connection_count=41
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 16459415 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fe4f0a802a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fe4ffffea70 thread_stack 0x400000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xed221b]
/usr/sbin/mysqld(handle_fatal_signal+0x499)[0x783ef9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7feb196cf390]
/usr/sbin/mysqld(_ZN14Arg_comparator13compare_e_intEv+0x26)[0x82d206]
/usr/sbin/mysqld(_ZN15Item_func_equal7val_intEv+0x2b)[0x82d42b]
/usr/sbin/mysqld(_Z19innobase_index_condPv+0x40)[0xef65b0]
/usr/sbin/mysqld[0x75318f]
/usr/sbin/mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x29bf)[0x101f77f]
/usr/sbin/mysqld(_ZN11ha_innobase13general_fetchEPhjj+0x17c)[0xf035ac]
/usr/sbin/mysqld(_ZN7handler18ha_index_next_sameEPhPKhj+0x1a6)[0x804f66]

/usr/sbin/mysqld[0xc533ea]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x13e)[0xc5a1fe]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x3c8)[0xc52d08]
/usr/sbin/mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x233)[0xccba53]
/usr/sbin/mysqld[0x747d10]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x790d)[0xc8ac9d]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x66d)[0xc8c87d]
/usr/sbin/mysqld[0xc8c9db]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1f85)[0xc8f3a5]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x27e)[0xc9060e]
/usr/sbin/mysqld(handle_connection+0x3b8)[0xd5bef8]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0x1274374]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7feb196c56ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7feb18b5a3dd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fe4f00507a0): is an invalid pointer
Connection ID (thread ID): 23349308
Status: NOT_KILLED

[Glafir (glafir)]

mysql -V
mysql Ver 14.14 Distrib 5.7.19-17, for debian-linux-gnu (x86_64) using 6.3

uname -a
uname -a
Linux db00 4.12.4-glafir #19 SMP Sat Aug 12 17:12:05 MSK 2017 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"

dpkg -l |grep mysql
ii libdbd-mysql-perl 4.033-1ubuntu0.1 amd64 Perl5 database interface to the MySQL database
ii libmysqlclient20:amd64 5.7.20-0ubuntu0.16.04.1 amd64 MySQL database client library
ii mysql-client 5.7.20-0ubuntu0.16.04.1 all MySQL database client (metapackage depending on the latest version)
ii mysql-common 5.7.20-0ubuntu0.16.04.1 all MySQL database common files, e.g. /etc/mysql/my.cnf
ii percona-xtradb-cluster-common-5.7 5.7.19-29.22-3.xenial amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1997