MySQL crashed with: double free or corruption (!prev): 0x00007fec0b86ade0

[Arty]

3 days ago (Jun, 6, 2017) I updated MySQL from version Percona Server 5.6.23-72.1 to version 5.7.18-15.1 (I also also ran system tables upgrade). OS Oracle Linux 7.2/x86_64.
The server is a master in master-slave pair. We use native 5.7 partitioning (it was the main goal of upgrade), and we use row_format=compressed for table with 0.5bln rows. The server was under heavy load (LA=12, 0% CPU IDLE, 2xCPU total)

Today server crashed with the following message (more details in attached file):

2017-06-09T01:00:00.101363Z 476565 [Note] Aborted connection 476565 to db: '***' user: '***' host: '192.168.*.*' (Got timeout reading communication packets)
*** Error in `/usr/sbin/mysqld': double free or corruption (!prev): 0x00007fec0b86ade0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7d053)[0x7fedc03ed053]
/usr/sbin/mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0xaa4)[0x11098c4]
/usr/sbin/mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x316)[0xff4326]
/usr/sbin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x248)[0x80bc58]
/usr/sbin/mysqld(_ZN7handler16read_range_firstEPK12st_key_rangeS2_bb+0x5c)[0x80c40c]
/usr/sbin/mysqld(_ZN7handler21multi_range_read_nextEPPc+0x99)[0x7ff059]
/usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x5a)[0xe0605a]
/usr/sbin/mysqld[0xc3747a]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x11b)[0xca8d8b]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x267)[0xca1967]
/usr/sbin/mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x17d)[0xd13f5d]
/usr/sbin/mysqld[0x75f7c1]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x482e)[0xcd518e]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x60d)[0xcd84dd]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xaba)[0xcd902a]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1df)[0xcdaabf]
/usr/sbin/mysqld(handle_connection+0x2b8)[0xda2f28]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xf22a94]
/lib64/libpthread.so.0(+0x7dc5)[0x7fedc2213dc5]
/lib64/libc.so.6(clone+0x6d)[0x7fedc0466c4d]

here is my.cnf:
[mysqld]
pid-file=/opt/mysql/mysqld.pid
socket = /var/lib/mysql/mysql.sock
max_connections = 500
character-set-server = utf8
skip-name-resolve
skip-slave-start
datadir = /opt/mysql
log-error = /var/log/mysqld.log
slow-query-log = 1
slow-query-log-file = /var/log/mysql-slow.log
max_slowlog_files = 5
max_slowlog_size = 10000000
long-query-time = 2
innodb_file_per_table
innodb_buffer_pool_size = 4500M
innodb_buffer_pool_load_at_startup = on
innodb_buffer_pool_dump_at_shutdown = on
innodb_file_format = Barracuda
userstat = 1
server-id=1
log-bin = /opt/binlogs/ismeta-binlog
expire_logs_days = 2
log-slave-updates
relay-log = mysqld-relay-bin
sync_binlog = 0
skip-slave-start
performance_schema = 0
innodb_empty_free_list_algorithm = legacy
event_scheduler = on

[Arty (nopius)]

Today 24.06.2017, we hit it again. MySQL crashed with the same bug:

2017-06-23T11:31:26.156921Z 2541990 [Note] Aborted connection 2541990 to db: 'xxxxx' user: 'xxxx' host: '192.168.xx.xx' (Got timeout reading communication packets)

*** Error in `/usr/sbin/mysqld': double free or corruption (!prev): 0x00007fb1a0007880 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7d053)[0x7fb3a34eb053]
/usr/sbin/mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0xaa4)[0x11098c4]
/usr/sbin/mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x316)[0xff4326]
/usr/sbin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x248)[0x80bc58]
/usr/sbin/mysqld(_ZN7handler16read_range_firstEPK12st_key_rangeS2_bb+0x5c)[0x80c40c]
/usr/sbin/mysqld(_ZN7handler21multi_range_read_nextEPPc+0x99)[0x7ff059]
/usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x5a)[0xe0605a]
/usr/sbin/mysqld[0xc3747a]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x11b)[0xca8d8b]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x267)[0xca1967]
/usr/sbin/mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x17d)[0xd13f5d]
/usr/sbin/mysqld[0x75f7c1]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x482e)[0xcd518e]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x60d)[0xcd84dd]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xaba)[0xcd902a]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1df)[0xcdaabf]
/usr/sbin/mysqld(handle_connection+0x2b8)[0xda2f28]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xf22a94]
/lib64/libpthread.so.0(+0x7dc5)[0x7fb3a5311dc5]
/lib64/libc.so.6(clone+0x6d)[0x7fb3a3564c4d]
======= Memory map: ========

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3709