InnoDB: Assertion failure in thread 140348638594816 in file ut0ut.cc line 936

[monty solomon]

After restarting PS today it continually crashes and restarts

2017-01-03T22:39:10.779268Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.13-6-log' socket: '/opt/mysql/dbprod/biden-q0.sock' port: 3306 Percona Server (GPL), Release 6, Revision e3d58bb
2017-01-03T22:39:11.455243Z 5 [Note] Access denied for user 'UNKNOWN_MYSQL_USER'@'localhost' (using password: YES)
2017-01-03T22:39:17.371279Z 19 [Note] Start binlog_dump to master_thread_id(19) slave_server(3255324397), pos(, 4)
2017-01-03T22:40:10.581217Z 0 [ERROR] InnoDB: Cannot allocate 18446744073709546280 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
2017-01-03 22:40:10 0x7fa576bf8700 InnoDB: Assertion failure in thread 140348638594816 in file ut0ut.cc line 936
InnoDB: Failing assertion: !m_fatal
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
22:40:10 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=16
max_threads=501
thread_count=14
connection_count=14
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 712565 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fa59140e000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa576bf7ca0 thread_stack 0x40000
2017-01-03T22:40:10.582165Z 0 [ERROR] InnoDB: Cannot allocate 18446744073709542952 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
2017-01-03 22:40:10 0x7fa577bfa700 InnoDB: Assertion failure in thread 140348655380224 in file ut0ut.cc line 936
InnoDB: Failing assertion: !m_fatal
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2017-01-03T22:40:10.876759Z mysqld_safe Number of processes running now: 0
2017-01-03T22:40:10.878021Z mysqld_safe mysqld restarted

[monty solomon (monty+launchpad)]

Linux free-mule 3.18.36-234.el6.x86_64 #1 SMP Tue Jul 5 18:21:45 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

CentOS release 6.7 (Final)

mysql Ver 14.14 Distrib 5.7.13-6, for Linux (x86_64) using 6.0

Server version: 5.7.13-6-log Percona Server (GPL), Release 6, Revision e3d58bb

# rpm -qa | grep Percona-Server
Percona-Server-server-57-5.7.13-6.1.el6.x86_64
Percona-Server-client-57-5.7.13-6.1.el6.x86_64
Percona-Server-devel-57-5.7.13-6.1.el6.x86_64
Percona-Server-shared-57-5.7.13-6.1.el6.x86_64

[monty solomon (monty+launchpad)]

2017-01-03T22:42:17.135231Z 0 [ERROR] InnoDB: Cannot allocate 18446744073709549656 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase t
he swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
2017-01-03 22:42:17 0x7fde367fa700 InnoDB: Assertion failure in thread 140592373802752 in file ut0ut.cc line 936
InnoDB: Failing assertion: !m_fatal
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
22:42:17 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=11
max_threads=501
thread_count=10
connection_count=10
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 712565 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fde50c0e000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fde367f9ca0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xec501c]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x79d141]
/lib64/libpthread.so.0(+0xf7e0)[0x7fe12bd017e0]
/lib64/libc.so.6(gsignal+0x35)[0x7fe129f3a625]
/lib64/libc.so.6(abort+0x175)[0x7fe129f3be05]
/usr/sbin/mysqld[0x76bd86]
/usr/sbin/mysqld(_ZN2ib14fatal_or_errorD2Ev+0x2d3)[0x114c413]
/usr/sbin/mysqld(_Z26mem_heap_create_block_funcP16mem_block_info_tmm+0x41c)[0x1027eec]
/usr/sbin/mysqld(_Z18mem_heap_add_blockP16mem_block_info_tm+0x4c)[0x102825c]
/usr/sbin/mysqld(_Z12mem_heap_dupP16mem_block_info_tPKvm+0x4c)[0x102840c]
/usr/sbin/mysqld(_Z25trx_undo_get_undo_rec_lowmP16mem_block_info_tb+0x1c8)[0x111dab8]
/usr/sbin/mysqld(_Z27trx_undo_prev_version_buildPKhP5mtr_tS0_P12dict_index_tPmP16mem_block_info_tPPhS7_PPK8dtuple_tm+0x2f3)[0x111df43]
/usr/sbin/mysqld[0x10db8bd]
/usr/sbin/mysqld(_Z28row_vers_old_has_index_entrymPKhP5mtr_tP12dict_index_tPK8dtuple_tmm+0x1059)[0x10dd2c9]
/usr/sbin/mysqld(_Z18row_purge_poss_secP12purge_node_tP12dict_index_tPK8dtuple_t+0x16f)[0x10aee9f]
/usr/sbin/mysqld(_Z27btr_cur_search_to_nth_levelP12dict_index_tmPK8dtuple_t15page_cur_mode_tmP9btr_cur_tmPKcmP5mtr_t+0x1ea9)[0x116c579]
/usr/sbin/mysqld(_Z22row_search_i...

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3624