Terminate called after throwing an instance of 'std::out_of_range'

[Dick Barton]

We experienced this crash in version: 5.6.35-80.0-log Percona Server (GPL), Release 80.0, Revision f113994f31

As you can see the crash was recoverable, but it did cause "MySQL server has gone away" errors in our applications.

terminate called after throwing an instance of 'std::out_of_range'
  what(): vector::_M_range_check
00:49:01 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=44
max_threads=502
thread_count=19
connection_count=19
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 265267 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8d7fdc]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x65a441]
/lib64/libpthread.so.0[0x39b840f7e0]
/lib64/libc.so.6(gsignal+0x35)[0x39b80325e5]
/lib64/libc.so.6(abort+0x175)[0x39b8033dc5]
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x39ba0bea7d]
/usr/lib64/libstdc++.so.6[0x39ba0bcbd6]
/usr/lib64/libstdc++.so.6[0x39ba0bcc03]
/usr/lib64/libstdc++.so.6[0x39ba0bcd22]
/usr/lib64/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x67)[0x39ba061db7]
/usr/sbin/mysqld[0xaf2920]
/usr/sbin/mysqld[0xaf6653]
/usr/sbin/mysqld[0xaf8695]
/usr/sbin/mysqld[0xaf9a80]
/lib64/libpthread.so.0[0x39b8407aa1]
/lib64/libc.so.6(clone+0x6d)[0x39b80e8aad]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
2017-02-14 00:49:03 53926 [Note] Plugin 'FEDERATED' is disabled.
2017-02-14 00:49:04 53926 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-02-14 00:49:04 53926 [Note] InnoDB: The InnoDB memory heap is disabled
2017-02-14 00:49:04 53926 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-02-14 00:49:04 53926 [Note] InnoDB: Memory barrier is not used
2017-02-14 00:49:04 53926 [Note] InnoDB: Compressed tables use zlib 1.2.3
2017-02-14 00:49:04 53926 [Note] InnoDB: Using Linux native AIO
2017-02-14 00:49:04 53926 [Note] InnoDB: Using CPU crc32 instructions
2017-02-14 00:49:04 53926 [Note] InnoDB: Initializing buffer pool, size = 4.9G
2017-02-14 00:49:04 53926 [Note] InnoDB: Completed initialization of buffer pool
2017-02-14 00:49:04 53926 [Note] InnoDB: Highest supported file format is Barracuda.
2017-02-14 00:49:04 53926 [Note] InnoDB: Log scan progressed past the checkpoint lsn 1858790035804
2017-02-14 00:49:04 53926 [Note] InnoDB: Database was not shutdown normally!
2017-02-14 00:49:04 53926 [Note] InnoDB: Starting crash recovery.
2017-02-14 00:49:04 53926 [Note] InnoDB: Reading tablespace information from the .ibd files...
2017-02-14 00:49:10 53926 [Note] InnoDB: Restoring possible half-written data pages
2017-02-14 00:49:10 53926 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 1858795278336
InnoDB: Doing recovery: scanned up to log sequence number 1858800521216
InnoDB: Doing recovery: scanned up to log sequence number 1858805764096
InnoDB: Doing recovery: scanned up to log sequence number 1858811006976
InnoDB: Doing recovery: scanned up to log sequence number 1858816249856
InnoDB: Doing recovery: scanned up to log sequence number 1858821492736
InnoDB: Doing recovery: scanned up to log sequence number 1858826735616
InnoDB: Doing recovery: scanned up to log sequence number 1858828902010
InnoDB: Transaction 6361926761 was in the XA prepared state.
InnoDB: 3 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 2 row operations to undo
InnoDB: Trx id counter is 6361927168
2017-02-14 00:49:13 53926 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 880999370, file name 665515-bin-log.000343
2017-02-14 00:49:32 53926 [Note] InnoDB: 128 rollback segment(s) are active.
InnoDB: Starting in background the rollback of uncommitted transactions
2017-02-14 00:49:32 7fb0037d8700 InnoDB: Rolling back trx with id 6361926762, 1 rows to undo
2017-02-14 00:49:32 53926 [Note] InnoDB: Waiting for purge to start
2017-02-14 00:49:32 53926 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.35-80.0 started; log sequence number 1858828902010
2017-02-14 00:49:32 53926 [Note] Recovering after a crash using /data/mysqllogs/665515-bin-log
2017-02-14 00:49:32 53926 [Note] InnoDB: Rollback of trx with id 6361926762 completed
2017-02-14 00:49:32 7fb0037d8700 InnoDB: Rolling back trx with id 6361926701, 1 rows to undo
2017-02-14 00:49:32 53926 [Note] InnoDB: Rollback of trx with id 6361926701 completed
2017-02-14 00:49:32 7fb0037d8700 InnoDB: Rollback of non-prepared transactions completed
2017-02-14 00:49:48 53926 [Note] Starting crash recovery...
2017-02-14 00:49:48 7fb1914517e0 InnoDB: Starting recovery for XA transactions...
2017-02-14 00:49:48 7fb1914517e0 InnoDB: Transaction 6361926761 in prepared state after recovery
2017-02-14 00:49:48 7fb1914517e0 InnoDB: Transaction contains changes to 1 rows
2017-02-14 00:49:48 7fb1914517e0 InnoDB: 1 transactions in prepared state after recovery

[Patrick Kane (pmk-0)]

We are seeing a similar crash with Percona-Server-shared-56-5.6.35-rel80.0.el6.x86_64. It appears to be a regression in this release.

In our case, the crash is repetitive -- it reoccurs on every server startup.

Crash details are below. We are going to revert to Percona-Server-5.6.34-79.1.

2017-02-17 04:48:46 4549 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.35-80.0-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 80.0, Revision f113994f31
terminate called after throwing an instance of 'std::out_of_range'
  what(): vector::_M_range_check
15:59:02 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=10
max_threads=4002
thread_count=10
connection_count=8
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1600567 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8d7fdc]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x65a441]
/lib64/libpthread.so.0(+0xf370)[0x7f9bb0721370]
/lib64/libc.so.6(gsignal+0x37)[0x7f9bae8b21d7]
/lib64/libc.so.6(abort+0x148)[0x7f9bae8b38c8]
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155)[0x7f9baf1b7515]
/usr/lib64/libstdc++.so.6(+0x5e6a6)[0x7f9baf1b56a6]
/usr/lib64/libstdc++.so.6(+0x5e6d3)[0x7f9baf1b56d3]
/usr/lib64/libstdc++.so.6(+0x5e8f3)[0x7f9baf1b58f3]
/usr/lib64/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x67)[0x7f9baf207557]
/usr/sbin/mysqld[0xaf2920]
/usr/sbin/mysqld[0xaf6653]
/usr/sbin/mysqld[0xaf8695]
/usr/sbin/mysqld[0xaf9a80]
/lib64/libpthread.so.0(+0x7dc5)[0x7f9bb0719dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f9bae9746ed]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

[Patrick Kane (pmk-0)]

Confirming that Percona-Server-5.6.34-79.1 resolved the issue.

[Nikita Beletskii (nikita-beletskii)]

We have the same error in 5.7.17-11.

[O. (bugreports-m)]

we observe the same error every 1..4 days since the upgrade from 5.6.34-79 to 5.6.35-80.0:

 terminate called after throwing an instance of 'std::out_of_range'
   what(): vector::_M_range_check
 12:27:09 UTC - mysqld got signal 6 ;
 This could be because you hit a bug. It is also possible that this binary
 or one of the libraries it was linked against is corrupt, improperly built,
 or misconfigured. This error can also be caused by malfunctioning hardware.
 We will try our best to scrape up some info that will hopefully help
 diagnose the problem, but since we have already crashed,
 something is definitely wrong and this may fail.
 Please help us make Percona Server better by reporting any
 bugs at http://bugs.percona.com/

 key_buffer_size=268435456
 read_buffer_size=2097152
 max_used_connections=186
 max_threads=1252
 thread_count=19
 connection_count=17
 It is possible that mysqld could use up to
 key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 13100010 K bytes of memory
 Hope that's ok; if not, decrease some variables in the equation.

 Thread pointer: 0x0
 Attempting backtrace. You can use the following information to find out
 where mysqld died. If you see no messages after this, something went
 terribly wrong...
 stack_bottom = 0 thread_stack 0x80000
 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8c0dce]
 /usr/sbin/mysqld(handle_fatal_signal+0x491)[0x682ee1]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7fa80b3f10a0]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fa80941b125]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7fa80941e3a0]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d)[0x7fa809c7389d]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x63996)[0x7fa809c71996]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x639c3)[0x7fa809c719c3]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x63bee)[0x7fa809c71bee]
 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x5d)[0x7fa809cc36fd]
 /usr/sbin/mysqld[0xa5677e]
 /usr/sbin/mysqld[0xa57da7]
 /usr/sbin/mysqld[0xa59448]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7fa80b3e8b50]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fa8094c6fbd]
 You may download the Percona Server operations manual by visiting
 http://www.percona.com/software/percona-server/. You may find information
 in the manual which will help you identify the cause of the crash.

[O. (bugreports-m)]

see upstream bugreport: https://bugs.mysql.com/bug.php?id=84940

[Dan Rogers (drogers-l)]

We're also seeing this. Is there an ETA on a fix?

[Laurynas Biveinis (laurynas-biveinis)]

Oracle claims to have this fixed in their upcoming 5.6.36 release, which should be imminent, and a Percona Server release will happen relatively shortly after upstream release.

[Laurynas Biveinis (laurynas-biveinis)]

https://github.com/percona/percona-server/pull/1678

[Tom Sommer (tomsommer)]

What about 5.7?

[Laurynas Biveinis (laurynas-biveinis)]

5.7 is in progress

[Laurynas Biveinis (laurynas-biveinis)]

https://github.com/percona/percona-server/pull/1695

[Edward Hibbert (edward-g)]

I'm seeing this every day or two on a production system running 5.7.17-29. It seems to result in the whole cluster coming down (either with the same crash or because the nodes kill themselves as the cluster partitions; I may not be using the right terminology).

I'm wondering whether to risk upgrading to the testing release, if the probably fix has been backported. Is that a good idea or a terrible idea?

[Edward Hibbert (edward-g)]

There's only limited info in the upstream bug because that is a dup of one which is hidden. But it mentions InnoDB statistics. So I'm wondering if it's possible to disable some statistics as a workaround to stop the crash until the fix is available. Any advice on that approach welcome too.

[Edward Hibbert (edward-g)]

I commented on the upstream bug, and someone suggested that turning off the stats might indeed be a workaround. I'll report back in case it helps other people who see this.

[mysqld]
innodb-stats-persistent = 0
innodb-stats-transient-sample-pages = 20
innodb-stats-auto-recalc = 0

[Jai Gupta (jai-g)]

Edward, please let us know if this workaround helps you.

[Edward Hibbert (edward-g)]

Will do - was ok today, but it'd take several days before I could be confident it had helped.

[Edward Hibbert (edward-g)]

Still ok today, so looking promising.

[Jai Gupta (jai-g)]

Thanks Edward. I see Percona Server with the fix has been released. I hope they soon release it for Cluster as well.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-1061