MySQL 5.7.17-11 and 5.7.16-10 are crashing after upgrading from MySQL 5.6.29-76.2

[KRISHAN KUMAR]

We upgraded Percona MySQL server version 5.6.29-76.2 to 5.7.16-10 (first) but it was crashing then we upgraded to 5.7.17-11 that is also crashing.

Error from log

00:01:39 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=268435456
read_buffer_size=1048576
max_used_connections=417
max_threads=1001
thread_count=408
connection_count=406
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1556244 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7ef95c03c880
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ef8e6df7d40 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xed056c]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x7a0631]
/lib64/libpthread.so.0[0x31d080f7e0]
/usr/sbin/mysqld(_ZN10Field_blob15copy_blob_valueEP11st_mem_root+0x28)[0x7e3d08]
/usr/sbin/mysqld(_Z25mysql_prepare_blob_valuesP3THDR4ListI4ItemEP11st_mem_root+0x2b8)[0xe1b468]
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP9COPY_INFOS4_+0x87d)[0xe1c02d]
/usr/sbin/mysqld(_ZN14Sql_cmd_insert12mysql_insertEP3THDP10TABLE_LIST+0x82d)[0xe1ca9d]
/usr/sbin/mysqld(_ZN14Sql_cmd_insert7executeEP3THD+0xc2)[0xe1d2d2]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x18d7)[0xca9e67]
/usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x357)[0xcd9e57]
/usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0xca)[0xcda25a]
/usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDmmPhm+0x13b)[0xcda58b]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x190f)[0xcb197f]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1b7)[0xcb24a7]
/usr/sbin/mysqld(handle_connection+0x2a0)[0xd76740]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xeed474]
/lib64/libpthread.so.0[0x31d0807aa1]
/lib64/libc.so.6(clone+0x6d)[0x31d00e8aad]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7ef95c12bec0): is an invalid pointer
Connection ID (thread ID): 4170
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
2017-02-24T00:01:39.635763Z 0 [Warning] The use of InnoDB is mandatory since MySQL 5.7. The former options like '--innodb=0/1/OFF/ON' or '--skip-innodb' are ignored.
2017-02-24T00:01:39.635857Z 0 [Warning] The syntax 'avoid_temporal_upgrade' is deprecated and will be removed in a future release
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
2017-02-23T18:01:46.687261-06:00 0 [Warning] CA certificate ca.pem is self signed.
2017-02-23T18:01:46.692576-06:00 0 [ERROR] /usr/sbin/mysqld: Table './mysql/user' is marked as crashed and should be repaired
2017-02-23T18:01:46.692898-06:00 0 [Warning] Checking table: './mysql/user'
2017-02-23T18:01:46.692933-06:00 0 [ERROR] 1 client is using or hasn't closed the table properly

The upgrade process was successful without any error.

Please let us know if you need further information?

[KRISHAN KUMAR (krishyadav)]

my.cnf

#
# Percona 5.6 InnoDB Master/Master config for 8GB RAM
#
#
[client]
socket = /var/run/mysql/mysql.sock
# Defaults for NOVA -- GBM #
default-character-set = utf8

[mysql]
port = 3306
socket = /var/run/mysql/mysql.sock

[mysql@upladevnsq01v ~]$ cat /etc/redhat-release
CentOS release 6.8 (Final)
[mysql@upladevnsq01v ~]$ cat /proc/meminfo
MemTotal: 16329908 kB

[mysqld]
# GENERAL #
read-only = ON
user = mysql
default-storage-engine = InnoDB
socket = /var/run/mysql/mysql.sock
pid-file = /var\/run\/mysqld\/mysql.pid
explicit-defaults-for-timestamp = 1
# Defaults for NOVA -- GBM #
character-set-server = utf8
collation-server = utf8_general_ci

# MyISAM #
key-buffer-size = 256M
myisam-recover-options = FORCE,BACKUP

# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000
skip-name-resolve
sysdate-is-now = 1
innodb = FORCE
innodb-strict-mode = 1

# DATA STORAGE #
datadir = /data/mysql/
tmpdir = /data/mysql_tmp/

# BINARY LOGGING #
log-bin = /data/mysql/mysql-bin
expire-logs-days = 3
sync-binlog = 1

# CACHES AND LIMITS #
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
read-buffer-size = 1048576
max-connections = 1000
thread-cache-size = 40
open-files-limit = 16384
table-definition-cache = 1000
table-open-cache = 1000

# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 512M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 9G

# LOGGING #
log-error = /var/log/mysql/mysql-error.log
log-queries-not-using-indexes = 0
slow-query-log = 1
long-query-time = 2
slow-query-log-file = /var/log/mysql/mysql-slow.log
log-slow-verbosity = full
log_timestamps = SYSTEM
log_error_verbosity = 2

# Replication settings
server-id = 1
auto_increment_increment = 2
auto_increment_offset = 1
relay_log = mysql-relay-bin
binlog_format = mixed
expire_logs_days = 2
report-host = upladevnsq01v
report-user = repl
master-info-repository = TABLE
relay-log-info-repository = TABLE
skip-slave-start = 1

[Sveta Smirnova (svetasmirnova)]

Thank you for the report.

Please provide steps you used when performed upgrade. Have you run mysql_upgrade?

[KRISHAN KUMAR (krishyadav)]

Yes we did run mysql_upgrade. It completed successfully without any issue.

Here are the steps we took

1) Take full backup
2) Skip-slave-start=1 ===>> we do not want to start slave automatically.
3) SET GLOBAL innodb_fast_shutdown=0;
4) sudo service mysql stop (STOP MYSQL)
5) Take snapshot of VM, Remove 5.6 binaries and install 5.7 binaries
6) If needed start MySQL (normally it is already started)
7) check the error log (/var/log/mysql/mysql-error.log)====>>> it reports the error since mysql_upgrade has not yet run.
8) Run mysql_upgrade -uadmin -p -h localhost ====>>> Completed successfully without any error.
9) Set in my.cnf log_error_verbosity=2, log_timestamps=SYSTEM
10) Do a final Restart of MySQL service mysql restart
11) check the error log (less /var/log/mysql/mysql-error.log)====>>> At this point there is no error reported in error log.
12) Start the slave and test replication
13) Failover traffic from non-upgraded node to upgraded node
14) Repeat above steps on other node.

[Jericho Rivera (jericho-rivera)]

Just adding here that the backtrace is pretty similar to the upstream bug that is still not verified - https://bugs.mysql.com/bug.php?id=79243. checking the upstream bug, we'll need a repeatable test case before this can be confirmed.

[KRISHAN KUMAR (krishyadav)]

We can reproduce this issue in one of our database though we have multiple database running on 5.7 but we see this issue in only one database.

Please let us know what you need from us to help you out.

Thanks

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3652