information_schema database corrupted

[Shiv Narasiah]

When I try to run,
use information_schema; select * from tables;
mysql crashes
tried running mysql_upgrade --force as well it crashes to.

Here is o/p I see in error log,

04:03:16 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=4
max_threads=1002
thread_count=4
connection_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 406872 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7faab8f4f000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fae890a8d00 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x8e4f7b]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0x65e121]
/lib64/libpthread.so.0(+0xf370)[0x7fae8c790370]
/usr/sbin/mysqld[0x9f60fe]
/usr/sbin/mysqld[0xaa2dc8]
/usr/sbin/mysqld[0xa4929a]
/usr/sbin/mysqld[0x990b2f]
/usr/sbin/mysqld[0x9973be]
/usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcii+0x33)[0x59e0c3]
/usr/sbin/mysqld[0xb71e7a]
/usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcii+0x33)[0x59e0c3]
/usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x6bc)[0x768fdc]
/usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0x1116)[0x693576]
/usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x6c5)[0x69b565]
/usr/sbin/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x60)[0x69be70]
/usr/sbin/mysqld[0x711a19]
/usr/sbin/mysqld(_Z14get_all_tablesP3THDP10TABLE_LISTP4Item+0x699)[0x724749]
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2ea)[0x72527a]
/usr/sbin/mysqld(_ZN4JOIN14prepare_resultEPP4ListI4ItemE+0xa5)[0x705715]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x15c)[0x6be91c]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x275)[0x70a035]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x195)[0x70a8c5]
/usr/sbin/mysqld[0x55d236]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1626)[0x6e21c6]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5b8)[0x6e7f58]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xfe4)[0x6e96e4]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1c2)[0x6b5752]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6b57f0]
/usr/sbin/mysqld(pfs_spawn_thread+0x146)[0x91d176]
/lib64/libpthread.so.0(+0x7dc5)[0x7fae8c788dc5]
/lib64/libc.so.6(clone+0x6d)[0x7fae8a9d773d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fa9f1473010): is an invalid pointer
Connection ID (thread ID): 12
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

[Muhammad Irfan (muhammad-irfan)]

What version of MySQL you are running ?

mysql> SHOW VARIABLES LIKE '%version%';

Did you recently upgraded ? Also, post contents of my.cnf

[Shiv Narasiah (snarasia)]

mysql 5.6.35

This has been running for a couple of months.

Here is my.cnf config,

[mysqld]
basedir = /usr
datadir = /mysql_data_new/db
plugin-dir = /usr/lib64/mysql/plugin
user = mysql
log-error = /var/log/mysql/mysqlacctarch01-prod.log
pid-file = /var/lib/mysql/mysql.pid
tmpdir = /mysql_data_new/tmp

###################################################
################## REPLICATION ####################
###################################################

#log-bin = mysqld-bin
log-bin = /mysql_data_new/binlogs/mysqlacctarch01-bin
log-bin-index = /mysql_data_new/binlogs/mysqlacctarch01-bin.index
relay-log = /mysql_data_new/binlogs/mysqlacctarch01-relay
relay-log-index = /mysql_data_new/binlogs/mysqlacctarch01-relay.index
relay-log-info-file = /mysql_data_new/binlogs/mysqlacctarch01-relay.info
binlog_format = row
gtid-mode = on
enforce-gtid-consistency
log-slave-updates
report-host = mysqlacctarch01-prod
server-id = 12
max_allowed_packet = 128M
max_connections = 1000
expire_logs_days = 4
binlog_cache_size = 8M

slave-skip-errors = 1032,1062

####################################################
################## PERFORMANCE #####################
####################################################

innodb_buffer_pool_size = 10G
innodb_flush_log_at_trx_commit = 2
max_heap_table_size = 4G
tmp_table_size = 4G
#max_heap_table_size = 32M
#tmp_table_size = 32M
thread_cache_size = 30
innodb_log_file_size = 2G
innodb_lock_wait_timeout = 150
innodb_file_per_table = 1
innodb_flush_method = O_DIRECT
innodb_read_io_threads = 16
innodb_write_io_threads = 16
innodb_file_format = Barracuda
innodb_sort_buffer_size = 71M
max_allowed_packet = 128M
max_connections = 1000
net_read_timeout = 60

####################################################
################## TOKUDB #####################
####################################################

tokudb_cache_size = 8G
tokudb_tmp_dir = /mysql_data_new/tmp
tokudb_row_format = TOKUDB_LZMA
innodb_use_native_aio=0

# skip-slave-start

[mysqld_safe]
preload-hotbackup
thp-setting=never

[Jericho Rivera (jericho-rivera)]

Please upload full mysql error log. Did you try restarting mysql after running mysql_upgrade and did it fix the issue? Is performance_schema enabled?

[Shiv Narasiah (snarasia)]

We rebuilt the DB to get around this error, so please close this.

[Muhammad Irfan (muhammad-irfan)]

I tried to reproduce this with exact same version and by using your my.cnf but no avail. I couldn't reproduce this problem.

mysql> SHOW VARIABLES LIKE '%version%';
+-------------------------+------------------------------------------------------+
| Variable_name | Value |
+-------------------------+------------------------------------------------------+
| innodb_version | 5.6.35-81.0 |
| protocol_version | 10 |
| slave_type_conversions | |
| tls_version | TLSv1.1,TLSv1.2 |
| tokudb_version | 5.6.35-81.0 |
| version | 5.6.35-81.0-log |
| version_comment | Percona Server (GPL), Release 81.0, Revision c96c427 |
| version_compile_machine | x86_64 |
| version_compile_os | Linux |
+-------------------------+------------------------------------------------------+
9 rows in set (0.00 sec)

-- created test tables using InnoDB and TokuDB engines.

mysql> use information_schema
mysql> show tables;
79 rows in set (0.00 sec)

mysql> SELECT * FROM TABLES;
161 rows in set (0.03 sec)

mysql> SHOW ENGINES;
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+
| Engine | Support | Comment | Transactions | XA | Savepoints |
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+
| CSV | YES | CSV storage engine | NO | NO | NO |
| MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |
| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO |
| BLACKHOLE | YES | /dev/null storage engine (anything you write to it disappears) | NO | NO | NO |
| MyISAM | YES | MyISAM storage engine | NO | NO | NO |
| TokuDB | YES | Percona TokuDB Storage Engine with Fractal Tree(tm) Technology | YES | YES | YES |
| PERFORMANCE_SCHEMA | YES | Performance Schema | NO | NO | NO |
| ARCHIVE | YES | Archive storage engine | NO | NO | NO |
| FEDERATED | NO | Federated MySQL storage engine | NULL ...

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3719