Invalid read of size 1

[Raghavendra D Prabhu]

# 2014-07-14T12:22:07 [1255] ==00:00:00:00.000 6930== Command: /rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.821.Linux.x86_64/bin/mysqld --no-defaults --basedir=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.821.Linux.x86_64 --datadir=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/148/tmp.oLEFOSVAAP/current1_6/node0/data --lc-messages-dir=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.821.Linux.x86_64/share --character-sets-dir=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.821.Linux.x86_64/share/charsets --tmpdir=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/148/tmp.oLEFOSVAAP/current1_6/node0/tmp --core-file --max-allowed-packet=128Mb --port=12540 --socket=/tmp/RQGmysql.12540.sock --pid-file=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/148/tmp.oLEFOSVAAP/current1_6/node0/mysql.pid --general-log --general-log-file=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/148/tmp.oLEFOSVAAP/current1_6/node0/mysql.log --wsrep_sst_method=rsync --innodb_autoinc_lock_mode=2 --default-storage-engine=InnoDB --binlog-format=row --wsrep_node_incoming_address=127.0.0.1 --wsrep_node_address=127.0.0.1 --wsrep_cluster_address=gcomm://?gmcast.listen_addr=tcp://127.0.0.1:5215&pc.ignore_sb=true --wsrep_sst_receive_address=127.0.0.1:5218 --skip-performance-schema --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb-buffer-pool-populate --innodb_flush_method=O_DIRECT --innodb_change_buffering=none --innodb_lock_wait_timeout=100 --lock_wait_timeout=1500 --innodb_adaptive_hash_index_partitions=1 --wsrep_retry_autocommit=1 --wsrep_slave_threads=4 --wsrep_causal_reads=ON --innodb_flush_log_at_trx_commit=0 --transaction-isolation=READ-COMMITTED --innodb_file_per_table=1 --innodb_file_format=barracuda --log-bin=mysql-bin --binlog_format=ROW --wsrep-provider=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.821.Linux.x86_64/lib/libgalera_smm.so
# 2014-07-14T12:22:07 [1255] ==00:00:00:00.000 6930==
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.004 6930== Thread 58:
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.004 6930== Invalid read of size 1
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.004 6930== at 0x4C28F52: strlen (mc_replace_strmem.c:403)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x67E359: Protocol::store(char const*, charset_info_st const*) (protocol.cc:894)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x747E52: mysqld_list_processes(THD*, char const*, bool) (sql_show.cc:2130)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x70A213: mysql_execute_command(THD*) (sql_parse.cc:4394)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x70D477: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:7289)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x70D5C0: wsrep_mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:7040)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x70F893: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1626)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x710C62: do_command(THD*) (sql_parse.cc:1133)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x6D9F7E: do_handle_one_connection(THD*) (sql_connect.cc:1557)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x6DA156: handle_one_connection (sql_connect.cc:1461)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x5037850: start_thread (in /lib64/libpthread-2.12.so)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== by 0x6B7B94C: clone (in /lib64/libc-2.12.so)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930== Address 0x37cd3db0 is not stack'd, malloc'd or (recently) free'd
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.005 6930==
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== Invalid read of size 1
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== at 0x4C28F64: strlen (mc_replace_strmem.c:403)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x67E359: Protocol::store(char const*, charset_info_st const*) (protocol.cc:894)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x747E52: mysqld_list_processes(THD*, char const*, bool) (sql_show.cc:2130)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x70A213: mysql_execute_command(THD*) (sql_parse.cc:4394)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x70D477: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:7289)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x70D5C0: wsrep_mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:7040)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x70F893: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1626)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x710C62: do_command(THD*) (sql_parse.cc:1133)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x6D9F7E: do_handle_one_connection(THD*) (sql_connect.cc:1557)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x6DA156: handle_one_connection (sql_connect.cc:1461)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x5037850: start_thread (in /lib64/libpthread-2.12.so)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== by 0x6B7B94C: clone (in /lib64/libc-2.12.so)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930== Address 0x37cd3db1 is not stack'd, malloc'd or (recently) free'd
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.007 6930==
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== Invalid read of size 1
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== at 0xB7E1F2: my_convert (ctype.c:1066)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x67EC97: Protocol::net_store_data(unsigned char const*, unsigned long, charset_info_st const*, charset_info_st const*) (sql_string.h:127)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x747E52: mysqld_list_processes(THD*, char const*, bool) (sql_show.cc:2130)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x70A213: mysql_execute_command(THD*) (sql_parse.cc:4394)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x70D477: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:7289)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x70D5C0: wsrep_mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:7040)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x70F893: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1626)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x710C62: do_command(THD*) (sql_parse.cc:1133)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x6D9F7E: do_handle_one_connection(THD*) (sql_connect.cc:1557)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x6DA156: handle_one_connection (sql_connect.cc:1461)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x5037850: start_thread (in /lib64/libpthread-2.12.so)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== by 0x6B7B94C: clone (in /lib64/libc-2.12.so)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930== Address 0x37cd3db0 is not stack'd, malloc'd or (recently) free'd
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.008 6930==
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== Invalid read of size 1
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== at 0xB7E20D: my_convert (ctype.c:1059)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x67EC97: Protocol::net_store_data(unsigned char const*, unsigned long, charset_info_st const*, charset_info_st const*) (sql_string.h:127)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x747E52: mysqld_list_processes(THD*, char const*, bool) (sql_show.cc:2130)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x70A213: mysql_execute_command(THD*) (sql_parse.cc:4394)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x70D477: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:7289)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x70D5C0: wsrep_mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:7040)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x70F893: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1626)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x710C62: do_command(THD*) (sql_parse.cc:1133)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x6D9F7E: do_handle_one_connection(THD*) (sql_connect.cc:1557)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x6DA156: handle_one_connection (sql_connect.cc:1461)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x5037850: start_thread (in /lib64/libpthread-2.12.so)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== by 0x6B7B94C: clone (in /lib64/libc-2.12.so)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930== Address 0x37cd3db0 is not stack'd, malloc'd or (recently) free'd
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.009 6930==
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== Invalid read of size 1
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== at 0xB7E208: my_convert (ctype.c:1066)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x67EC97: Protocol::net_store_data(unsigned char const*, unsigned long, charset_info_st const*, charset_info_st const*) (sql_string.h:127)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x747E52: mysqld_list_processes(THD*, char const*, bool) (sql_show.cc:2130)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x70A213: mysql_execute_command(THD*) (sql_parse.cc:4394)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x70D477: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:7289)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x70D5C0: wsrep_mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:7040)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x70F893: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1626)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x710C62: do_command(THD*) (sql_parse.cc:1133)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x6D9F7E: do_handle_one_connection(THD*) (sql_connect.cc:1557)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x6DA156: handle_one_connection (sql_connect.cc:1461)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x5037850: start_thread (in /lib64/libpthread-2.12.so)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== by 0x6B7B94C: clone (in /lib64/libc-2.12.so)
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930== Address 0x37cd3db1 is not stack'd, malloc'd or (recently) free'd
# 2014-07-14T12:22:07 [1255] ==00:00:06:40.011 6930==

[Raghavendra D Prabhu (raghavendra-prabhu)]

http://jenkins.percona.com/job/PXC-5.6-mrandgen/148/BTYPE=release,Host=pxc-rqg/artifact/results-148/trial22.log/*view*/

[Raghavendra D Prabhu (raghavendra-prabhu)]

# 2014-07-14T12:14:17 [1255] Copyright (c) 2010,2011 Oracle and/or its affiliates. All rights reserved. Use is subject to license terms.
# 2014-07-14T12:14:17 [1255] Please see http://forge.mysql.com/wiki/Category:RandomQueryGenerator for more information on this test framework.
# 2014-07-14T12:14:17 [1255] Starting
# 2014-07-14T12:14:17 [1255] # /rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/pxcgen/runall-new.pl \
# 2014-07-14T12:14:17 [1255] # --queries=100000000 \
# 2014-07-14T12:14:17 [1255] # --seed=random \
# 2014-07-14T12:14:17 [1255] # --short_column_names \
# 2014-07-14T12:14:17 [1255] # --mysqld=--skip-performance-schema \
# 2014-07-14T12:14:17 [1255] # --mysqld=--log-output=none \
# 2014-07-14T12:14:17 [1255] # --mysqld=--sql_mode=ONLY_FULL_GROUP_BY \
# 2014-07-14T12:14:17 [1255] # --grammar=conf/galera/galera_stress.yy \
# 2014-07-14T12:14:17 [1255] # --gendata=conf/galera/galera_stress.zz \
# 2014-07-14T12:14:17 [1255] # --threads=25 \
# 2014-07-14T12:14:17 [1255] # --valgrind \
# 2014-07-14T12:14:17 [1255] # --reporter=ValgrindErrors,Shutdown,Backtrace,ErrorLog,ErrorLogAlarm,Deadlock \
# 2014-07-14T12:14:17 [1255] # --validator=MarkErrorLog \
# 2014-07-14T12:14:17 [1255] # --mysqld=--innodb-buffer-pool-populate \
# 2014-07-14T12:14:17 [1255] # --mysqld=--innodb_flush_method=O_DIRECT \
# 2014-07-14T12:14:17 [1255] # --mysqld=--innodb_change_buffering=none \
# 2014-07-14T12:14:17 [1255] # --mysqld=--innodb_lock_wait_timeout=100 \
# 2014-07-14T12:14:17 [1255] # --mysqld=--lock_wait_timeout=1500 \
# 2014-07-14T12:14:17 [1255] # --mysqld=--innodb_adaptive_hash_index_partitions=1 \
# 2014-07-14T12:14:17 [1255] # --mysqld=--wsrep_retry_autocommit=1 \
# 2014-07-14T12:14:17 [1255] # --mysqld=--wsrep_slave_threads=4 \
# 2014-07-14T12:14:17 [1255] # --mysqld=--wsrep_causal_reads=ON \
# 2014-07-14T12:14:17 [1255] # --mysqld=--innodb_flush_log_at_trx_commit=0 \
# 2014-07-14T12:14:17 [1255] # --mysqld=--transaction-isolation=READ-COMMITTED \
# 2014-07-14T12:14:17 [1255] # --mysqld=--innodb_file_per_table=1 \
# 2014-07-14T12:14:17 [1255] # --mysqld=--innodb_file_format=barracuda \
# 2014-07-14T12:14:17 [1255] # --mysqld=--log-bin=mysql-bin \
# 2014-07-14T12:14:17 [1255] # --mysqld=--binlog_format=ROW \
# 2014-07-14T12:14:17 [1255] # --rows=10000 \
# 2014-07-14T12:14:17 [1255] # --galera=ms \
# 2014-07-14T12:14:17 [1255] # --mysqld=--wsrep-provider=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.821.Linux.x86_64/lib/libgalera_smm.so \
# 2014-07-14T12:14:17 [1255] # --mtr-build-thread=254 \
# 2014-07-14T12:14:17 [1255] # --mask=17323 \
# 2014-07-14T12:14:17 [1255] # --duration=180 \
# 2014-07-14T12:14:17 [1255] # --basedir1=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.19-25.6.821.Linux.x86_64 \
# 2014-07-14T12:14:17 [1255] # --gendata=conf/galera/galera_stress-56.zz \
# 2014-07-14T12:14:17 [1255] # --grammar=conf/galera/galera_stress-56.yy \
# 2014-07-14T12:14:17 [1255] # --seed=time \
# 2014-07-14T12:14:17 [1255] # --vardir1=/rqg/workspace/PXC-5.6-mrandgen/BTYPE/release/Host/pxc-rqg/148/tmp.oLEFOSVAAP/current...

[David Bennett (dbpercona)]

After a bit of searching it looks like this is a bug in file: sql/sql_show.cc function: mysqld_list_processes() . I noticed a recent commit in the upstream MySQL 5.6 code base that makes a copy of the global thread list and changes the thread lock to monitor removed threads. This insures that newly created threads aren't included in the loop and that old threads cannot be removed during processing.

It's just a hunch but it might be worth taking a quick look at:

http://bazaar.launchpad.net/~mysql/mysql-server/5.6/revision/5927#sql/sql_show.cc

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1699