Percona Server 5.5.24 crashes on userstats=1 with any replication configured

[Jaime Sicam]

How to repeat:
Setup Master-Master replication on Percona Server 5.5, enable userstats on both instances. Try to create database on both instances and the server will crash

node1:
mysql> show global variables like 'userstat';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| userstat | OFF |
+---------------+-------+
1 row in set (0.00 sec)

mysql> set global userstat=1;
Query OK, 0 rows affected (0.00 sec)

mysql> create database db1;
Query OK, 1 row affected (0.00 sec)

mysql> create database db2;
Query OK, 1 row affected (0.00 sec)

mysql> create database db6;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 3
Current database: *** NONE ***

Query OK, 1 row affected (0.00 sec)

node2:
mysql> show global variables like 'userstat';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| userstat | OFF |
+---------------+-------+
1 row in set (0.00 sec)

mysql> create database db3;
Query OK, 1 row affected (0.00 sec)

mysql> create database db4;
Query OK, 1 row affected (0.01 sec)

mysql> set global userstat=1;
Query OK, 0 rows affected (0.00 sec)

mysql> create database db5;
Query OK, 1 row affected (0.00 sec)

Stack trace:
Thread pointer: 0x7fb9b0000990
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fb9c63e7e78 thread_stack 0x40000
/home/user/mysql_binaries/5.5.24/bin/mysqld(my_print_stacktrace+0x35)[0x7d6005]
/home/user/mysql_binaries/5.5.24/bin/mysqld(handle_fatal_signal+0x3e1)[0x691651]
/lib64/libpthread.so.0(+0xf4a0)[0x7fb9df2804a0]
/home/user/mysql_binaries/5.5.24/bin/mysqld[0x61c266]
/home/user/mysql_binaries/5.5.24/bin/mysqld(_Z24update_global_user_statsP3THDbl+0x516)[0x61cb76]
/home/user/mysql_binaries/5.5.24/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0xae)[0x5809fe]
/home/user/mysql_binaries/5.5.24/bin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0x100b)[0x7554db]
/home/user/mysql_binaries/5.5.24/bin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x152)[0x518192]
/home/user/mysql_binaries/5.5.24/bin/mysqld[0x5216b6]
/home/user/mysql_binaries/5.5.24/bin/mysqld(handle_slave_sql+0xc19)[0x5229c9]
/lib64/libpthread.so.0(+0x77f1)[0x7fb9df2787f1]
/lib64/libc.so.6(clone+0x6d)[0x7fb9de4feccd]

Workaround:
Disable userstat on one of the instances (SET GLOBAL userstat=0;)

[yinfeng (yinfeng-zwx)]

this bug is repeatable in Percona Server 5.5.24, but can not repeat in Percona Server 5.5.18.

below is the stack

#0 increment_count_by_name (name=0xdfb4f0 "#mysql_system#", role_name=0xdfb4f0 "#mysql_system#", users_or_clients=0xe4c280, thd=0x3f0bc50)
    at /tmp/Percona-Server-5.5.24-rel26.0/sql/sql_connect.cc:468
#1 0x000000000060eee4 in update_global_user_stats (thd=0x3f0bc50, create_user=true, now=1338860077)
    at /tmp/Percona-Server-5.5.24-rel26.0/sql/sql_connect.cc:668
#2 0x00000000005702b5 in mysql_parse (thd=0x3f0bc50, rawbuf=0x3f20ad0 "create database db27", length=66115608, parser_state=0x4a2933d0)
    at /tmp/Percona-Server-5.5.24-rel26.0/sql/sql_parse.cc:5869
#3 0x0000000000741359 in Query_log_event::do_apply_event (this=0x3f0f990, rli=0x3ec3c98, query_arg=0x3f0faa9 "create database db27", q_len_arg=20)
    at /tmp/Percona-Server-5.5.24-rel26.0/sql/log_event.cc:3440
#4 0x0000000000503f3e in apply_event (rli=<optimized out>, this=<optimized out>) at /tmp/Percona-Server-5.5.24-rel26.0/sql/log_event.h:1136
#5 apply_event_and_update_pos (ev=0x3f0f990, thd=<optimized out>, rli=0x3ec3c98) at /tmp/Percona-Server-5.5.24-rel26.0/sql/slave.cc:2401
#6 0x0000000000506527 in exec_relay_log_event (thd=0x3f0bc50, rli=0x3ec3c98) at /tmp/Percona-Server-5.5.24-rel26.0/sql/slave.cc:2561
#7 0x000000000050ac34 in handle_slave_sql (arg=<optimized out>) at /tmp/Percona-Server-5.5.24-rel26.0/sql/slave.cc:3378
#8 0x0000003e638064a7 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e630d3c2d in clone () from /lib64/libc.so.6
#10 0x0000000000000000 in ?? ()
(gdb) p thd->net.vio
$4 = (Vio *) 0x0

and in function increment_count_by_name, thd->net.vio is used:

 542 if (thd->net.vio->type == VIO_TYPE_SSL)
 543 thread_stats->total_ssl_connections++;
 544 return 0;

this code can be changed like this:
 if ( !thd->slave_thread && thd->net.vio->type == VIO_TYPE_SSL)
    thread_stats->total_ssl_connections++;

because in sql thread, thd->net.vio is always 0x0

[yinfeng (yinfeng-zwx)]

 while starting mysqld with --bootstrap and setting userstat=1 in my.cnf , also crashed at the same place(Bug #1008609).

I don't know if only the two cases can lead to this problem
so this fix may be much safer:

if ( thd->net.vio != NULL && thd->net.vio->type == VIO_TYPE_SSL)

[Alexey Kopytov (akopytov)]

Confirmed, this bug is a regression introduced with https://blueprints.launchpad.net/percona-server/+spec/ssl-connections-stats

[Alexey Kopytov (akopytov)]

Bug #1008609 and bug #1008968 has been marked as duplicates of this one. Please make sure the fix for this bug will also fix those ones.

[yinfeng (yinfeng-zwx)]

Bug #1008968, Bug #1008278 and Bug #1008609 were triggered by the same code:
thd->net.vio should be guaranteed not equal to NULL.

 I've tested these three bugs and this small patch seems really works.

--- a/sql/sql_connect.cc 2012-05-30 01:42:58.000000000 +0800
+++ b/sql/sql_connect.cc 2012-06-10 20:27:39.000000000 +0800
@@ -499,7 +499,7 @@
     }
   }
   user_stats->total_connections++;
- if (thd->net.vio->type == VIO_TYPE_SSL)
+ if (thd->net.vio != NULL && thd->net.vio->type == VIO_TYPE_SSL)
     user_stats->total_ssl_connections++;
   return 0;
 }

[Olivier Doucet (odoucet)]

No need to have replication to trigger this bug. A single server with user_stat can trigger it (bug #1008609)

[Alexey Kopytov (akopytov)]

This bug is a regression and should be fixed in the next release.

[Vlad Lesin (vlad-lesin)]

The bug affects not only master-master replication but any replication with userstat=on.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1220

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-343