Shutdown impossible when handlersocket active; blocked or ignored

[jolan]

Hi,

I have 3 nodes that are mostly idle and they refuse to shutdown gracefully.

They are all 64-bit Ubuntu Trusty with version 5.5.39-25.11-816.

- /etc/init.d/mysql stop returns "fail"
- service mysql stop does the same as above
- kill -TERM is ignored
- mysqladmin shutdown just hangs until I ^C: Warning; Aborted waiting on pid file: '/var/lib/mysql/testdbe2.pid' after 351 seconds

Only kill -9 works.

This is on a small test cluster with nearly 0 load:

Uptime: 1413 Threads: 52 Questions: 777 Slow queries: 0 Opens: 53 Flush tables: 1 Open tables: 46 Queries per second avg: 0.549

mysql> SHOW VARIABLES LIKE 'thread_handling%';
+-----------------+---------------------------+
| Variable_name | Value |
+-----------------+---------------------------+
| thread_handling | one-thread-per-connection |
+-----------------+---------------------------+
1 row in set (0.00 sec)

mysql> SHOW STATUS LIKE '%dirty%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| Innodb_buffer_pool_pages_dirty | 0 |
| Innodb_buffer_pool_bytes_dirty | 0 |
+--------------------------------+-------+
2 rows in set (0.00 sec)

There's nothing present in the log that indicates that a shutdown was requested. It's like it's being ignored or blocked.

Absolutely nothing about self-leave, innodb shutdown, etc.

I can't think of any other pertinent information.

This behavior is present across all 3 nodes.

[jolan (jolan)]

This seems to have been related to handlersocket, after I commented it out in our config:

#loose_handlersocket_port = 9998
#loose_handlersocket_port_wr = 9999
#loose_handlersocket_threads = 16
#loose_handlersocket_threads_wr = 1

all nodes can now shutdown properly.

[Przemek (pmalkowski)]

I can confirm this problem, unable to shutdown a node when the handlersocket plugin is enabled together with it's threads and ports open. Nodes where handlersocket plugin is loaded but no ports open due to missing variables, can be still shutdowned with no problem.

Behavior is bit different between 5.5.39 and 5.6.20. In the first case, there is not even a single error log entry showing that shutdown sequence was started. Both service stop, mysqladmin shutdown nor normal kill make no impact on mysqld. While on 5.6.20 we can see that shutdown sequence actually started but then it stalls and mysqld does not stop.

percona1 mysql> select @@version,@@version_comment;
+----------------+---------------------------------------------------------------------------------------------------+
| @@version | @@version_comment |
+----------------+---------------------------------------------------------------------------------------------------+
| 5.6.20-68.0-56 | Percona XtraDB Cluster (GPL), Release rel68.0, Revision 888, WSREP version 25.7, wsrep_25.7.r4126 |
+----------------+---------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

percona1 mysql> \! netstat -lpn|grep mysql
tcp 0 0 0.0.0.0:9998 0.0.0.0:* LISTEN 4649/mysqld
tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTEN 4649/mysqld
tcp 0 0 0.0.0.0:4567 0.0.0.0:* LISTEN 4649/mysqld
tcp 0 0 :::3306 :::* LISTEN 4649/mysqld

percona1 mysql> select * from information_schema.processlist where user="system user";
+----+-------------+-----------------+---------------+---------+------+-------------------------------------------+------+---------+-----------+---------------+
| ID | USER | HOST | DB | COMMAND | TIME | STATE | INFO | TIME_MS | ROWS_SENT | ROWS_EXAMINED |
+----+-------------+-----------------+---------------+---------+------+-------------------------------------------+------+---------+-----------+---------------+
| 6 | system user | connecting host | NULL | Connect | 0 | handlersocket: mode=rd, 0 conns, 0 active | NULL | 0 | 0 | 0 |
| 4 | system user | connecting host | NULL | Connect | 0 | handlersocket: mode=rd, 0 conns, 0 active | NULL | 0 | 0 | 0 |
| 5 | system user | connecting host | NULL | Connect | 0 | handlersocket: mode=rd, 0 conns, 0 active | NULL | 0 | 0 | 0 |
| 3 | system user | connecting host | NULL | Connect | 0 | handlersocket: mode=rd, 0 conns, 0 active | NULL | 0 | 0 | 0 |
| 13 | system user | connecting host | NULL | Connect | 0 | handlersocket: mode=rd, 0 conns, 0 active | NULL | 0 | 0 | 0 |
| 15 | system user | connecting host | NULL ...

[Raghavendra D Prabhu (raghavendra-prabhu)]

This is an issue in upstream Percona Server: https://bugs.launchpad.net/percona-server/+bug/1319904

Hence, I am marking this a duplicate of that bug.