SIGTERM kills server instead of shutdown w/ threadpool, spurious mysqld_safe server restarts

[Josip Deanovic]

OS: Centos 6.7 x86_64
Percona server version: 5.7.10-1rc1.1.el6.x86_64

When the option "thread_handling = pool-of-threads" is added to the "[mysqld]" section of the /etc/my.cnf config file the mysql service cannot be gracefully shutdown using corresponding init script.

There is a similar but very old bug report for Percona XtraDB Cluster:
https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1185523

However it seems that the bug still exist in the Percona Server 5.7.10-1rc1.

[Josip Deanovic (djosip)]

I have just noticed that the bug does not exist with 5.6.27-rel76.0.el6.x86_64 (Centos 6.7 x86_64).

[Josip Deanovic (djosip)]

I have just tested Percona-Server on Centos 6.7 x86_64 with Percona-Server version 5.7.10-2rc2 and the bug still exists.

[Zupo Llask (zupollask)]

I'm having this very same problem with current Percona Server 5.7.11-4-1.trusty on fully updated Ubuntu 14.04.4 LTS with kernel 4.4.0-22-generic #40~14.04.1-Ubuntu SMP (package linux-virtual-lts-xenial).

[Zupo Llask (zupollask)]

I can add this only happens with "pool-of-threads" value. Specifying "one-thread-per-connection" doesn't generate this bug.

[Zupo Llask (zupollask)]

Laurynas,

I hope you don't mind but I've subscribed you to this bug as I think this is pretty serious.

Without this issue solved I don't know how to enable threadpool, which is a key feature of Percona Server.

Thanks.

[Zupo Llask (zupollask)]

Latest Percona Server 5.7.12-5 is also affected by this issue, running in latest Ubuntu LTS 14.04.4.

[John Allen (johnjallen)]

Seeing this with 5.7.14-7:

If I strace mysqld_safe and kill $pid of mysql, mysqld_safe waiting for signal gets:

wait4(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGTERM}], 0, NULL) = 84866

With thread_handling=one-thread-per-connection, it gets:

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 85754

In the second case mysqld_safe exits, in the first it starts the mysqld process again.

[Josip Deanovic (djosip)]

I have just tested Centos 6.8 x86_64 with Percona MySQL server 5.7.14-7.
The problem still persists.

[John Allen (johnjallen)]

As a stop gap, I modified the mysql init script to do "mysqladmin shutdown" instead of kill $pid.

[Tomislav Plavcic (tplavcic)]

Verified as described on centos 6 for PS 5.7, but 5.5 and 5.6 seem unaffected.

5.5
with: thread_handling=pool-of-threads
[vagrant@localhost ~]$ sudo service mysql stop
Shutting down MySQL (Percona Server)... SUCCESS!
[vagrant@localhost ~]$ ps aux|grep mysql
<empty>

5.6
with: thread_handling=pool-of-threads
[vagrant@localhost ~]$ sudo service mysql start
Starting MySQL (Percona Server). SUCCESS!
[vagrant@localhost ~]$ sudo service mysql stop
Shutting down MySQL (Percona Server).. SUCCESS!
[vagrant@localhost ~]$ ps aux|grep mysql
<empty>

5.7
[vagrant@localhost ~]$ sudo service mysql start
Starting mysqld: [ OK ]
[vagrant@localhost ~]$ sudo service mysql stop
Stopping mysqld: [ OK ]

But server is still running:
[vagrant@localhost ~]$ ps aux|grep mysql
root 2797 0.0 0.3 108304 1660 pts/0 S 07:30 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql 3127 6.8 34.7 1224520 174360 pts/0 Sl 07:31 0:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock

[Zupo Llask (zupollask)]

Come on @laurynas-biveinis, is this forgotten?

[Zupo Llask (zupollask)]

lp:~laurynas-biveinis

[Laurynas Biveinis (laurynas-biveinis)]

Not forgotten, but still in the queue.

[Josip Deanovic (djosip)]

I have just tested Centos 6.8 x86_64 with Percona MySQL server 5.7.15-9.1
The problem still persists.

[sch (schery)]

Hi, i tested on Debian 8.7 and Percona 5.7.17-11 => Same problem.

Regards

[Laurynas Biveinis (laurynas-biveinis)]

    https://github.com/percona/percona-server/pull/1445
    https://github.com/percona/percona-server/pull/1446
    https://github.com/percona/percona-server/pull/1447

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-961