SIGTERM kills server instead of shutdown w/ threadpool, spurious mysqld_safe server restarts

Bug #1537554 reported by Josip Deanovic on 2016-01-24
46
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Status tracked in 5.7
5.5
Invalid
Undecided
Unassigned
5.6
Invalid
Undecided
Unassigned
5.7
Fix Released
High
Laurynas Biveinis

Bug Description

OS: Centos 6.7 x86_64
Percona server version: 5.7.10-1rc1.1.el6.x86_64

When the option "thread_handling = pool-of-threads" is added to the "[mysqld]" section of the /etc/my.cnf config file the mysql service cannot be gracefully shutdown using corresponding init script.

There is a similar but very old bug report for Percona XtraDB Cluster:
https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1185523

However it seems that the bug still exist in the Percona Server 5.7.10-1rc1.

Josip Deanovic (djosip) wrote :

I have just noticed that the bug does not exist with 5.6.27-rel76.0.el6.x86_64 (Centos 6.7 x86_64).

Josip Deanovic (djosip) wrote :

I have just tested Percona-Server on Centos 6.7 x86_64 with Percona-Server version 5.7.10-2rc2 and the bug still exists.

tags: added: threadpool
removed: pool-of-thread
Zupo Llask (zupollask) wrote :

I'm having this very same problem with current Percona Server 5.7.11-4-1.trusty on fully updated Ubuntu 14.04.4 LTS with kernel 4.4.0-22-generic #40~14.04.1-Ubuntu SMP (package linux-virtual-lts-xenial).

Zupo Llask (zupollask) wrote :

I can add this only happens with "pool-of-threads" value. Specifying "one-thread-per-connection" doesn't generate this bug.

summary: - Fail to gracefully shutdown when using pool-of-thread
+ Fail to gracefully shutdown when using pool-of-threads

Laurynas,

I hope you don't mind but I've subscribed you to this bug as I think this is pretty serious.

Without this issue solved I don't know how to enable threadpool, which is a key feature of Percona Server.

Thanks.

Zupo Llask (zupollask) wrote :

Latest Percona Server 5.7.12-5 is also affected by this issue, running in latest Ubuntu LTS 14.04.4.

John Allen (johnjallen) wrote :

Seeing this with 5.7.14-7:

If I strace mysqld_safe and kill $pid of mysql, mysqld_safe waiting for signal gets:

wait4(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGTERM}], 0, NULL) = 84866

With thread_handling=one-thread-per-connection, it gets:

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 85754

In the second case mysqld_safe exits, in the first it starts the mysqld process again.

Changed in percona-server:
status: New → Triaged
importance: Undecided → High
Josip Deanovic (djosip) wrote :

I have just tested Centos 6.8 x86_64 with Percona MySQL server 5.7.14-7.
The problem still persists.

John Allen (johnjallen) wrote :

As a stop gap, I modified the mysql init script to do "mysqladmin shutdown" instead of kill $pid.

Tomislav Plavcic (tplavcic) wrote :

Verified as described on centos 6 for PS 5.7, but 5.5 and 5.6 seem unaffected.

5.5
with: thread_handling=pool-of-threads
[vagrant@localhost ~]$ sudo service mysql stop
Shutting down MySQL (Percona Server)... SUCCESS!
[vagrant@localhost ~]$ ps aux|grep mysql
<empty>

5.6
with: thread_handling=pool-of-threads
[vagrant@localhost ~]$ sudo service mysql start
Starting MySQL (Percona Server). SUCCESS!
[vagrant@localhost ~]$ sudo service mysql stop
Shutting down MySQL (Percona Server).. SUCCESS!
[vagrant@localhost ~]$ ps aux|grep mysql
<empty>

5.7
[vagrant@localhost ~]$ sudo service mysql start
Starting mysqld: [ OK ]
[vagrant@localhost ~]$ sudo service mysql stop
Stopping mysqld: [ OK ]

But server is still running:
[vagrant@localhost ~]$ ps aux|grep mysql
root 2797 0.0 0.3 108304 1660 pts/0 S 07:30 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql 3127 6.8 34.7 1224520 174360 pts/0 Sl 07:31 0:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock

tags: added: regression
Zupo Llask (zupollask) wrote :

Come on @laurynas-biveinis, is this forgotten?

Not forgotten, but still in the queue.

Josip Deanovic (djosip) wrote :

I have just tested Centos 6.8 x86_64 with Percona MySQL server 5.7.15-9.1
The problem still persists.

sch (schery) wrote :

Hi, i tested on Debian 8.7 and Percona 5.7.17-11 => Same problem.

Regards

summary: - Fail to gracefully shutdown when using pool-of-threads
+ SIGTERM kills server instead of shutdown w/ threadpool, spurious
+ mysqld_safe server restarts

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-961

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers