rhel7 / systemd -- killing mysqld causes lingering systemd processes

[Jay Janssen]

If I have a running mysqld on Centos 7 and it is killed, systemd likes to spawn some garbage:

32246 ? Ss 0:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr --wsrep-new-cluster
32680 ? Sl 0:00 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/

[root@node1 ~]# killall -9 mysqld

  364 ? Ss 0:00 /bin/bash -ue /usr/bin/mysql-systemd stop-post
  408 ? S 0:00 \_ sleep 1

This prevents me from restarting mysql, any subsequent mysql start action hangs until that process is killed.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Jay,

The 'garbage' is due to the fact that mysqld is SIGkilled but its PID file still remains, so the script waits for it upto timeout (900s) and then exits. This is same behavior as 'service mysql stop'. So, removing the stale PID file should also do.

In any case, I have modified it to check for this and mysql-systemd will exit early.

On a long term, this can be fixed by replacing mysqld_safe and calling mysqld directly in unit file. That should avoid any need for these pre/post scripts.

[Jay Janssen (jay-janssen)]

Ah, thanks for the clarification. systemctl stop mysql is a reasonable workaround and we had to do that before systemd, it’s just much less obvious that it is what is needed.

Is it a bug that systemctl start mysql hangs when this timeout is in play? Isn’t an error message better, or is that not systemd’s way?

> On Nov 27, 2014, at 6:13 AM, Raghavendra D Prabhu <email address hidden> wrote:
>
> The 'garbage' is due to the fact that mysqld is SIGkilled but its PID
> file still remains, so the script waits for it upto timeout (900s) and
> then exits. This is same behavior as 'service mysql stop'. So,
> removing the stale PID file should also do.
>
> In any case, I have modified it to check for this and mysql-systemd
> will exit early.
>
> On a long term, this can be fixed by replacing mysqld_safe and calling
> mysqld directly in unit file. That should avoid any need for these
> pre/post scripts.

Jay Janssen, Managing Consultant, Percona
http://about.me/jay.janssen

[Raghavendra D Prabhu (raghavendra-prabhu)]

The fix for this bug has been released as well. Changing status to reflect that (released only for centos7 since it matters only there).

systemctl stop is good for graceful stop. (equivalent of service stop)

If you want to kill the process (like with SIGKILL etc.), you can use systemctl kill

%%%%%%%%
       kill PATTERN...
           Send a signal to one or more processes of the unit. Use --kill-who= to select which process to kill. Use --signal= to select the signal to send.
%%%%%%%%%%%

"systemctl start mysql" may not have been hung, just that it was waiting for another mysqld to start, but mysqld itself won't start since innodb uses locks on the files, so the new one mysqld will remain spinning. (you need to kill the first mysqld to fix this, this is a mysql bug of sorts).

Anyway, the use case for this bug should be fixed.

One more thing, systemctl doesn't write anything to stderr/stdout. For logs and messages, you can do journalctl -u mysql.service (or mysql@bootstrap etc.). (You can also add -f to follow the logs).

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1776