Multiple mysqld instances launched by systemd

[Ceri WIlliams]

Firstly, there may be some relation to https://bugs.launchpad.net/percona-server/+bug/1594806

I was having some issues with an install PS 5.7.14 whereby systemd failed to start MySQL during an `apt-get install --only-upgrade`. This left apt rather unhappy, but after some purging to get back to a clean state the problem still occurred.

The output during the install would be something like:

| Job for mysql.service failed. See 'systemctl status mysql.service' and 'journalctl -xn' for details.
| invoke-rc.d: initscript mysql, action "start" failed.
| dpkg: error processing package percona-server-server-5.7 (--configure):
| subprocess installed post-installation script returned error exit status 1
| Processing triggers for systemd (215-17+deb8u2) ...
| Errors were encountered while processing:
| percona-server-server-5.7
| E: Sub-process /usr/bin/dpkg returned an error code (1)

Then I noticed more than one MySQL instance running:

| $ pgrep -x mysqld | xargs ps -opid,ppid,start_time,user,cmd --forest
| PID PPID START USER CMD
| 30103 1 10:51 mysql /usr/sbin/mysqld --log_error_verbosity=2
| 30064 1 10:50 mysql /usr/sbin/mysqld --log_error_verbosity=2
| 30025 1 10:50 mysql /usr/sbin/mysqld --log_error_verbosity=2
| 29985 1 10:50 mysql /usr/sbin/mysqld --log_error_verbosity=2
| 29945 1 10:50 mysql /usr/sbin/mysqld --log_error_verbosity=2
| 29869 1 10:50 mysql /usr/sbin/mysqld --log_error_verbosity=2

Disabling the actions in /usr/share/mysql/mysql-systemd-start and an `apt-get -f install` returned everythin to normal:

| case $1 in
| pre|post) echo $1 ;;
| esac

The MySQL datadir was fine and MySQL would run if executed outside of systemd or via the noop start script.

| # bash -x /usr/share/mysql/mysql-systemd-start pre 2>&1
| + case $1 in
| + sanity
| + MYSQLRUN=/var/run/mysqld
| ++ get_path datadir
| ++ my_print_defaults mysqld
| ++ grep datadir
| ++ cut -d= -f2
| ++ tail -n 1
| + MYSQLDATA=
| + MYSQLFILES=/var/lib/mysql-files
| + MYSQLLOG=/var/log/mysql
| + MYSQLKEYRING=/var/lib/mysql-keyring
| + '[' '!' -d -a '!' -L ']'
| /usr/share/mysql/mysql-systemd-start: line 37: [: too many arguments
| + '[' '!' -d /var/lib/mysql-files -a '!' -L /var/lib/mysql-files ']'
| + '[' '!' -d /var/lib/mysql-keyring -a '!' -L /var/lib/mysql-keyring ']'
| + '[' '!' -d /mysql -a '!' -L /mysql ']'
| ++ ls -A /mysql
| + '[' '!' '' ']'
| + su - mysql -s /bin/bash -c '/usr/sbin/mysqld --initialize-insecure=on 2>&1 > /dev/null'
| 2016-08-26T09:22:15.129354Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
| 2016-08-26T09:22:15.130684Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
| 2016-08-26T09:22:15.130697Z 0 [ERROR] Aborting
|
| + su - mysql -s /bin/bash -c '/usr/sbin/mysqld --log_error_verbosity=2 2>&1 > /dev/null &'
| + pinger
| + /bin/true
| + sleep 1
| 2016-08-26T09:22:15.288898Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
| 2016-08-26T09:22:15.391030Z 0 [Warning] CA certificate ca.pem is self signed.
| + mysqladmin ping
| + break
| + mysql -e 'INSTALL PLUGIN auth_socket SONAME '\''auth_socket.so'\'''
| ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
| + mysql -e 'USE mysql; UPDATE user SET plugin='\''auth_socket'\'' WHERE user='\''root'\'''
| ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
| + mysqladmin shutdown
| mysqladmin: connect to server at 'localhost' failed
| error: 'Access denied for user 'root'@'localhost' (using password: YES)'
| + '[' -x /usr/bin/mysql_ssl_rsa_setup -a '!' -e /server-key.pem ']'
| + /usr/bin/mysql_ssl_rsa_setup --datadir= --uid=mysql
| + '[' '!' -d /var/log/mysql -a '!' -L /var/log/mysql ']'
| + '[' '!' -d /var/run/mysqld -a '!' -L /var/run/mysqld ']'
| + '[' '!' -r /etc/mysql/my.cnf ']'
| + echo 'Percona Server configuration not found at /etc/mysql/my.cnf. Please install one using update-alternatives.'
| Percona Server configuration not found at /etc/mysql/my.cnf. Please install one using update-alternatives.
| + exit 1

After the script exited MySQL is still running:

| $ pgrep -x mysqld| xargs ps -opid,ppid,start_time,cmd --forest
| PID PPID START CMD
| 10349 1 11:22 /usr/sbin/mysqld --log_error_verbosity=2

A password had been set during install.

[Vishal kasle (vishal.percona)]

Hi, can you please send the version of the OS that you are using and also the exact steps that you followed to encounter this problem, so it can be reproduced. Also I tried it on an Ubuntu system, it worked fine. A copy of the my.cnf will also be helpful.

[Vishal kasle (vishal.percona)]

We noticed, that is on Debian 8. Will need to test and try to reproduce the bug.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3533