mysql stop/restart fails + breaking mysql + reboots

[David Favor]

[Status]

Cannot reproduce - see comment 20. Without steps that someone else can follow to reproduce the problem from a fresh system, we don't expect to make any progress on this issue.

[Original Description]

This is a huge problem.

After several iterations of - service mysql {start|stop|restart} - the mysql service becomes broken + unrecoverable.

The problem becomes - service mysql stop - hangs forever, upon attempting...

     exec systemctl stop mysql.service

What's required then is one of these...

     mysqladmin -uroot -p shutdown
     pkill mysqld

Neither of these is acceptable.

The biggest problem is this breakage also breaks reboots.

The reboot sequence depends on services doing what's expected, so when systemctl stop mysql.service hangs, reboot hangs too.

Because of the lunacy of the entire systemctl subsystem, sshd is correctly killed off while msqld hangs.

At this point, there's no way to ssh into system + a power recycle must be done to complete a reboot sequence.

/bin/systemctl is an executable with no apparent way to debug it's logic.

Please update this bug with details about how to debug /bin/systemctl + I'll post output.

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Unfortunately, we cannot work on this bug because your description didn't include enough information. You may find it helpful to read "How to report bugs effectively" http://www.chiark.greenend.org.uk/~sgtatham/bugs.html. We'd be grateful if you would then provide a more complete description of the problem.

We have instructions on debugging some types of problems at http://wiki.ubuntu.com/DebuggingProcedures

At a minimum, we need:

1. the specific steps or actions you took that caused you to encounter the problem,
2. the behavior you expected, and
3. the behavior you actually encountered (in as much detail as possible).

Instead of "several iterations" please can you describe exactly how to reproduce the problem on a fresh Ubuntu machine?

You've also not said anything about Ubuntu release or version numbers, and you've filed this bug against mysql-dfsg-5.1. MySQL 5.1 never shipped in a release that also shipped systemd. Please reassign to the correct package (maybe mysql-5.6) and then execute the following command, as it will automatically gather debugging information, in a terminal:

apport-collect 1468804

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

Once you've fixed this bug reporting issues, please change the bug status back to New.

Thanks!

[David Favor (davidfavor)]

Redneckery to allow reboots till this bug fixed don't work because apparently systemctl is seriously brain dead.

So the normal shut magic won't work...

echo 'mysqladmin --defaults-extra-file=/etc/mysql/debian.cnf shutdown' > /etc/rc6.d/K00-msyql-stop
chmod +x /etc/rc6.d/K00-msyql-stop

Will update when I determine a workaround.

[David Favor (davidfavor)]

apport-collect 1468804 does not seem to work correctly on headless (server) systems.

Gets into some infinite loop with lynx or whatever tool is running.

ovhfine# uname -a
Linux fine.bizcooker.com 3.19.0-21-generic #21-Ubuntu SMP Sun Jun 14 18:31:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
ovhfine# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.04
Release: 15.04
Codename: vivid
ovhfine# dpkg -l | egrep ^mysql
ovhfine# dpkg -l | egrep mysql
ii libdbd-mysql-perl 4.028-2 amd64 Perl5 database interface to the MySQL database
ii libmysqlclient-dev 5.6.24-0ubuntu2 amd64 MySQL database development files
ii libmysqlclient18:amd64 5.6.24-0ubuntu2 amd64 MySQL database client library
ii mysql-client-5.6 5.6.24-0ubuntu2 amd64 MySQL database client binaries
ii mysql-client-core-5.6 5.6.24-0ubuntu2 amd64 MySQL database core client binaries
ii mysql-common 5.6.24-0ubuntu2 all MySQL database common files, e.g. /etc/mysql/my.cnf
ii mysql-server 5.6.24-0ubuntu2 all MySQL database server (metapackage depending on the latest version)
ii mysql-server-5.6 5.6.24-0ubuntu2 amd64 MySQL database server binaries and system database setup
ii mysql-server-core-5.6 5.6.24-0ubuntu2 amd64 MySQL database server binaries
ii php5-mysql 5.6.4+dfsg-4ubuntu6 amd64 MySQL module for php5

All packages up to date.

[David Favor (davidfavor)]

This doesn't work either...

echo 'mysqladmin --defaults-extra-file=/etc/mysql/debian.cnf shutdown' > /etc/init.d/mysql-hard-stop

ln -s /etc/init.d/mysql-hard-stop /etc/rc0.d/K01-aaa-mysql-hard-stop
ln -s /etc/init.d/mysql-hard-stop /etc/rc6.d/K01-aaa-mysql-hard-stop

[David Favor (davidfavor)]

dirwatch /etc | grep OPEN | egrep 'init.d|rc' shows that shutdown scripts are no longer run sensibly either.

The K01mysql script never runs, so likely there's some other shutdown script with a dependency which is touching mysql + hanging before the K01mysql runs.

[David Favor (davidfavor)]

Changing the /etc/rc{016}.d/K01mysql scripts to do mysqladmin shutdown doesn't work, because inotifywait shows these files are never touched at shutdown.

[David Favor (davidfavor)]

Using inotifywait, reboot sequence is very odd...

ovhfine# inotifywait -mr -e OPEN /etc | egrep 'init.d|rc'
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/etc/init.d/ OPEN rng-tools
/etc/init.d/ OPEN rng-tools
/etc/init.d/ OPEN ntp
/etc/init.d/ OPEN ntp
/etc/init.d/ OPEN hddtemp
/etc/init.d/ OPEN hddtemp
/etc/init.d/ OPEN apport
/etc/init.d/ OPEN apport
/etc/init.d/ OPEN grub-common
/etc/init.d/ OPEN grub-common
/etc/default/ OPEN rcS
/etc/init.d/ OPEN mdadm
/etc/init.d/ OPEN mdadm
/etc/init.d/ OPEN preload
/etc/init.d/ OPEN preload
/etc/init.d/ OPEN ondemand
/etc/init.d/ OPEN ondemand
/etc/default/ OPEN rcS
/etc/init.d/ OPEN sysstat
/etc/init.d/ OPEN sysstat
/etc/init.d/ OPEN apache2
/etc/init.d/ OPEN apache2
/etc/default/ OPEN rcS
/etc/init.d/ OPEN rollerd
/etc/init.d/ OPEN rollerd
/etc/default/ OPEN rcS
/etc/init.d/ OPEN dns-clean
/etc/init.d/ OPEN dns-clean
/etc/default/ OPEN rcS
/etc/init.d/ OPEN irqbalance
/etc/init.d/ OPEN irqbalance
/etc/apt/apt.conf.d/ OPEN 20archive

Notice the mysql shutdown script is never accessed. Looking at the apache2 shutdown script, there seems to be no connection which might interfere with mysql.

Placing mysqladmin shutdown in /etc/init.d/apache2 fails to allow reboot either.

[David Favor (davidfavor)]

Nothing related shows up in /var/log/syslog at reboot either...

Jun 25 12:03:54 fine systemd[1]: Deactivating swap /dev/sdb3...
Jun 25 12:03:54 fine systemd[1]: Starting Unattended Upgrades...
Jun 25 12:03:54 fine systemd[1]: Deactivating swap /dev/sda3...
Jun 25 12:03:54 fine systemd[1]: Stopped Setup Virtual Console.
Jun 25 12:03:54 fine systemd[1]: Stopping Setup Virtual Console...
Jun 25 12:03:54 fine systemd[1]: Stopped target Timers.
Jun 25 12:03:54 fine systemd[1]: Stopping Timers.
Jun 25 12:03:54 fine systemd[1]: Stopped Daily Cleanup of Temporary Directories.
Jun 25 12:03:54 fine systemd[1]: Stopping Daily Cleanup of Temporary Directories.
Jun 25 12:03:54 fine systemd[1]: Stopping Authenticate and Authorize Users to Run Privileged Tasks...
Jun 25 12:03:54 fine systemd[1]: Stopped target Graphical Interface.
Jun 25 12:03:54 fine systemd[1]: Stopping Graphical Interface.
Jun 25 12:03:54 fine systemd[1]: Stopped target Multi-User System.
Jun 25 12:03:54 fine systemd[1]: Stopping Multi-User System.
Jun 25 12:03:54 fine systemd[1]: Stopping Login Service...
Jun 25 12:03:54 fine systemd[1]: Stopping Regular background program processing daemon...
Jun 25 12:03:54 fine systemd[1]: Stopping LSB: Cleans up any mess left by 0dns-up...
Jun 25 12:03:54 fine systemd[1]: Stopped target Login Prompts.
Jun 25 12:03:54 fine systemd[1]: Stopping Login Prompts.
Jun 25 12:03:54 fine systemd[1]: Stopping Getty on tty1...
Jun 25 12:03:54 fine systemd[1]: Stopping LSB: disk temperature monitoring daemon...
Jun 25 12:03:54 fine systemd[1]: Stopping Fail2Ban Service...
Jun 25 12:03:54 fine rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="1027" x-info="http://www.rsyslog.com"] exiting on signal 15.

[David Favor (davidfavor)]

There is no easy fix to deal with systemd problems.

Other physical servers with Vivid + mysql or mariadb have worse problems.

After a number of starts/stops, mysql simply won't start. There is no debug info anywhere.

/var/log/syslog is quiet... Crickets...

Downgrading back to Utopic, as this appears to be the last stable version of Ubuntu. While the basic plumbing for systemd exists in Utopic, the actual /etc/init.d scripts are still working (non-systemd) versions.

[Robie Basak (racb)]

As I said, please give us exact steps to reproduce, preferably verified on a fresh installation. Until then this bug correctly remains both Incomplete and unconfirmed and so nobody will be working on fixing it.

[David Favor (davidfavor)]

Steps to reproduce.

1) install fresh copy of Vivid

2) apt-get -yqq install mysql-server libmysqlclient-dev

3) reboot + system hangs forever (because mysql can't seem to be stopped in all cases by systemd)

4) In another shell - mysqladmin shutdown - allows reboot continues

5) Also - service mysql stop - hangs forever, so only way to consistently stop mysql is via - mysqladmin shutdown

This situation only occurs after several mysql restarts + this situation seems to occur on every Vivid install I've done.

Down leveling to Utopic is how I've had to fix this.

[David Favor (davidfavor)]

This failure occurs consistently on every installation of Vivid.

Downgrading to Utopic, which avoids use of systemd, fixes problem.

[David Favor (davidfavor)]

Problem still persists after latest systemd updates.

[David Favor (davidfavor)]

Cleanest fix is to downgrade to Utopic, as systemd also fails for many other services like Apache + fail2ban.

If running Vivid is a required, this hack seems to correctly top + restart MySQL.

alias mysql-stop='mysqladmin --defaults-extra-file=/etc/mysql/debian.cnf shutdown'
alias mysql-start='service mysql restart'
alias mysql-restart='mysql-stop ; mysql-start'

Only way I've found to get around systemd hang.

[Launchpad Janitor (janitor)]

[Expired for mysql-5.6 (Ubuntu) because there has been no activity for 60 days.]

[David Favor (davidfavor)]

This is a horrible bug + best be fixed.

[David Favor (davidfavor)]

Bug still persists.

http://askubuntu.com/questions/615129/systemd-mysql-wont-stop provides one solution + this is ugly.

Normal mysql should correctly stop itself, without hacking an etc systemd service override.

[beniwtv (beniwtv-deactivatedaccount)]

Also happens with MariaDB. systemctl does not stop it, neither does a restart work.

Unfortunate when you try to update your system, as APT will hang for a long time before the MySQL upgrade script errors out, since MySQL/MariaDB are still running.

[David Favor (davidfavor)]

Bug still persists + apport-collect 1468804 fails to work on headless browsers.

Same problem seems to exist in Xenial.

[Robie Basak (racb)]

I cannot reproduce this neither on Vivid nor on Xenial using your steps in comment 11. I used current release cloud images using uvtool (KVM):

uvt-simplestreams-libvirt sync release=vivid arch=amd64
uvt-kvm create test release=vivid
uvt-kvm ssh --insecure test

and "virsh console test" to watch the system console.

There is no hang on reboot, and starting and stopping mysql using "sudo service mysql stop" (and start) repeatedly works, as does multiple reboots. A mysql service stop seems to take about two seconds.

Can you figure out how your reproduction steps are different from mine? Without this, I don't expect any progress on this.

[Faustin (fauust)]

Ubuntu 15.04 (vivid) is no more supported:
https://www.ubuntu.com/info/release-end-of-life

[Faustin (fauust)]

Thank you for reporting this bug to Ubuntu. RELEASE reached EOL on February 4, 2016.

See this document for currently supported Ubuntu releases: https://wiki.ubuntu.com/Releases

Please upgrade to the latest version and re-test. If the bug is still reproducible, increase the verbosity of the steps to recreate it so we can try again.

Do feel free to report any other bugs you may find.