Wily/15.10 - systemd hangs reboot + shutdown for long periods of time

[David Favor]

Yet another horrible bug related to systemd which delays shutdowns by minutes.

https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1374759 surfaces this problem + the bug I'm reporting here is a separate issue.

Neither reboot or shutdown -r now work sensibly any more. Both rely on systemd, which appears to be the root of most Ubuntu instabilities now, reading the many reported systemd bugs.

Both the reboot + shutdown -r now commands should do what the mean. Shutdown + Reboot now...

Not 150-250 seconds after requested, which appears to be some arbitrary systemd timeout.

Please fix this problem + update this ticket with some workaround to get an instant reboot, till fix is available.

Thanks.

[David Favor (davidfavor)]

It appears - reboot -f - still works.

And this is a harsh way to reboot a system.

[David Favor (davidfavor)]

The systemd facility changes sysvinit behavior considerably, especially sending no SIGKILL after SIGTERM.

Suggested solution, send SIGKILL after timeout, following SIGTERM.
________

Primary difference is systemd defers forever to underlying processes, rather than shooting them in the head eventually.

This means processes which trap SIGTERM + never end, hang system shutdown.

https://bugs.launchpad.net/ubuntu/+source/mysql-5.6/+bug/1468804 describes this problem with MySQL, not MariaDB, which does a very fast shutdown.

Another culprit is fail2ban, which takes far to long to shutdown. I'll push a shutdown fix for fail2ban shortly.

Worst offenders are /etc/rc.local processes, especially where a supervisor program runs children + one or more children fail to die or the supervisor restarts them when they die. Issuing - systemctl --no-pager status rc.local.service - in a shell while issuing reboot in another shell shows this problem.

The easiest way to debug a reboot hang is to run two shells + do reboot in one.

Then a few seconds later issue - systemctl --no-pager status - in the surviving shell. A quick scan of this output usually surfaces the culprit.

Starting the attached script in /etc/rc.local as follows...

looper > /var/log/looper.log 2>&1

show systemd sends SIGTERM + never follows up with a SIGKILL, at least within 5 minutes, which is when I killed the process myself to allow reboot to finish shutdown + do restart.

net4-dev# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.10
Release: 15.10
Codename: wily

net4-dev# uname -a
Linux net4.bizcooker.com 4.2.0-22-generic #27-Ubuntu SMP Thu Dec 17 22:57:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Pitt (pitti)]

So apparently you already identified some processes that hang on shutdown. /usr/share/doc/systemd/README.Debian.gz describes how to find and debug those.

If you want to keep this open against systemd, please attach /var/log/syslog right after a reboot, which should still contain most of the shutdown log from the previous boot (if that isn't sufficient, we'll need a complete journal for shutdown, after enabling persistent journal -- but let's get to that when we need it).

[Launchpad Janitor (janitor)]

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]