Systemd does not send SIGTERM first on shutdown

[Olli Niemi]

It has been normal that applications first get the SIGTERM signal before SIGKILL on shutdown/reboot in order to successfully finish any pending tasks. Now it seem this logic has been changed to something else, causing problems to mosh and many others:

https://bugs.launchpad.net/ubuntu/+source/mosh/+bug/1446982

SIGTERM suggestion can be seen here:

http://unixhelp.ed.ac.uk/CGI/man-cgi?shutdown+8

I created this error report to find out the correct way for applications to fix this problem or to create one fix to systemd, bringing back the old "BSD shutdown" functionality.

This report is for Ubuntu 15.04.

SRU TEST CASE:
 - Open a terminal, enter some commands, then run "reboot".
 - After a reboot, chances are very high that your bash history does not contain your most recently typed commands
 - With the updated package, the bash history should be intact.

REGRESSION POTENTIAL:
 - The original commit was applied because of an inherent race condition with cgroup's release_agent -- in rare corner cases an nspawn container (probably also LXC) can miss them. In that case it's possible that you instead get a 90s timeout on the unit that is shutting down. But this does not mean data loss, just a rare shutdown hang from containers (for the record, I never actually saw that hanging with LXC), so I think it's a good trade-off.

[Keith Winstein (keithw)]

This does seem to be a systemd bug that is affecting several applications -- basically any application that would like a SIGTERM on reboot/shutdown so that it can save or clean up resources gracefully.

Please see https://bugzilla.redhat.com/show_bug.cgi?id=1141137 (upstream systemd bug)

https://bugzilla.redhat.com/show_bug.cgi?id=1170765 (bug affects bash's attempt to save history)

https://github.com/cockpit-project/cockpit/issues/1155 (bug affects cockpit)

http://lists.freedesktop.org/archives/systemd-devel/2014-October/024452.html (discussion)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Pitt (pitti)]

https://launchpad.net/~pitti/+archive/ubuntu/sru-test now has packages with this fix. I can't upload it yet as there's a previous systemd being tested in -proposed still. Feedback appreciated from all bug reporters, though! (Also for bash history and mosh).

[Maarten Fonville (maarten-fonville)]

I tested the packages, and bash history is now properly retained during a reboot :-)

[Olli Niemi (olliniem)]

I also tested tested the packages. The mosh case is now working but the included test.c tester still fails this case.

[Olli Niemi (olliniem)]

When I compare the documentation I see that with systemd SIGTERM has been removed from it:

Old behavior: http://unixhelp.ed.ac.uk/CGI/man-cgi?shutdown+8
Systemd behavior: http://man7.org/linux/man-pages/man8/shutdown.8.html

Is it now that SIGTERM is sent in case-by-case basis to specific applications (based on error reports)? Should SIGTERM be mentioned in the documentation?

[Martin Pitt (pitti)]

It's still more or less the same, except that every unit and scope is being shut down individually. "man systemd.kill" has the details.

[Martin Pitt (pitti)]

Fixed in Debian experimental branch: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=experimental&id=6068e872da

Cherry-picked into ubuntu branch: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=ubuntu&id=28640752854

[Brian Murray (brian-murray)]

This hasn't made it to -proposed or -updates yet, setting back to Triaged.

[Chris Halse Rogers (raof)]

Hello Olli, or anyone else affected,

Accepted systemd into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/219-7ubuntu5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 219-8ubuntu1

---------------
systemd (219-8ubuntu1) wily; urgency=medium

  * Merge with Debian experimental branch. Remaining Ubuntu changes:
    - Hack to support system-image read-only /etc, and modify files in
      /etc/writable/ instead.
    - Keep our much simpler udev maintainer scripts (all platforms must
      support udev, no debconf).
    - initramfs init-top: Drop $ROOTDELAY, we do that in a more sensible way
      with wait-for-root. Will get applicable to Debian once Debian gets
      wait-for-root in initramfs-tools.
    - initramfs init-bottom: If LVM is installed, settle udev,
      otherwise we get missing LV symlinks. Workaround for LP #1185394.
    - Add debian/udev.lvm2.init: Dummy SysV init script to satisfy insserv
      dependencies to "lvm2" which is handled with udev rules in Ubuntu.
    - Add debian/udev.lvm2.service to avoid running the dummy lvm2 init
      script.
    - Provide shutdown fallback for upstart. (LP: #1370329)
    - debian/extra/ifup@.service: Additionally run for "auto" class. We don't
      really support "allow-hotplug" in Ubuntu at the moment, so we need to
      deal with "auto" devices appearing after "/etc/init.d/networking start"
      already ran. (LP: #1374521) Also, check if devices are actually defined
      in /etc/network/interfaces as we don't use Debian's net.agent.
      Also run ifup in the background during boot, to avoid blocking
      network.target. (LP: #1425376)
    - ifup@.service: Drop dependency on networking.service (i. e.
      /etc/init.d/networking), and merely ensure that /run/network exists.
      This avoids unnecessary dependencies/waiting during boot and dependency
      cycles if hooks wait for other interfaces to come up (like ifenslave
      with bonding interfaces). (LP: #1414544)
    - Add Get-RTC-is-in-local-time-setting-from-etc-default-rc.patch: In
      Ubuntu we currently keep the setting whether the RTC is in local or UTC
      time in /etc/default/rcS "UTC=yes|no", instead of /etc/adjtime.
      (LP: #1377258)
    - Put session scopes into all cgroup controllers. This makes unprivileged
      user LXC containers work under systemd. (LP: #1346734)
    - systemctl: Don't forward telinit u to upstart. This works around
      upstart's Restart() always reexec'ing /sbin/init on Restart(), even if
      that changes to point to systemd during the upgrade. This avoids running
      systemd during a dist-upgrade. (LP: #1430479)
    - Drop hwdb-update dependency from udev-trigger.service, which got
      introduced in v219-stable. This causes udev and plymouth to start too
      late and isn't really needed in Ubuntu yet as we don't support stateless
      systems yet and handle hwdb.bin updates through dpkg triggers. This can
      be dropped again with initramfs-tools 0.117.
    - Lower Breaks: to plymouth version which has the udev inotify fix in
      Ubuntu.
    - Lower libappamor dep to the Ubuntu version where it moved to /lib.
    - Change systemd-sysv's conflicts to upstart-sysv. (LP: #1422681)
    - Make failure of boot-and-services NSpawn.test_boot non-fatal for now.
      This currently fails when being t...

[Martin Pitt (pitti)]

Olli, Maarten, could you please test the package in vivid-proposed instead of my PPA, so that we can officially verify this to get it into vivid-updates? Thank you in advance!

[Maarten Fonville (maarten-fonville)]

For me the package in vivid-proposed works! :-)

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 219-7ubuntu5

---------------
systemd (219-7ubuntu5) vivid; urgency=medium

  * Revert upstream commit 743970d which immediately SIGKILLs units during
    shutdown. This leads to problems like bash not being able to write its
    history, mosh not saving its state, and similar failed cleanup actions.
    (LP: #1448259)
  * ifup@.service: Set IgnoreOnIsolate, so that "systemctl default" does not
    shut down network interfaces. (LP: #1449380). Add PartOf=network.target,
    so that stopping network.target also stops network interfaces.
  * 75-persistent-net-generator.rules: Fix rules for ibmveth (it's a driver,
    not a subsystem). (LP: #1437375)
  * debian/tests/unit-config: Add tests for systemctl enable/disable on a
    SysV-only unit. Reproduces LP #1447807.
  * Fix systemctl enable for SysV scripts without a native unit. We must not
    try and enable the nonexisting unit then. (LP: #1447807)

 -- Martin Pitt <email address hidden> Thu, 07 May 2015 07:45:34 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Olli Niemi (olliniem)]

Tested with Ubuntu Mate 15.04 in Raspberry Pi 2 (test.c, included here). It has systemd 219-7ubuntu5 package. After a reboot one can see that a file "testing.txt" is created but its size is zero, i.e. it was not given enough time to write the file contents. The fix is not working with two laptops and one desktop PC ("testing.txt" file is not created), Ubuntu 15.04. I currently have no access to "development hardware" so I can only test released changes.