ifdown stops LTSP's "manual" interface on shutdown

[Alkis Georgopoulos]

The unit /lib/systemd/system/ifup@.service is Debian and Ubuntu specific, it's not part of upstream systemd.
On shutdown, it unconditionally ifdowns all interfaces:
  ExecStop=/sbin/ifdown %I

This is a regression from previous init systems (sysvinit and upstart) which cared so that when a network file system was in use, they didn't ifdown the interfaces.

Specifically, both /etc/init.d/networking and /etc/init/networking contain these functions:
  check_network_file_systems()
  check_network_swaps()
which output the message "not deconfiguring network interfaces: network file systems still mounted" and exit.

So, please call the same functions in the ExecStop= part of ifup@.service.

[Alkis Georgopoulos (alkisg)]

This also affects LTSP, e.g. Wily clients can't shutdown because /sbin/poweroff is no longer accessible once ifup@.service unmounts /dev/nbd0.

If systemd is fixed in Ubuntu before 15.10 is released, problem solved,
otherwise we'll need to commit the following in upstream LTSP.

Script /usr/share/ltsp/init-ltsp.d/50-systemd-ifup:

if [ -f /lib/systemd/system/ifup@.service ]; then
    mkdir -p /etc/systemd/system/ifup@$DEVICE.service.d
    cat > /etc/systemd/system/ifup@$DEVICE.service.d/ltsp.conf <<EOF
[Service]
ExecStop=
EOF
fi

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Pitt (pitti)]

http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=7b9a61f1

[Alkis Georgopoulos (alkisg)]

Thanks a lot Martin, works fine for me.
/me reverts the LTSP workaround...

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 225-1ubuntu7

---------------
systemd (225-1ubuntu7) wily; urgency=medium

  * Don't restart logind on upgrades any more. This kills X.org (#798097)
    while logind doesn't save/restore its open fds (issue #1163), and also
    gets confused about being idle in between (LP: #1473800)
  * debian/extra/initramfs-tools/hooks/udev: Copy all
    /etc/udev/rules.d/*.rules rules which are not merely overriding the one in
    /lib/, not just 70-persistent-net.rules. They might contain network names
    or other bits which are relevant for the initramfs. (Closes: #795494)
  * ifup@.service: Drop PartOf=network.target; we don't want to stop these
    units during shutdown. Stopping networking.service already shuts down the
    interfaces, but contains the safeguard for NFS or other network file
    systems. Isolating emergency.target still keeps working as before as well,
    as this also stops networking.service. (Closes: #761909, LP: #1492546)
  * networkd: Change IPForward= default to "kernel". This keeps compatibility
    with lots of packages which expect to be able to enable global forwarding
    in /proc/sys/net/ipv4/ip_forward. (LP: #1500992)

 -- Martin Pitt <email address hidden> Mon, 05 Oct 2015 15:53:26 +0200

[Alkis Georgopoulos (alkisg)]

Martin, I'm seeing a regression for this with systemd 228-2ubuntu1 in Ubuntu Xenial.
Now I again have to remove the "ExecStop=/sbin/ifdown %I" stanza in order to get netbooted clients to shut down.

Maybe something else is stopping the ifup service on shutdown now?
I tried to downgrade to the version shipped with Wily, but I was blocked by some dependency issues.

[Martin Pitt (pitti)]

Alkis, can you please attach /var/log/syslog after a reboot, so that I can see what's happening on shutdown?

[Alkis Georgopoulos (alkisg)]

Hi Martin, I've set up remote syslogging and I'm attaching the relevant lines for the "ltsp33" client.

[Martin Pitt (pitti)]

Alkis, unfortunately rsyslog stops too early (before stopping ifup@.service), the log shows no teardown of network at all. Not a surprise with remote logging :-)

Is it possible for you to mount a local disk on /var/log/, enable persistent journal (sudo install -d -g systemd-journal /var/log/journal), reboot, and attach the previous journal with "sudo journalctl -b -1 > /tmp/journal.txt"?

[Alkis Georgopoulos (alkisg)]

Thank you Martin, I'm attaching journal.txt.

[Martin Pitt (pitti)]

This script provides a test case for bisection. Start a schroot with e. g.

  schroot -b -c xenial -n systemd-test

and run the test script on the unpacked schroot dir:

  sudo ./systemd-letmelive.sh /var/lib/schroot/unpack/systemd-test/

which will boot the schroot in nspawn and test if [e-mail] gets killed during shutdown.

[Martin Pitt (pitti)]

bisected to https://github.com/systemd/systemd/commit/8c8da0e0cb4

[Martin Pitt (pitti)]

Reported to https://github.com/systemd/systemd/issues/2189. If this can't/won't get fixed upstream, we'll drop the ExecStop= as a workaround.

[Martin Pitt (pitti)]

These units moved to ifupdown in the meantime, moving component. There is now a fix for this, by adding Slice=system.slice to ifup@.service.

[Martin Pitt (pitti)]

Fix sent to Debian bug.

[Martin Pitt (pitti)]

@Alkis: It's possible that this still affects Xenial even with fixing ifup@.service, as networking.service (which does not do that network file check any more) might also shut down the interfaces; it just does it later on.

But this bears the question why on an LTSP system ifupdown has a configuration for the interface that the root fs is mounted on? This interface is being set up by the initramfs already, via udhcpd; ifupdown should not have a config stanza (or at least only a manual one) for this. How does this work? In particular, what's /etc/network/interfaces and interfaces.d/* on that system?

Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package ifupdown - 0.8.8ubuntu1

---------------
ifupdown (0.8.8ubuntu1) xenial; urgency=medium

  * Merge with Debian unstable. Remaining Ubuntu changes:
    - Always call dhclient with -1, Ubuntu carries a patch so that renewal
      won't time out.
    - link.defn, tests/testbuild-linux: Disable upping of
      bridges as it's not covering all the cases supported by the vlan and
      bridge hooks and so causes more harm than good at this point.
      (LP: #1295304)
    - debian/postinst: Bring back missing /etc/rcS.d/networking symlink, to
      clean up after old broken upgrades. (LP: #1439109) Needs to be kept
      until after 16.04 LTS.
    - ifup@.service, ifupdown-hotplug: Additionally run for "auto" class. We
      don't really support "allow-hotplug" in Ubuntu at the moment, so we need
      to deal with "auto" devices appearing after the "networking" init script
      already ran. (LP: #1374521)

ifupdown (0.8.8) unstable; urgency=medium

  [ Martin Pitt ]
  * Fix ifquery crash if interface state file does not exist yet.
    (Closes: #810779, LP: #1532722)
  * ifup@.service: Avoid stopping on shutdown via stopping system-ifup.slice
    (changed behaviour in systemd 228). (Closes: #761909, LP: #1492546)

  [ Guus Sliepen ]
  * Remove quotes around TimeoutStartSec parameter. Closes: #810656

ifupdown (0.8.7) unstable; urgency=medium

  * Impose a 5 minute timeout for starting network services.
    Closes: #318577, #810656
  * Give an error when trying to up a dhcp interface when no DHCP client is
    installed. Closes: #471926, #419139
  * Add a section in the manual about extensions provided by other packages.
    Closes: #482405, #597274
  * Use absolute paths to binaries called by ifup/ifdown. Closes: #365114
  * Return an error when trying to ifup/ifdown/ifquery an unknown interface
    (unless it is ignored due to --allow or --exclude).
  * Ignore SIGPIPE. Closes: #311054
  * Update the bridge example. Closes: #488315
  * Use the onlink option when adding a gateway route. Closes: #378506
  * Also allow the metric option for tunnels and IPv6 interfaces.

 -- Martin Pitt <email address hidden> Wed, 13 Jan 2016 07:00:46 +0100

[Alkis Georgopoulos (alkisg)]

> ifupdown should not have a config stanza (or at least only a manual one) for this.

Hi Martin, yup, when network manager was introduced in Ubuntu I've had added code to LTSP to dynamically put "iface $DEVICE inet manual" in /etc/network/interfaces, to prevent network manager from assigning an IP to it, breaking netboot.
An empty line didn't do the trick, "manual" was necessary.

I'll test and leave feedback when systemd 0.8.8ubuntu1 reaches the repositories.

Thanks a lot for your work in this!

[Martin Pitt (pitti)]

Alkis Georgopoulos [2016-01-13 8:47 -0000]:
> An empty line didn't do the trick, "manual" was necessary.

Ah, that's good. "manual" should indeed keep NM from touching it, but
current ifupdown should also not tear down "manual" interfaces on
shutdown. So in theory, with 0.8.8ubuntu1 your shutdown should be good
now. Confirming this would be really helpful.

Thanks!

[Alkis Georgopoulos (alkisg)]

Well...

This is *not* needed anymore, i.e. the current bug was solved:
# sed '/^ExecStop=/d' -i /lib/systemd/system/ifup@.service

Unfortunately now this is needed, i.e. the same bug appeared elsewhere:
# sed '/^ExecStop=/d' -i /lib/systemd/system/networking.service

The networking.service file now got an unconditional ifdown command:
$ grep ExecStop /lib/systemd/system/networking.service
ExecStop=/sbin/ifdown -a --read-environment

Wouldn't it be easier to create a small.... "careful-ifdown" script, that borrows code from /etc/init/networking.conf, and have all the .service and init files call that one instead?

[Martin Pitt (pitti)]

Adjusting the Debian bug link, I discussed this on https://bugs.debian.org/809166 already. ifupdown should *not* stop "manual" interfaces with ifdown, unless you have an explicit "down" action in /etc/network/interfaces. I assume this is not the case. It was an explicit decision by the Debian maintainer (Guus) to not take these "network file system" checks into the new .service, as this hides real bugs and should really not be necessary.

So I take it the bug is now that "ifdown"ing a manual interface actually "ifconfig down"s it?

[Martin Pitt (pitti)]

Updating bug title and reopening.

[Martin Pitt (pitti)]

I fixed ifupdown to not "down" an interface which is "manual". This behaves according to the documentation again, and should fix this even without the old check_network_file_systems() from /etc/init.d/networking.

Alkis, would be great if you could test with 0.8.8ubuntu2!

[Launchpad Janitor (janitor)]

This bug was fixed in the package ifupdown - 0.8.8ubuntu2

---------------
ifupdown (0.8.8ubuntu2) xenial; urgency=medium

  * inet{,6}.defn: Don't down an interface when stopping. This breaks use
    cases like LTSP which create a "manual" interfaces stanza to prevent
    NetworkManager or other management software from touching the
    interface of the remote root file system. In the past we had the
    check_network_file_systems() check in /etc/init.d/networking that papered
    over this misbehaviour, but that is not being done any more in
    networking.service. (Closes: #809166, LP: #1492546)

 -- Martin Pitt <email address hidden> Wed, 13 Jan 2016 16:00:49 +0100

[Alkis Georgopoulos (alkisg)]

> Alkis, would be great if you could test with 0.8.8ubuntu2!

I confirm that ifupdown 0.8.8ubuntu2 solves the issue.
I tested both with plain `poweroff` and with `ifdown -a`, which didn't bring DOWN the "manual" interface.

Thanks a lot Martin!

[Martin Pitt (pitti)]

I also filed a (Debian) bug against LTSP yesterday, about improving this "hiding" of the eth interface from NetworkManager directly. This avoids having to rely on these subtleties of ifupdown.

[Alkis Georgopoulos (alkisg)]

I'm closing the LTSP task as LTSP 5.5.6 got released and was published in Xenial without the fix being included, and the newer ifupdown makes the fix not necessary anymore.