networking stop incorrectly disconnects from (network) root filesystem
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Invalid
|
Undecided
|
Unassigned | ||
ifupdown (Debian) |
Fix Released
|
Unknown
|
|||
ifupdown (Ubuntu) |
Fix Released
|
High
|
Scott Moser | ||
Xenial |
Fix Released
|
High
|
Scott Moser | ||
Yakkety |
Fix Released
|
Medium
|
Unassigned |
Bug Description
=== Begin SRU Template ===
[Impact]
The systemd networking.service unit will bring down the loopback device (lo)
when it is stopped. This behavior differs from the behavior in other
Ubuntu releases (upstart's networking.conf), where 'lo' is not taken down.
The problem that was seen was that iscsi root over ipv6 would hang on
shutdown.
[Test Case]
Test is fairly simple and can be demonstrated in lxc container.
The key is really that the lo device should not have its link set down
after stopping networking.service. So, below:
out=$(ip address show dev lo up); [ -n "$out" ] && echo "$out" || echo empty
should not show 'empty', but should have LOOPBACK,UP,LOWER in its output.
$ release=yakkety; name=y1
$ lxc launch ubuntu-
$ sleep 10
$ lxc exec $name /bin/bash
## show only things that are up (note output has LOOPBACK,
% ip link show dev lo
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
% ip address show dev lo up
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
## Stop the service and show lo link is down (no 'UP' or 'LOWER_UP').
% systemctl stop networking.service
% ip link show dev lo
1: lo: <LOOPBACK> mtu 65536 qdisc noqueue state DOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
% out=$(ip address show dev lo up); [ -n "$out" ] && echo "$out" || echo empty
empty
## Now enable proposed, install update, reboot and show.
% rel=$(lsb_release -sc)
% echo "deb http://
sudo tee /etc/apt/
% sudo apt update -qy && sudo apt install -qy ifupdown </dev/null
% dpkg-query --show ifupdown
ifupdown 0.8.13ubuntu3
% sudo reboot
## in rebooted system
% ip link show dev lo
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
% ip address show dev lo up
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
% systemctl stop networking.service
% ip link show dev lo
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
% out=$(ip address show dev lo up); [ -n "$out" ] && echo "$out" || echo empty
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
[Regression Potential]
Should be pretty low. zesty and yakkety-proposed have this.
Taking down 'lo' is often cause of problems, and never the solution to
problems as far as I'm aware.
[Other Info]
=== End SRU Template ===
With the switch to systemd, all support for iscsi root (and other) filesystems disappeared, since shutdown yanks the rug out from under us.
Rather than just relying on /etc/iscsi/
Related bugs:
* bug 1229458: grub2 needed changes
* bug 1621615: network not configured when ipv6 netbooted into cloud-init
* bug 1621507: ipv6 network boot does not work
description: | updated |
Changed in maas: | |
status: | New → Triaged |
milestone: | none → 2.1.0 |
Changed in ifupdown (Ubuntu): | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in ifupdown (Ubuntu Xenial): | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in maas: | |
milestone: | 2.1.0 → 2.1.1 |
Changed in ifupdown (Debian): | |
status: | Unknown → New |
Changed in maas: | |
milestone: | 2.1.1 → 2.1.2 |
Changed in maas: | |
milestone: | 2.1.2 → 2.1.3 |
Changed in ifupdown (Ubuntu Xenial): | |
status: | Confirmed → In Progress |
tags: |
added: verification-done removed: verification-needed |
Changed in maas: | |
status: | Triaged → Confirmed |
Changed in ifupdown (Debian): | |
status: | New → Fix Released |
ifupdown is actually behaving correctly in this case. It's likely that cloud-initramfs -tools should be marking the interface "iface foo inet[6] manual" in every case, since that's the indicator that seems to prevent ifdown from downing the interface.
When I had it do that, what I then found was that we had gotten to "reached shutdown target", and the system was then having timeouts in systemd watchdogs. Looking at it with systemd's debug shell, /media/root-ro (the iscsi volume) has been unmounted, and /media/root-rw (the overlayfs) is still mounted.
Interestingly, the last few things are:
[ OK ] Unmounted /media/root-rw.
[ OK ] Reached target Unmount All Filesystems.
[ OK ] Stopped target Local File Systems (Pre).
Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[ OK ] Stopped Remount Root and Kernel File Systems
[ OK ] Stopped Create Static Device Nodes in /dev.
[ OK ] Reached target Shutdown
5 seconds later is the "connection 1:0: ping timeout of 5 secs expired, .... lowerdir= /media/ root-ro, upperdir= /media/ root-rw/ /overlay, workdir= /media/ root-rw/ /overlay- workdir
and / is still (or is that "again") mounted on overlayfs rw,relatime,
And then there is the collection of "INFO: task systemd:1 blocked for more than 120 seconds." log entries on the console.