cloud-init with iscsi ibft keeps interface at 'manual' so the system gets no dns search domain

Bug #1806777 reported by Dan Streetman on 2018-12-04
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Undecided
Unassigned
cloud-init (Ubuntu)
Undecided
Unassigned
Trusty
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned
Disco
Undecided
Unassigned
open-iscsi (Debian)
New
Unknown
open-iscsi (Ubuntu)
Medium
Dan Streetman
Trusty
Undecided
Unassigned
Xenial
Medium
Dan Streetman
Bionic
Medium
Dan Streetman
Cosmic
Medium
Dan Streetman
Disco
Medium
Dan Streetman

Bug Description

[impact]

in bug 1752391, cloud-init was updated to check if the initramfs has specific iscsi root device config (e.g. static initramfs config, or ibft), and if so then set up the interface in e/n/i but do not mark it as 'auto'. This prevents ifupdown from actually running dhclient for the interface; the reason for the change in that bug was to prevent a hang at shutdown, as taking down the interface removes access to the iscsi root disk (thus hanging the system).

However, iBFT has no mechanism in its spec to provide the DNS search domain, so while the interface ip address/netmask and gateway is correctly set up when the system finishes booting, it does not contain any search domain in /etc/resolv.conf. There is no way to get this information except from an actual dhcp query/response.

In the initramfs, if open-iscsi does not detect ibft (or have static config built-in) then it will not create a /run/net-IFACE.conf file; if it does find ibft (or have static config) it will create a /run/net-IFACE.conf file with the config data that it has (e.g. with ibft, all the interface configuration that ibft is able and configured to provide). After this (setting up the interface and mounting the root dev), ipconfig runs and checks for the existence of /run/net-IFACE.conf file. If it does NOT exist, then ipconfig creates it containing the result of the dhcp lease response that ipconfig received - which includes the dns search domain. However, if there is already a /run/net-IFACE.conf file, ipconfig does NOT update it. So, the file created by open-iscsi remains, without any dns search domain.

Once the system boots, resolvconf checks for existance of the /etc/net-IFACE.conf file, and if found it transfers some of the data into /etc/resolv.conf (e.g. the dns search domain). Since it does not contain any dns search domain, the /etc/resolv.conf file does not contain it either.

If the system is configured to start dhclient for the interface, then dhclient receives and sets up the system resolv.conf to use the dns search domain correctly.

For trusty: cloud-init is very different and sets up the running system ifupdown to start dhclient for the root device interface. The dns search domain is therefore configured by dhclient in the runtime system.

For xenial: cloud-init sets up the iscsi root device's interface with a e/n/i configuration for dhcp, but leaves its 'auto' line commented out, e.g.:

# control-manual ens3f0
iface ens3f0 inet dhcp
    dns-nameservers 169.254.169.254
    gateway 10.0.0.1

this results in no dns search domain for the system. (note: as mentioned above, it does this due to bug 1752391).

For bionic and later: cloud-init sets up netplan instead of ifupdown; netplan has no such 'auto' mechanism. The initramfs-tools scripts create a /run/netplan/$DEVICE.yaml file which (when ibft uses dhcp) contains the 'critical' parameter, causing netplan to create a systemd-networkd .network file with the "CriticalConnection" parameter set to true, which prevents systemd-networkd from ever taking the interface down, which avoids the hang on shutdown (or any other time dhcp fails/stops for the interface). However, it is still possible to remove netplan and install ifupdown in bionic (and later), and when that is done this bug exists, so it should be fixed in those releases also.

[test case]

start a xenial system that uses ibft and check /etc/resolv.conf; there will be no dns search domain.

for bionic or later, start a system that uses ibft, then install ifupdown and resolvconf and remove netplan.io. When the system is rebooted, the /etc/resolv.conf will not have any dns search domain.

[regression potential]

as this slightly changes when/how open-iscsi creates a /run/net-$DEVICE.conf file, this has the potential to boot a system with wrong or even no network configuration, when the system configures an iscsi device at boot, e.g. with iBFT. However, the patch does perform a 'fallback' which should prevent such failure when iBFT does work but ipconfig does not.

[other info]

this appears to have been introduced by bug 1752391. This bug may possibly also exist if the "ip=" parameter is used, which also triggers the cloud-init code that sets the runtime interface to 'manual' instead of 'auto', though I have not specifically tested that.

Trusty's open-iscsi does *not* create a /run/net-$DEVICE.conf file, so the initramfs there will always run ipconfig to gather dhcp data and save that; so this bug should not affect trusty.

Bionic and later do not have this problem with netplan, which is their default, but as noted above if an end-user replaces netplan with ifupdown, this problem still exists.

Side note: bug 1752391 might not be fixed for bionic and later for systems with static iBFT (i.e. not DHCP iBFT) becuase the 'critical' param is only added if DHCP is used (for ipv4 or ipv6).

Related branches

Dan Streetman (ddstreet) wrote :

It seems the option to restore cloud-init functionality to setting the iscsi root device interface to ifupdown 'auto' dhcp would lead back to the hang-on-shutdown.

Possibly another way to address this would be to update klibc's ipconfig to update the /run/net-IFACE.conf file with the dns search domain it gets. I will look at that; i can't think of any problematic side effects of ipconfig updating that specific field in the conf file without changing anything else. open-iscsi always sets it to an empty string, since it is per-spec impossible for ibft to ever provide a dns search domain. I shoudl check the possibility of static root iscsi config built-in to the initramfs as well, in case that might contain dns search domain that ipconfig shoudl not overwrite.

Changed in cloud-init:
status: New → Invalid
Changed in cloud-init (Ubuntu Trusty):
status: New → Invalid
Changed in cloud-init (Ubuntu Bionic):
status: New → Invalid
Changed in cloud-init (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Dan Streetman (ddstreet)
Changed in cloud-init (Ubuntu):
status: New → Invalid
Changed in cloud-init (Ubuntu Xenial):
importance: Undecided → Medium
Dan Streetman (ddstreet) wrote :

I should note that while I marked this against cloud-init, it may be that something else (e.g. ipconfig as mentioned in the last comment) may need to be changed to fix this.

Dan Streetman (ddstreet) on 2018-12-05
tags: added: sts
Dan Streetman (ddstreet) wrote :

Discussed with @smoser and we think this can be fixed in open-iscsi's initramfs local-top script; I am patching and testing now.

Changed in open-iscsi (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Dan Streetman (ddstreet)
Dan Streetman (ddstreet) on 2018-12-05
Changed in cloud-init (Ubuntu Xenial):
status: In Progress → Invalid
assignee: Dan Streetman (ddstreet) → nobody
Changed in open-iscsi (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Medium
status: New → In Progress
Changed in open-iscsi (Ubuntu Cosmic):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Medium
status: New → In Progress
Changed in open-iscsi (Ubuntu Disco):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Medium
status: New → In Progress
Changed in open-iscsi (Ubuntu Xenial):
importance: Undecided → Medium
Dan Streetman (ddstreet) on 2018-12-05
description: updated
Changed in cloud-init (Ubuntu Cosmic):
status: New → Invalid
Changed in cloud-init (Ubuntu Xenial):
importance: Medium → Undecided
Dan Streetman (ddstreet) on 2018-12-05
Changed in open-iscsi (Ubuntu Trusty):
status: New → Invalid
description: updated
Dan Streetman (ddstreet) wrote :

On bionic iBFT system, I switched to ifupdown:

$ sudo apt install ifupdown
...
$ sudo apt install resolvconf
...
$ sudo apt remove netplan.io
...
$ sudo vi /etc/network/interfaces
...
$ cat /etc/network/interfaces

auto lo
iface lo inet loopback

source /etc/network/interfaces.d/*.cfg

$ sudo update-initramfs -u
...
$ sudo reboot

...[system reboots]...

$ cat /etc/network/interfaces.d/50-cloud-init.cfg
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
auto lo
iface lo inet loopback

# control-manual ens3f0
iface ens3f0 inet dhcp
    dns-nameservers 169.254.169.254
    gateway 10.0.0.1

$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 169.254.169.254
nameserver 127.0.0.53

So this bug can be reproduced on bionic (and later) by replacing netplan with ifupdown and resolvconf.

Dan Streetman (ddstreet) wrote :

Testing on bionic with above ppa:

$ dpkg -l |grep open-iscsi
ii open-iscsi 2.0.874-5ubuntu2.4+test1806777v20181205b2 amd64 iSCSI initiator tools

$ cat /etc/network/interfaces.d/50-cloud-init.cfg
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
auto lo
iface lo inet loopback

# control-manual ens3f0
iface ens3f0 inet dhcp
    broadcast 10.0.0.255
    dns-nameservers 169.254.169.254
    dns-search default.oraclevcn.com
    gateway 10.0.0.1

$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 169.254.169.254
nameserver 127.0.0.53
search default.oraclevcn.com

Dan Streetman (ddstreet) on 2018-12-05
no longer affects: cloud-init (Debian)
Changed in open-iscsi (Debian):
status: Unknown → New
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu12

---------------
open-iscsi (2.0.874-5ubuntu12) disco; urgency=medium

  [Scott Moser]
  * debian/extra/initramfs.local-top: handle iSCSI iBFT DHCP to correctly
    run ipconfig to gather all DHCP config info, including DNS search
    domain, which iBFT can't provide. (LP: #1806777)

 -- Dan Streetman <email address hidden> Wed, 05 Dec 2018 11:28:12 -0500

Changed in open-iscsi (Ubuntu Disco):
status: In Progress → Fix Released

Hello Dan, or anyone else affected,

Accepted open-iscsi into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-iscsi/2.0.874-5ubuntu9.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in open-iscsi (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Brian Murray (brian-murray) wrote :

Hello Dan, or anyone else affected,

Accepted open-iscsi into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-iscsi/2.0.874-5ubuntu2.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in open-iscsi (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Brian Murray (brian-murray) wrote :

Hello Dan, or anyone else affected,

Accepted open-iscsi into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-iscsi/2.0.873+git0.3b4b4500-14ubuntu3.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in open-iscsi (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Dan Streetman (ddstreet) wrote :

Xenial:

ubuntu@lp1806777-xenial:~$ dpkg -l |grep open-iscsi
ii open-iscsi 2.0.873+git0.3b4b4500-14ubuntu3.6 amd64 iSCSI initiator tools
ubuntu@lp1806777-xenial:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 169.254.169.254

...
(upgrade open-iscsi and reboot)
...

ubuntu@lp1806777-xenial:~$ dpkg -l | grep open-iscsi
ii open-iscsi 2.0.873+git0.3b4b4500-14ubuntu3.7 amd64 iSCSI initiator tools
ubuntu@lp1806777-xenial:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 169.254.169.254
search default.oraclevcn.com

tags: added: verification-done-xenial
removed: verification-needed-xenial
Dan Streetman (ddstreet) wrote :

Bionic (modified per comment 4 to replace netplan with ifupdown):

ubuntu@lp1806777-bionic:~$ dpkg -l | grep open-iscsi
ii open-iscsi 2.0.874-5ubuntu2.4 amd64 iSCSI initiator tools
ubuntu@lp1806777-bionic:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 169.254.169.254
nameserver 127.0.0.53

...
(upgrade open-iscsi and reboot)
...

ubuntu@lp1806777-bionic:~$ dpkg -l | grep open-iscsi
ii open-iscsi 2.0.874-5ubuntu2.5 amd64 iSCSI initiator tools
ubuntu@lp1806777-bionic:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 169.254.169.254
nameserver 127.0.0.53
search default.oraclevcn.com

tags: added: verification-done-bionic
removed: verification-needed-bionic
Dan Streetman (ddstreet) wrote :

Cosmic (modified per comment 4):

ubuntu@lp1806777-cosmic:~$ dpkg -l | grep open-iscsi
ii open-iscsi 2.0.874-5ubuntu9.1 amd64 iSCSI initiator tools
ubuntu@lp1806777-cosmic:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 169.254.169.254
nameserver 127.0.0.53

...
(upgrade and reboot)
...

ubuntu@lp1806777-cosmic:~$ dpkg -l | grep open-iscsi
ii open-iscsi 2.0.874-5ubuntu9.2 amd64 iSCSI initiator tools
ubuntu@lp1806777-cosmic:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 169.254.169.254
nameserver 127.0.0.53
search default.oraclevcn.com

tags: added: verification-done verification-done-cosmic
removed: verification-needed verification-needed-cosmic
Brian Murray (brian-murray) wrote :

Hello Dan, or anyone else affected,

Accepted open-iscsi into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-iscsi/2.0.874-5ubuntu9.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-cosmic
removed: verification-done verification-done-cosmic
tags: added: verification-needed-bionic
removed: verification-done-bionic
Brian Murray (brian-murray) wrote :

Hello Dan, or anyone else affected,

Accepted open-iscsi into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-iscsi/2.0.874-5ubuntu2.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Steve Langasek (vorlon) wrote :

verification was done with the previous build, but this code hasn't changed in the subsequent SRU which is a one-character change, so re-marking this verified for bionic and cosmic.

tags: added: verification-done verification-done-bionic verification-done-cosmic
removed: verification-needed verification-needed-bionic verification-needed-cosmic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package open-iscsi - 2.0.873+git0.3b4b4500-14ubuntu3.7

---------------
open-iscsi (2.0.873+git0.3b4b4500-14ubuntu3.7) xenial; urgency=medium

  [Scott Moser]
  * debian/extra/initramfs.local-top: handle iSCSI iBFT DHCP to correctly
    run ipconfig to gather all DHCP config info, including DNS search
    domain, which iBFT can't provide. (LP: #1806777)

 -- Dan Streetman <email address hidden> Wed, 05 Dec 2018 14:33:36 -0500

Changed in open-iscsi (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for open-iscsi has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu2.6

---------------
open-iscsi (2.0.874-5ubuntu2.6) bionic; urgency=medium

  * debian/iscsi-disk.rules: Fix bug with LVM on top of iscsi devices.
    (LP: #1807978)

open-iscsi (2.0.874-5ubuntu2.5) bionic; urgency=medium

  [Scott Moser]
  * debian/extra/initramfs.local-top: handle iSCSI iBFT DHCP to correctly
    run ipconfig to gather all DHCP config info, including DNS search
    domain, which iBFT can't provide. (LP: #1806777)

 -- Scott Moser <email address hidden> Tue, 11 Dec 2018 19:14:17 -0500

Changed in open-iscsi (Ubuntu Bionic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu9.3

---------------
open-iscsi (2.0.874-5ubuntu9.3) cosmic; urgency=medium

  * debian/iscsi-disk.rules: Fix bug with LVM on top of iscsi devices.
    (LP: #1807978)

open-iscsi (2.0.874-5ubuntu9.2) cosmic; urgency=medium

  [Scott Moser]
  * debian/extra/initramfs.local-top: handle iSCSI iBFT DHCP to correctly
    run ipconfig to gather all DHCP config info, including DNS search
    domain, which iBFT can't provide. (LP: #1806777)

 -- Scott Moser <email address hidden> Tue, 11 Dec 2018 19:10:50 -0500

Changed in open-iscsi (Ubuntu Cosmic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.