DNS doesn't work in no-cloud as launched by ubuntu
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| cloud-init |
High
|
Unassigned | ||
| cloud-init (Ubuntu) |
Critical
|
Unassigned | ||
| Zesty |
Undecided
|
Unassigned | ||
| Artful |
Critical
|
Unassigned | ||
| systemd (Ubuntu) |
Critical
|
Canonical Foundations Team | ||
| Zesty |
Undecided
|
Unassigned | ||
| Artful |
High
|
Unassigned | ||
| Bionic |
Critical
|
Canonical Foundations Team |
Bug Description
[Impact]
* resolved does not start early enough in the boot-process preventing DNS resolution to be operational during early boot, for example as required by special early stages of cloud-init, resulting in failure to boot / provision the instance fully.
[Test Case]
* Boot container or a VM with a nocloud-net data source, and a URL pointing to the datasource as explained below
* Observe that boot completes and provisioning is successful
* Check that there are no dns-resolution errors in the cloud-init log / boot log
[Regression Potential]
* starting resolved earlier may prevent it from connecting to dbus, and may require a restart later on when re-triggered over dbus. This is on artful only, as in bionic resolved has gained ability to reconnected to dbus post-start. Backporting that, however, is too large for an SRU as it requires sd-bus changes.
[Other Info]
* Original bug report.
I use no-cloud to test the kernel in CI (I am maintainer of the bcache subsystem), and have been running it successfully under 16.04 cloud images from qemu, using a qemu command that includes:
-smbios "type=1,
As documented here:
http://
Under the new 17.10 cloud images, this doesn't work: the network comes up, but name resolution doesn't work-- /etc/resolv.conf is a symlink to a nonexistent file at this point of the boot and systemd-resolved is not running. When I manually hack /etc/resolv.conf in the cloud image to point to 4.2.2.1 it works fine.
I don't know if nameservice not working is by design, but it seems like it should work. The documentation states:
"With ds=nocloud-net, the seedfrom value must start with http://, https:// or ftp://"
And https is not going to work for a raw IP address.
Related bugs:
* bug 1734939: #include fails silently.
CVE References
Michael Lyle (mlyle) wrote : | #1 |
Michael Lyle (mlyle) wrote : | #2 |
I'm not using the included kernel or initrd, so I decided to test without that.
kvm -machine pc-i440fx-
Properly gets the hostname of 'linuxtst' and all associated configuration on xenial, but not on artful.
Scott Moser (smoser) wrote : | #3 |
$ wget http://
## set up dns locally for 'qemu-host' to the default ip for user networking.
$ grep qemu-host /etc/hosts
10.0.2.2 qemu-host
$ cat data/user-data
#cloud-config
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True
$ cat data/meta-data
instance-id: i-test
## webserv is http://
$ webserve 44225 data
:: 44225
## backdoor the image so you can login with 'backdoor:passw0rd'
# backdoor-image is http://
$ sudo backdoor-image -v --password=passw0rd
$ url="http://
$ qemu-system-x86_64 -enable-kvm \
-device virtio-
-netdev type=user,id=net00 \
-drive file=artful-
-device virtio-
-vga none -nographic -snapshot -echr 0x5 \
-smbios type=1,
## console does show
## [ 20.388179] cloud-init[606]: 2017-11-24 17:03:13,786 - util.py[WARNING]: Gett
## ing data from <class 'cloudinit.
failed
## login
$ pastebinit /var/log/
http://
## interesting part of that is
2017-11-24 17:03:12,779 - url_helper.
2017-11-24 17:03:12,782 - url_helper.
2017-11-24 17:03:13,783 - url_helper.
2017-11-24 17:03:13,786 - handlers.py[DEBUG]: finish: init-network/
2017-11-24 17:03:13,786 - util.py[WARNING]: Getting data from <class 'cloudinit.
2017-11-24 17:03:13,794 - util.py[DEBUG]: Getting data from <class 'cloudinit.
Traceback (most recent call last):
File "/usr/lib/
if s.get_data():
File "/usr/lib/
(md_seed, ud) = util.read_
File "/usr/lib/
md_resp = read_file_
File "/usr/lib/
exception_
File "/usr/lib/
raise excps[-1]
cloudinit.
Scott Moser (smoser) wrote : | #4 |
Heres some more info that is from failed system using bionic.
$ sudo journalctl -o short-monotonic --no-pager | pastebinit
http://
$ sudo base64 /run/log/
http://
Changed in cloud-init: | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in cloud-init (Ubuntu): | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in systemd (Ubuntu): | |
status: | New → Confirmed |
importance: | Undecided → High |
Scott Moser (smoser) wrote : | #5 |
I think the primary issue is that cloud-init.service is depending on using the network fully.
cloud-init.service runs:
After=
After=
Before=
But systemd-
After=
Before=
I tried adding to cloud-init.service.
After=
but that did not help things.
Dimitri John Ledkov (xnox) wrote : | #6 |
<xnox> smoser, yeah, so like cloud-init.service should want/after systemd-
<xnox> smoser, i think changing it in systemd unit might be better.
Scott Moser (smoser) wrote : | #7 |
zesty does not show this problem. neither does xenial. I reflected that in the status.
Changed in cloud-init (Ubuntu Artful): | |
status: | New → Confirmed |
importance: | Undecided → Medium |
importance: | Medium → High |
Changed in systemd (Ubuntu Artful): | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in systemd (Ubuntu Zesty): | |
status: | New → Confirmed |
status: | Confirmed → Fix Released |
Changed in cloud-init (Ubuntu Zesty): | |
status: | New → Fix Released |
Scott Moser (smoser) wrote : | #8 |
zesty does not show this problem. neither does xenial. I reflected that in the status.
$ sudo journalctl -b -o short-monotonic | pastebinit
http://
$ sudo journalctl -o short-precise | pastebinit
http://
Nov 24 17:49:25.193028 ubuntu systemd[1]: systemd-
Nov 24 17:49:25.193038 ubuntu systemd[1]: systemd-
Nov 24 17:49:25.193050 ubuntu systemd[1]: systemd-
Nov 24 17:49:25.193060 ubuntu systemd[1]: systemd-
Scott Moser (smoser) wrote : | #9 |
that ordering cycle is if we add 'After=
Scott Moser (smoser) wrote : | #10 |
To be clear, the suggestion that xnox made causes a ordering cycle.
Changed in systemd (Ubuntu Bionic): | |
assignee: | nobody → Canonical Foundations Team (canonical-foundations) |
Ryan Harper (raharper) wrote : | #11 |
I suspect because in bionic/artful we're missing resolvconf package, that the systemd-resolved service ends up starting later in boot. The systemd-
I suspect that systemd-networkd itself isn't poking DNS service properly after acquiring information.
The dependency loop comes from systemd-resolved using default dependencies which run after when cloud-init.service would run.
This then needs systemd-resolved to specify DefaultDependen
I modified cloud-init.service to include an After=systemd-
Steve Langasek (vorlon) wrote : | #12 |
I agree that systemd-resolved should be DefaultDependen
Of the individual dependencies of sysinit.
Ryan Harper (raharper) wrote : | #13 |
We will still need something that helps ensure systemd-resolved runs we reach network-
Scott Moser (smoser) wrote : | #14 |
I've verified that this is reproducible within lxc, and then filed a bug i
saw (bug 1734939) as a result.
Heres a trivial reproduce:
## just showing content of the url.
$ curl --silent https:/
#!/bin/sh
cat /proc/uptime | tee /run/user-
$ name=btest
$ lxc launch ubuntu-daily:bionic $name \
"--config=
$ sleep 20
$ lxc exec b4 grep WARN /var/log/
2017-11-28 16:49:12,251 - user_data.
Changed in cloud-init (Ubuntu Bionic): | |
importance: | High → Critical |
Changed in cloud-init (Ubuntu Artful): | |
importance: | High → Critical |
Changed in systemd (Ubuntu Bionic): | |
importance: | High → Critical |
description: | updated |
Changed in systemd (Ubuntu Bionic): | |
status: | Confirmed → Fix Committed |
Changed in systemd (Ubuntu Artful): | |
status: | Confirmed → In Progress |
Scott Moser (smoser) wrote : | #15 |
Dimitri,
What is the fix that you put in? I assume it was to systemd ?
Launchpad Janitor (janitor) wrote : | #16 |
This bug was fixed in the package systemd - 235-3ubuntu3
---------------
systemd (235-3ubuntu3) bionic; urgency=medium
* netwokrd: add support for RequiredForOnline stanza. (LP: #1737570)
* resolved.service: set DefaultDependen
* systemd.postinst: enable persistent journal. (LP: #1618188)
* core: add support for non-writable unified cgroup hierarchy for container support.
(LP: #1734410)
-- Dimitri John Ledkov <email address hidden> Tue, 12 Dec 2017 13:25:32 +0000
Changed in systemd (Ubuntu Bionic): | |
status: | Fix Committed → Fix Released |
Scott Moser (smoser) wrote : | #17 |
Marked as fix-released.
I tested today with 20180115.1 image from bionic.
wget http://
url="https:/
qemu-system-x86_64 -enable-kvm -m 768 \
-net nic -net user \
-drive file=disk.
-smbios "type=1,
Just for info, showing:
$ curl https:/
#cloud-config
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True
$ curl https:/
instance-id: iid-brickies-
no longer affects: | cloud-init (Ubuntu Bionic) |
Changed in cloud-init (Ubuntu): | |
status: | Confirmed → Fix Released |
tags: | added: id-5a1c7e7be1c6883c5a843d1f |
description: | updated |
Hello Michael, or anyone else affected,
Accepted systemd into artful-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
Changed in systemd (Ubuntu Artful): | |
status: | In Progress → Fix Committed |
tags: | added: verification-needed verification-needed-artful |
Scott Moser (smoser) wrote : | #19 |
tags: |
added: verification-done verification-done-artful removed: verification-needed verification-needed-artful |
Scott Moser (smoser) wrote : | #20 |
See my attached log for verification of artful.
Launchpad Janitor (janitor) wrote : | #21 |
This bug was fixed in the package systemd - 234-2ubuntu12.3
---------------
systemd (234-2ubuntu12.3) artful; urgency=medium
[ Dimitri John Ledkov ]
* Fix test-functions failing with Ubuntu units. LP: #1750608
* tests: switch to using ext4 by default, instead of ext3. LP: #1750608
* Fix kdump service not starting, due to systemd not loading dropins.
Cherrypick a fix from upstream. (LP: #1708409)
* systemd-fsckd: Fix ADT tests to work on s390x too. (LP: #1736955)
* netwokrd: add support for RequiredForOnline stanza. (LP: #1737570)
* resolved.service: set DefaultDependen
* systemd.postinst: enable persistent journal. (LP: #1618188)
* core: add support for non-writable unified cgroup hierarchy for container support.
Rebase and de-fuzz. (LP: #1734410)
* Prevent MemoryDenyWrite
CVE-2017-15908 (LP: #1725348)
* networkd: enable promote_secondaries on networkd managed dhcp links.
This fixes failing to renew DHCP lease, on networkd managed devices.
(LP: #1721223)
[ Kleber Sacilotto de Souza ]
* systemd-rfkill service times out when a new rfkill device is added
- rfkill-
udev_
check the device received from udev monitor instead.
- rfkill-
-- Dimitri John Ledkov <email address hidden> Tue, 20 Feb 2018 16:11:58 +0000
Changed in systemd (Ubuntu Artful): | |
status: | Fix Committed → Fix Released |
The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
Changed in cloud-init: | |
status: | Confirmed → Fix Released |
Changed in cloud-init (Ubuntu Artful): | |
status: | Confirmed → Won't Fix |
Entire command lines of how I'm doing this:
build@nestvirt:~$ qemu-img create -f qcow2 -b artful- server- cloudimg- amd64.img cloudy.img 20G zesty,accel= kvm,usb= off,dump- guest-core= off -m 4096 -smp 3 -cpu Opteron_G3 -device virtio- net-pci, netdev= hostnet0, id=net0, mac=52: 54:00:31: 33:70,bus= pci.0,addr= 0x3 -netdev bridge,id=hostnet0 -drive file=cloudy. img,if= virtio -smbios "type=1, serial= ds=nocloud- net;s=https:/ /raw.githubuser content. com/mlyle/ mlyle/master/ cloud-metadata/ linuxtst/" -kernel bzImage -append "root=/dev/vda1 ro console=ttyS0"
build@nestvirt:~$ kvm -nographic -machine pc-i440fx-