bionic LXD containers on bionic hosts get incorrect /etc/resolve.conf files
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
High
|
Eric Claude Jones | ||
2.3 |
Fix Released
|
High
|
Eric Claude Jones |
Bug Description
I just tried:
juju bootstrap --bootstrap-
juju deploy cs:~jameinel/
After doing so, the container fails to start up because it "cannot resolve archive.ubuntu.com" (note I think we've seen this in CI runs as well).
However, the reason for that is because the host machine has this /etc/resolve.conf:
nameserver 127.0.0.53
search maas
Presumably this means we're running a local DNS proxy that is actually itself configured to talk to MAAS for any more information.
However, when launching a container, we read the host machines DNS information if we don't have any other information. But it is very clear that 127.0.0.53 is not going to be available from inside the container.
It appears that 127.0.0.53 is being created from systemd-resolved. (just looking around at other bugs like: https:/
I don't see the other config files for "resolved.conf" namely
/etc/systemd/
/etc/systemd/
/run/systemd/
/usr/lib/
I did find on the host system:
/run/systemd/
nameserver 10.0.0.1
search maas
I injected that inside the container, and then ran
systemctl restart systemd-resolved
And then I was able to get:
# systemctl status systemd-resolved
...
Apr 16 07:53:20 juju-930c9c-0-lxd-0 systemd-
Apr 16 07:53:20 juju-930c9c-0-lxd-0 systemd-
Apr 16 07:53:20 juju-930c9c-0-lxd-0 systemd[1]: Started Network Name Resolution.
However,
root@juju-
;; connection timed out; no servers could be reached
While this does work:
# host archive.ubuntu.com 10.0.0.1
Using domain server:
Name: 10.0.0.1
Address: 10.0.0.1#53
Aliases:
archive.ubuntu.com has address 91.189.88.152
archive.ubuntu.com has address 91.189.88.149
archive.ubuntu.com has address 91.189.88.161
archive.ubuntu.com has address 91.189.88.162
archive.ubuntu.com has IPv6 address 2001:67c:
archive.ubuntu.com has IPv6 address 2001:67c:
archive.ubuntu.com has IPv6 address 2001:67c:
archive.ubuntu.com has IPv6 address 2001:67c:
From what I can tell, systemd-resolved might read /etc/resolv.conf on startup if it is not a symlink to /run/systemd/
Running on the host machine I see:
c# systemd-resolve --status
Global
DNSSEC NTA: 10.in-addr.arpa
Link 6 (vethXG5VHE)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 4 (br-enp0s25)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: 10.0.0.1
DNS Domain: maas
Link 3 (lxdbr0)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 2 (enp0s25)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Those seem to be set up by juju in the 'netplan' configuration at:
/etc/netplan/
network:
version: 2
ethernets:
enp0s25:
match:
macaddress: b8:ae:ed:79:c7:92
set-name: enp0s25
mtu: 1500
bridges:
br-enp0s25:
interfaces: [enp0s25]
addresses:
- 10.0.0.156/24
gateway4: 10.0.0.1
nameservers:
search: [maas]
addresses: [10.0.0.1]
mtu: 1500
However, inside the container we end up with:
network:
version: 2
ethernets:
eth0:
match:
macaddress: 00:16:3e:4a:39:18
addresses:
- 10.0.0.26/24
gateway4: 10.0.0.1
nameservers:
search: [maas]
addresses: [127.0.0.53]
Editing /etc/netplan/
netplan generate
netplan apply
Does show the container getting the right dns server to forward to.
So we need to figure out how we are getting the right DNS server for the host machine, so that we can put it into the host's 99-juju.yaml, but we are overriding that value with the host's /etc/resolve.conf which is no longer the right value.
I'm guessing we'll also be doing the wrong thing for a KVM container.
tags: | added: uosci |
Changed in juju: | |
assignee: | nobody → Eric Claude Jones (ecjones) |
Changed in juju: | |
milestone: | none → 2.4-beta2 |
Changed in juju: | |
status: | Fix Committed → Fix Released |
To conirm: this is impacting Bionic with OpenStack Queens. Some workloads are sensitive to dns failures, as they expect basic a/ptr record resolution sanity (nova-cloud- controller, rabbitmq, and I think even non-OpenStack workloads such as hadoop and k8s).