Incorrect DNS (resolv.conf) setup inside docker images

[Daniel Moisset]

I've encountered a problem after upgrading docker.io to 18.09.2-0ubuntu1~18.04.1 . I currently workarounded it by downgrading back to 18.06.1-0ubuntu1.2~18.04.1 , so I can verify that it's related to it.

In my current setup, I'm using ubuntu 18.04 and connecting from home to a local ISP, and then using a VPN to access some private resources from the company I work for (I use globalprotect as VPN software). I have no connectivity issues whatsoever from my OS. Before connecting to the VPN, my /etc/resolv.conf looks like

# This file is managed by man:systemd-resolved(8). Do not edit.
# A lot of comments cut-out
nameserver 217.169.20.20
nameserver 217.169.20.21

After connecting to the VPN I get this:

search ml.com adminml.com mercadolibre.com
nameserver 10.x.x.x # I edited out the actual IPs, I'm not allowed to share them
nameserver 10.x.x.x

After upgrading to docker.io 18.09.2, when I start a docker container (using docker run, and an ubuntu 16.04 image inside the container), the container has my ISP config (ie the 217.x.x.x nameservers) in its /etc/resolv.conf, while the older version (docker.io 18.06.1) uses the correct VPN configuration (with the search domains and the 10.x.x.x address). This new behaviour makes my containers unusable (given that I expect to access some VPN-only resources from the software running inside them)

Some extra information I found while trying to diagnose this:
- my host system /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf
- the "old" (217.x.x.x) resolver configuration seems to be kept at /run/systemd/resolve/resolv.conf
- docker 18.09.2 seems to be copying this old configuration; if I 1) edit the /run/systemd/resolve/resolv.conf adding a comment 2) docker -ti run myimage 3) cat /etc/resolv.conf , then I see the comment I added. This doesn't happen in docker.io 18.06.1

I expect the old behaviour to be kept, can you confirm it is a bug?

[Michael Hudson-Doyle (mwhudson)]

This looks likely to be the result of https://github.com/moby/moby/pull/37485

On the face of it, it looks like your VPN software is stomping over the systemd-resolved config. I admin to finding DNS config fairly confusing but it seems unlikely that this is going to lead to happiness. How are you configuring your VPN?

[Daniel Moisset (dmoisset-meli)]

I'm connecting to the VPN through Palo Alto Networks's globalprotect (https://www.paloaltonetworks.com/products/globalprotect/overview). I just run "globalprotect connect --portal some.company.address.com -u my_vpn_user" (so I assume most of the network config was bundled in the installer I got)

Is there any way to diagnose if the VPN software is the offender? and if that's the case, any workaround you think I could use (because I have no choice to choose a different VPN software)

[Tianon Gravi (tianon)]

Have you tried restarting the Docker daemon? (IIRC it caches the DNS settings during startup)

[Daniel Moisset (dmoisset-meli)]

Yes, I tried that while connected to the vpn with no effect

On Mon, 18 Mar 2019 at 13:05, Tianon Gravi <email address hidden> wrote:

> Have you tried restarting the Docker daemon? (IIRC it caches the DNS
> settings during startup)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1820278
>
> Title:
> Incorrect DNS (resolv.conf) setup inside docker images
>
> Status in docker.io package in Ubuntu:
> New
>
> Bug description:
> I've encountered a problem after upgrading docker.io to
> 18.09.2-0ubuntu1~18.04.1 . I currently workarounded it by downgrading
> back to 18.06.1-0ubuntu1.2~18.04.1 , so I can verify that it's related
> to it.
>
> In my current setup, I'm using ubuntu 18.04 and connecting from home
> to a local ISP, and then using a VPN to access some private resources
> from the company I work for (I use globalprotect as VPN software). I
> have no connectivity issues whatsoever from my OS. Before connecting
> to the VPN, my /etc/resolv.conf looks like
>
> # This file is managed by man:systemd-resolved(8). Do not edit.
> # A lot of comments cut-out
> nameserver 217.169.20.20
> nameserver 217.169.20.21
>
> After connecting to the VPN I get this:
>
> search ml.com adminml.com mercadolibre.com
> nameserver 10.x.x.x # I edited out the actual IPs, I'm not allowed to
> share them
> nameserver 10.x.x.x
>
> After upgrading to docker.io 18.09.2, when I start a docker container
> (using docker run, and an ubuntu 16.04 image inside the container),
> the container has my ISP config (ie the 217.x.x.x nameservers) in its
> /etc/resolv.conf, while the older version (docker.io 18.06.1) uses the
> correct VPN configuration (with the search domains and the 10.x.x.x
> address). This new behaviour makes my containers unusable (given that
> I expect to access some VPN-only resources from the software running
> inside them)
>
> Some extra information I found while trying to diagnose this:
> - my host system /etc/resolv.conf is a symlink to
> ../run/systemd/resolve/stub-resolv.conf
> - the "old" (217.x.x.x) resolver configuration seems to be kept at
> /run/systemd/resolve/resolv.conf
> - docker 18.09.2 seems to be copying this old configuration; if I 1)
> edit the /run/systemd/resolve/resolv.conf adding a comment 2) docker -ti
> run myimage 3) cat /etc/resolv.conf , then I see the comment I added. This
> doesn't happen in docker.io 18.06.1
>
> I expect the old behaviour to be kept, can you confirm it is a bug?
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1820278/+subscriptions
>

--
--
Daniel F. Moisset - Consultor

[Tianon Gravi (tianon)]

Anything useful in Docker's daemon logs?

As a workaround you should be able to set --dns on your container or adjust /etc/docker/daemon.json to adjust that setting more permanently.

[Michael Hudson-Doyle (mwhudson)]

I assume there is "systemd-resolved is running, so using resolvconf: /run/systemd/resolve/resolv.conf" in the deamon logs, or I don't understand what is going on at all.

I think there is a docker bug upstream here: if it sees a systemd-resolved process and /run/systemd/resolve/resolv.conf exists, it assumes systemd-resolved and copies /run/systemd/resolve/resolv.conf into the container. But it doesn't check that /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf, which is probably should.

Although... reading your report again, I wanted to check, when you activate the VPN does the /etc/resolv.conf symlink get replaced with a regular file? Or does the VPN software mutate /run/systemd/resolve/stub-resolv.conf? Because if it's the latter I'm not sure what docker can really do.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Trent Lloyd (lathiat)]

I ran into this on upgrade today.

From reading above, I think the root of the problem is that I am using a system upgraded from Xenial to Bionic, so I am still using ifupdown and resolvconf (and not netplan)

ifupdown doesn't update systemd-resoved, resolvconf/ifupdown updated /etc/resolv.conf to add my IP address, but /run/systemd/resolve/resolv.conf has no servers listed

lrwxrwxrwx 1 root root 29 Oct 12 2017 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

[Trent Lloyd (lathiat)]

I stopped systemd-resolved, disabled it, restarted docker, no change.

I then removed /etc/resolv.conf's symlink and placed my own file there, restarted docker, still no change.