network-manager 1.10.14-0ubuntu2 ignores systemd-resolved configured dns

[Mal]

On 18.04.2 the `upgrade network-manager:amd64 1.10.6-2ubuntu1.1 1.10.14-0ubuntu2` lead to scoped DNS servers defined in `/etc/systemd/resolved.conf.d/*.conf` being ignored.

Downgrading with `sudo apt-get install network-manager=1.10.6-2ubuntu1.1` has resolved the issue for now.

Example systemd-resolved conf:

[Resolve]
Cache=no
DNS=127.0.0.54
Domains=~.local.org.com

Where 127.0.0.54:53 is bound to a dnsmasq server capable of resolving queries in that subdomain.

[Brandon Rochon (br0ch0n)]

I have a similar setup and am experiencing the same thing since the update. Thanks for narrowing it down to network-manager!

[Till Kamppeter (till-kamppeter)]

Looks like that the new version of network-manager is not working correctly with the systemd-resolved of Bionic.

[Till Kamppeter (till-kamppeter)]

Are these DNS servers listed in the output of

systemd-resolve --status

If yes, could you try the updated network-manager in combination with the proposed systemd update of bug 1754671?

[Mal (madmalibu)]

Yup, it shows up at the top of the global resolver list:

$ systemd-resolve --status
Global
         DNS Servers: 127.0.0.54
         DNSSEC NTA: [...]

Unfortunately, I'm not au fait with debdiffs, and don't have the bandwidth to get into world of compiling custom packages at the moment. :(

[Till Kamppeter (till-kamppeter)]

Unfortunately, the SRU for systemd did not yet get processed. Therefore I have now uploaded this version of systemd to my PPA:

https://launchpad.net/~till-kamppeter/+archive/ubuntu/ppa

Please follow this link, follow the instructions in the section "Adding this PPA to your system", then update your system with the command

sudo apt dist-upgrade

This will update only systemd as I did not upload any other package for Bionic to my PPA.

Make also sure you have the update of network-manager (1.10.14-0ubuntu2) installed. Reboot and check whether everything works correctly now.

[Steve Langasek (vorlon)]

Due to the SRU regressions reported in LP: #1829838 and LP: #1829566, I have reverted this SRU for the moment, restoring network-manager 1.10.6-2ubuntu1.1 to bionic-updates.

network-manager 1.10.14-0ubuntu2 remains available in bionic-proposed for testing purposes.

[Mal (madmalibu)]

Installed the systemd from the ppa and the .14 NM package:

ii network-manager 1.10.14-0ubuntu2
ii systemd 237-3ubuntu10.22~ppa1

Rebooted, gave it a spin and no good, I'm afraid. The 127.0.0.54 resolver was still left out of the loop. I've since gone back to the public systemd version and once again downgraded NM to restore functionality as I need it day-to-day, sorry there isn't better news.

[Till Kamppeter (till-kamppeter)]

I am rather new to network-manager internals, but could you try the command

sudo nmcli con modify "$COMPANY VPN" ipv4.dns-priority -1 ipv4.dns-search ~.

Does this solve your problem?

[Till Kamppeter (till-kamppeter)]

1. /etc/systemd/journald.d/noratelimit.conf containing

RateLimitIntervalSec=0
RateLimitBurst=0

2. /etc/NetworkManager/conf.d/debug.conf

[logging]
level=TRACE
domains=ALL

Then restart journald:

sudo systemctl restart systemd-journald

and NetworkManager:

sudo systemctl restart network-manager

Then you get the full debug log of NetworkManager via

journalctl -u NetworkManager

After all that, reboot and/or connect to your VPN and do

journalctl -u NetworkManager > log.txt

and attach the log.txt file to this bug report. Do not compress the file and do not package it together with other files.

[Till Kamppeter (till-kamppeter)]

Sorry there was a part missing. Let us try again:

Please create the following files (and directories if needed for them):

1. /etc/systemd/journald.d/noratelimit.conf containing

RateLimitIntervalSec=0
RateLimitBurst=0

2. /etc/NetworkManager/conf.d/debug.conf

[logging]
level=TRACE
domains=ALL

Then restart journald:

sudo systemctl restart systemd-journald

and NetworkManager:

sudo systemctl restart network-manager

Then you get the full debug log of NetworkManager via

journalctl -u NetworkManager

After all that, reboot and/or connect to your VPN and do

journalctl -u NetworkManager > log.txt

and attach the log.txt file to this bug report. Do not compress the file and do not package it together with other files.

[Till Kamppeter (till-kamppeter)]

The SRU for systemd has arrived in bionic-proposed (see bug 1754671). Could you make sure that you have installed BOTH the network-manager and systemd SRUs from bionic-proposed (to make sure that I did not perhaps do something wrong with the systemd update in my PPA). Versions should be:

network-manager: 1.10.14-0ubuntu2
systemd: 237-3ubuntu10.22

Does this combination solve your problem? Please reboot to make sure that everything gets replaced by the newer versions.

If this still does not help, please follow my instructions in comment #8 and comment #10.

[Mal (madmalibu)]

Can only attach one file at a time, so this and the next comment are heavily linked.

- Updated to systemd 237-3ubuntu10.22 (from bionic-updates)
- Enabled debug logging for network manager and disabled flood protection on journald.
- Rebooted and successfully ran dig (second attempt)
  - First run did not return expected IP.
  - Second run onwards resolved as expected.
- Log file from reboot until after the second dig attached.

[Mal (madmalibu)]

Process follows directly from previous comment.

- Updated to network-manager 1.10.14-0ubuntu2 (from bionic-proposed)
- Rebooted and ran dig twice to mirror previous test, never resolved to expected result. :(
- Log file from reboot until after the second dig attached.

Ran dig a few more times over a couple of minutes, but never resolved successfully.

[Till Kamppeter (till-kamppeter)]

Mal, thanks for the logs, could you tell me, to make it easier for me to find what went wrong, tell me which host names you queried with "dig", which IP they should return, and through which DNS? Thank you.

[mzmrk (mzmrk)]

I have some extra information that might lead you into the right direction.

I used the following extra line in my openvpn connection config file (/etc/NetworkManager/system-connections/vpn_connection_name):
[ipv4]
dns-priority=-1

After the update DNS received from OpenVPN connection was not respected anymore.

Today I replaced that line with the following:
[ipv4]
dns-search=~

with following versions:
network-manager: 1.10.14-0ubuntu2
systemd: 237-3ubuntu10.22

Now my DNS is correctly updated after connecting to OpenVPN.

[mzmrk (mzmrk)]

Some clarification:
1. dns-priority=-1 used to work (openvpn had highest DNS priority over other connections) before the regression happend.
2. I rebooted my system after changing config to dns-search=~

[Mal (madmalibu)]

Here you are, Till. Three sections, one for each NM version containing the two dig requests during the tests, plus one for a dig to contact the local DNS explicitly.

nm-1.10.6:

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER' # first attempt always fails
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 127.0.0.1

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER' # second onwards always works
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 172.22.0.2

nm-1.10.14:

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER'
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 127.0.0.1 # first attempt always fails

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER'
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 127.0.0.1 # all subsequent attempts fail too :(

Direct dig example:

$ dig dnsmasq.local.org.com @127.0.0.54 | grep -FA1 ';; ANSWER' # direct query to local dns specified in config (see main issue body)
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 172.22.0.2

[Till Kamppeter (till-kamppeter)]

I asked on the upstream mailing list and got the following answer:

----------
Hi,

the domain specification in the configuration reported on launchpad:

 [Resolve]
 Cache=no
 DNS=127.0.0.54
 Domains=~.local.org.com

is invalid:

 systemd-resolved[13415]: Failed to add search domain '~.local.org.com', ignoring: Invalid argument

The correct form should be without the dot:

 Domains=~local.org.com

Can you ask to fix that configuration error and try again? Otherwise,
queries for the local.org.com domain are not routed to the local
dnsmasq instance.

Beniamino
----------

Could you correct your configuration and try again? Please report back here, thanks.

[Till Kamppeter (till-kamppeter)]

Mal, regarding your log files of comments #12 and #13, did you do the booting and the dig commands in less than 1 minute for each NM version? Each of the logs spans a time frame of little less than 1 minute?

Could you also repeat your tests after correcting your configuration according to comment #18? Thanks.

[Till Kamppeter (till-kamppeter)]

Mal, could you also provide logs of sytemd-resolved (if needed complete journal) for your tests?

[Mal (madmalibu)]

Sorry, I've been a bit swamped and this fell through the cracks, I'll try and find some time this week to try with the updated config.

In answer to your previous question: yes, the dig was the first thing I did following a reboot each time with the intention of providing the smallest number of log lines that would need to be examined but during which the problem had definitely occurred.

I'll let you know the results of trying with the tweaked config ASAP.

[Till Kamppeter (till-kamppeter)]

Please also run the command

systemd-resolve --status

and post the output here for each of your tests.

[Till Kamppeter (till-kamppeter)]

Best would even be if you could run the tests as shown in the "[Test case]" section of the description of bug 1754671, but in addition capture the full journal with all messages of systemd-resolved.

[Mal (madmalibu)]

Well ... that configuration correction appears to have done the trick! I'm very sorry to have inadvertently put you through this for the sake an off-by-one-character invalid config. Re-reading the documentation I can now see where the confusion occurred when crafting the config originally, and I guess until now I was just unknowingly benefiting from a non-spec compliant edge-case in the previous version or some other chance alignment of stars. Cue the facepalm.

Here are the package versions and the updated config I've just tested successfully with:

ii network-manager 1.10.14-0ubuntu2
ii systemd 237-3ubuntu10.24

/etc/systemd/resolved.conf.d/local.org.com.conf
[Resolve]
Cache=no
DNS=127.0.0.54
Domains=~local.org.com

Huge thanks for sticking with this, it's really appreciated, and happy to provide any further information if you still feel there are questions here that need answering, but if not, maybe we can call this a wrap?

[Till Kamppeter (till-kamppeter)]

Does this mean that for you there is now no regression in the Bionic SRU of Network Manager (1.10.14-0ubuntu2)? Can I mark this bug report as "Invalid" then?

[Mal (madmalibu)]

Correct, the "regression", such as it was, seemingly exists only as a change to how that invalid systemd-resolved conf was handled. I did go back and verify that the invalid config did function on the old version, but no on the new, so there is a functional change there, but given the invalidity in the config I've no objection to marking this bug as Invalid.

[Till Kamppeter (till-kamppeter)]

OK, then this is actually no regression. Closing ...

[Till Kamppeter (till-kamppeter)]

Thank you very much for your great cooperation. I wish other bug reporters do so, too.