network-manager 1.10.14-0ubuntu2 ignores systemd-resolved configured dns

Bug #1829566 reported by Mal on 2019-05-17
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
network-manager (Ubuntu)
High
Unassigned
Bionic
High
Till Kamppeter

Bug Description

On 18.04.2 the `upgrade network-manager:amd64 1.10.6-2ubuntu1.1 1.10.14-0ubuntu2` lead to scoped DNS servers defined in `/etc/systemd/resolved.conf.d/*.conf` being ignored.

Downgrading with `sudo apt-get install network-manager=1.10.6-2ubuntu1.1` has resolved the issue for now.

Example systemd-resolved conf:

[Resolve]
Cache=no
DNS=127.0.0.54
Domains=~.local.org.com

Where 127.0.0.54:53 is bound to a dnsmasq server capable of resolving queries in that subdomain.

Brandon Rochon (br0ch0n) wrote :

I have a similar setup and am experiencing the same thing since the update. Thanks for narrowing it down to network-manager!

Till Kamppeter (till-kamppeter) wrote :

Looks like that the new version of network-manager is not working correctly with the systemd-resolved of Bionic.

tags: added: regression-update
Changed in network-manager (Ubuntu):
importance: Undecided → High
Till Kamppeter (till-kamppeter) wrote :

Are these DNS servers listed in the output of

systemd-resolve --status

If yes, could you try the updated network-manager in combination with the proposed systemd update of bug 1754671?

Mal (madmalibu) wrote :

Yup, it shows up at the top of the global resolver list:

$ systemd-resolve --status
Global
         DNS Servers: 127.0.0.54
         DNSSEC NTA: [...]

Unfortunately, I'm not au fait with debdiffs, and don't have the bandwidth to get into world of compiling custom packages at the moment. :(

Till Kamppeter (till-kamppeter) wrote :

Unfortunately, the SRU for systemd did not yet get processed. Therefore I have now uploaded this version of systemd to my PPA:

https://launchpad.net/~till-kamppeter/+archive/ubuntu/ppa

Please follow this link, follow the instructions in the section "Adding this PPA to your system", then update your system with the command

sudo apt dist-upgrade

This will update only systemd as I did not upload any other package for Bionic to my PPA.

Make also sure you have the update of network-manager (1.10.14-0ubuntu2) installed. Reboot and check whether everything works correctly now.

Changed in network-manager (Ubuntu Bionic):
status: New → Incomplete
Changed in network-manager (Ubuntu):
status: New → Incomplete
Steve Langasek (vorlon) wrote :

Due to the SRU regressions reported in LP: #1829838 and LP: #1829566, I have reverted this SRU for the moment, restoring network-manager 1.10.6-2ubuntu1.1 to bionic-updates.

network-manager 1.10.14-0ubuntu2 remains available in bionic-proposed for testing purposes.

Mal (madmalibu) wrote :

Installed the systemd from the ppa and the .14 NM package:

ii network-manager 1.10.14-0ubuntu2
ii systemd 237-3ubuntu10.22~ppa1

Rebooted, gave it a spin and no good, I'm afraid. The 127.0.0.54 resolver was still left out of the loop. I've since gone back to the public systemd version and once again downgraded NM to restore functionality as I need it day-to-day, sorry there isn't better news.

Changed in network-manager (Ubuntu):
status: Incomplete → Confirmed
Changed in network-manager (Ubuntu Bionic):
status: Incomplete → Confirmed
importance: Undecided → High
tags: added: bionic
Till Kamppeter (till-kamppeter) wrote :

I am rather new to network-manager internals, but could you try the command

sudo nmcli con modify "$COMPANY VPN" ipv4.dns-priority -1 ipv4.dns-search ~.

Does this solve your problem?

Till Kamppeter (till-kamppeter) wrote :

1. /etc/systemd/journald.d/noratelimit.conf containing

RateLimitIntervalSec=0
RateLimitBurst=0

2. /etc/NetworkManager/conf.d/debug.conf

[logging]
level=TRACE
domains=ALL

Then restart journald:

sudo systemctl restart systemd-journald

and NetworkManager:

sudo systemctl restart network-manager

Then you get the full debug log of NetworkManager via

journalctl -u NetworkManager

After all that, reboot and/or connect to your VPN and do

journalctl -u NetworkManager > log.txt

and attach the log.txt file to this bug report. Do not compress the file and do not package it together with other files.

Changed in network-manager (Ubuntu Bionic):
status: Confirmed → Incomplete
Changed in network-manager (Ubuntu):
status: Confirmed → Incomplete

Sorry there was a part missing. Let us try again:

Please create the following files (and directories if needed for them):

1. /etc/systemd/journald.d/noratelimit.conf containing

RateLimitIntervalSec=0
RateLimitBurst=0

2. /etc/NetworkManager/conf.d/debug.conf

[logging]
level=TRACE
domains=ALL

Then restart journald:

sudo systemctl restart systemd-journald

and NetworkManager:

sudo systemctl restart network-manager

Then you get the full debug log of NetworkManager via

journalctl -u NetworkManager

After all that, reboot and/or connect to your VPN and do

journalctl -u NetworkManager > log.txt

and attach the log.txt file to this bug report. Do not compress the file and do not package it together with other files.

The SRU for systemd has arrived in bionic-proposed (see bug 1754671). Could you make sure that you have installed BOTH the network-manager and systemd SRUs from bionic-proposed (to make sure that I did not perhaps do something wrong with the systemd update in my PPA). Versions should be:

network-manager: 1.10.14-0ubuntu2
systemd: 237-3ubuntu10.22

Does this combination solve your problem? Please reboot to make sure that everything gets replaced by the newer versions.

If this still does not help, please follow my instructions in comment #8 and comment #10.

Mal (madmalibu) wrote :

Can only attach one file at a time, so this and the next comment are heavily linked.

- Updated to systemd 237-3ubuntu10.22 (from bionic-updates)
- Enabled debug logging for network manager and disabled flood protection on journald.
- Rebooted and successfully ran dig (second attempt)
  - First run did not return expected IP.
  - Second run onwards resolved as expected.
- Log file from reboot until after the second dig attached.

Mal (madmalibu) wrote :

Process follows directly from previous comment.

- Updated to network-manager 1.10.14-0ubuntu2 (from bionic-proposed)
- Rebooted and ran dig twice to mirror previous test, never resolved to expected result. :(
- Log file from reboot until after the second dig attached.

Ran dig a few more times over a couple of minutes, but never resolved successfully.

tags: added: regression-proposed
removed: regression-update
Changed in network-manager (Ubuntu):
status: Incomplete → Triaged
Changed in network-manager (Ubuntu Bionic):
status: Incomplete → Triaged

Mal, thanks for the logs, could you tell me, to make it easier for me to find what went wrong, tell me which host names you queried with "dig", which IP they should return, and through which DNS? Thank you.

mzmrk (mzmrk) wrote :

I have some extra information that might lead you into the right direction.

I used the following extra line in my openvpn connection config file (/etc/NetworkManager/system-connections/vpn_connection_name):
[ipv4]
dns-priority=-1

After the update DNS received from OpenVPN connection was not respected anymore.

Today I replaced that line with the following:
[ipv4]
dns-search=~

with following versions:
network-manager: 1.10.14-0ubuntu2
systemd: 237-3ubuntu10.22

Now my DNS is correctly updated after connecting to OpenVPN.

mzmrk (mzmrk) wrote :

Some clarification:
1. dns-priority=-1 used to work (openvpn had highest DNS priority over other connections) before the regression happend.
2. I rebooted my system after changing config to dns-search=~

Mal (madmalibu) wrote :

Here you are, Till. Three sections, one for each NM version containing the two dig requests during the tests, plus one for a dig to contact the local DNS explicitly.

nm-1.10.6:

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER' # first attempt always fails
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 127.0.0.1

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER' # second onwards always works
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 172.22.0.2

nm-1.10.14:

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER'
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 127.0.0.1 # first attempt always fails

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER'
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 127.0.0.1 # all subsequent attempts fail too :(

Direct dig example:

$ dig dnsmasq.local.org.com @127.0.0.54 | grep -FA1 ';; ANSWER' # direct query to local dns specified in config (see main issue body)
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 172.22.0.2

Changed in network-manager (Ubuntu Bionic):
assignee: nobody → Till Kamppeter (till-kamppeter)

I asked on the upstream mailing list and got the following answer:

----------
Hi,

the domain specification in the configuration reported on launchpad:

 [Resolve]
 Cache=no
 DNS=127.0.0.54
 Domains=~.local.org.com

is invalid:

 systemd-resolved[13415]: Failed to add search domain '~.local.org.com', ignoring: Invalid argument

The correct form should be without the dot:

 Domains=~local.org.com

Can you ask to fix that configuration error and try again? Otherwise,
queries for the local.org.com domain are not routed to the local
dnsmasq instance.

Beniamino
----------

Could you correct your configuration and try again? Please report back here, thanks.

Changed in network-manager (Ubuntu Bionic):
status: Triaged → Incomplete

Mal, regarding your log files of comments #12 and #13, did you do the booting and the dig commands in less than 1 minute for each NM version? Each of the logs spans a time frame of little less than 1 minute?

Could you also repeat your tests after correcting your configuration according to comment #18? Thanks.

Mal, could you also provide logs of sytemd-resolved (if needed complete journal) for your tests?

Changed in network-manager (Ubuntu):
status: Triaged → Incomplete
Mal (madmalibu) wrote :

Sorry, I've been a bit swamped and this fell through the cracks, I'll try and find some time this week to try with the updated config.

In answer to your previous question: yes, the dig was the first thing I did following a reboot each time with the intention of providing the smallest number of log lines that would need to be examined but during which the problem had definitely occurred.

I'll let you know the results of trying with the tweaked config ASAP.

Please also run the command

systemd-resolve --status

and post the output here for each of your tests.

Best would even be if you could run the tests as shown in the "[Test case]" section of the description of bug 1754671, but in addition capture the full journal with all messages of systemd-resolved.

Mal (madmalibu) wrote :

Well ... that configuration correction appears to have done the trick! I'm very sorry to have inadvertently put you through this for the sake an off-by-one-character invalid config. Re-reading the documentation I can now see where the confusion occurred when crafting the config originally, and I guess until now I was just unknowingly benefiting from a non-spec compliant edge-case in the previous version or some other chance alignment of stars. Cue the facepalm.

Here are the package versions and the updated config I've just tested successfully with:

ii network-manager 1.10.14-0ubuntu2
ii systemd 237-3ubuntu10.24

/etc/systemd/resolved.conf.d/local.org.com.conf
[Resolve]
Cache=no
DNS=127.0.0.54
Domains=~local.org.com

Huge thanks for sticking with this, it's really appreciated, and happy to provide any further information if you still feel there are questions here that need answering, but if not, maybe we can call this a wrap?

Does this mean that for you there is now no regression in the Bionic SRU of Network Manager (1.10.14-0ubuntu2)? Can I mark this bug report as "Invalid" then?

Mal (madmalibu) wrote :

Correct, the "regression", such as it was, seemingly exists only as a change to how that invalid systemd-resolved conf was handled. I did go back and verify that the invalid config did function on the old version, but no on the new, so there is a functional change there, but given the invalidity in the config I've no objection to marking this bug as Invalid.

OK, then this is actually no regression. Closing ...

Changed in network-manager (Ubuntu):
status: Incomplete → Invalid
Changed in network-manager (Ubuntu Bionic):
status: Incomplete → Invalid
tags: removed: bionic regression-proposed

Thank you very much for your great cooperation. I wish other bug reporters do so, too.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers