network-manager 1.10.14-0ubuntu2 ignores systemd-resolved configured dns

Bug #1829566 reported by Mal on 2019-05-17
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
network-manager (Ubuntu)
High
Unassigned
Bionic
High
Till Kamppeter

Bug Description

On 18.04.2 the `upgrade network-manager:amd64 1.10.6-2ubuntu1.1 1.10.14-0ubuntu2` lead to scoped DNS servers defined in `/etc/systemd/resolved.conf.d/*.conf` being ignored.

Downgrading with `sudo apt-get install network-manager=1.10.6-2ubuntu1.1` has resolved the issue for now.

Example systemd-resolved conf:

[Resolve]
Cache=no
DNS=127.0.0.54
Domains=~.local.org.com

Where 127.0.0.54:53 is bound to a dnsmasq server capable of resolving queries in that subdomain.

Revision history for this message
Brandon Rochon (br0ch0n) wrote :

I have a similar setup and am experiencing the same thing since the update. Thanks for narrowing it down to network-manager!

Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Looks like that the new version of network-manager is not working correctly with the systemd-resolved of Bionic.

tags: added: regression-update
Changed in network-manager (Ubuntu):
importance: Undecided → High
Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Are these DNS servers listed in the output of

systemd-resolve --status

If yes, could you try the updated network-manager in combination with the proposed systemd update of bug 1754671?

Revision history for this message
Mal (madmalibu) wrote :

Yup, it shows up at the top of the global resolver list:

$ systemd-resolve --status
Global
         DNS Servers: 127.0.0.54
         DNSSEC NTA: [...]

Unfortunately, I'm not au fait with debdiffs, and don't have the bandwidth to get into world of compiling custom packages at the moment. :(

Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Unfortunately, the SRU for systemd did not yet get processed. Therefore I have now uploaded this version of systemd to my PPA:

https://launchpad.net/~till-kamppeter/+archive/ubuntu/ppa

Please follow this link, follow the instructions in the section "Adding this PPA to your system", then update your system with the command

sudo apt dist-upgrade

This will update only systemd as I did not upload any other package for Bionic to my PPA.

Make also sure you have the update of network-manager (1.10.14-0ubuntu2) installed. Reboot and check whether everything works correctly now.

Changed in network-manager (Ubuntu Bionic):
status: New → Incomplete
Changed in network-manager (Ubuntu):
status: New → Incomplete
Revision history for this message
Steve Langasek (vorlon) wrote :

Due to the SRU regressions reported in LP: #1829838 and LP: #1829566, I have reverted this SRU for the moment, restoring network-manager 1.10.6-2ubuntu1.1 to bionic-updates.

network-manager 1.10.14-0ubuntu2 remains available in bionic-proposed for testing purposes.

Revision history for this message
Mal (madmalibu) wrote :

Installed the systemd from the ppa and the .14 NM package:

ii network-manager 1.10.14-0ubuntu2
ii systemd 237-3ubuntu10.22~ppa1

Rebooted, gave it a spin and no good, I'm afraid. The 127.0.0.54 resolver was still left out of the loop. I've since gone back to the public systemd version and once again downgraded NM to restore functionality as I need it day-to-day, sorry there isn't better news.

Mathew Hodson (mhodson) on 2019-05-25
Changed in network-manager (Ubuntu):
status: Incomplete → Confirmed
Changed in network-manager (Ubuntu Bionic):
status: Incomplete → Confirmed
importance: Undecided → High
tags: added: bionic
Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

I am rather new to network-manager internals, but could you try the command

sudo nmcli con modify "$COMPANY VPN" ipv4.dns-priority -1 ipv4.dns-search ~.

Does this solve your problem?

Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

1. /etc/systemd/journald.d/noratelimit.conf containing

RateLimitIntervalSec=0
RateLimitBurst=0

2. /etc/NetworkManager/conf.d/debug.conf

[logging]
level=TRACE
domains=ALL

Then restart journald:

sudo systemctl restart systemd-journald

and NetworkManager:

sudo systemctl restart network-manager

Then you get the full debug log of NetworkManager via

journalctl -u NetworkManager

After all that, reboot and/or connect to your VPN and do

journalctl -u NetworkManager > log.txt

and attach the log.txt file to this bug report. Do not compress the file and do not package it together with other files.

Changed in network-manager (Ubuntu Bionic):
status: Confirmed → Incomplete
Changed in network-manager (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Sorry there was a part missing. Let us try again:

Please create the following files (and directories if needed for them):

1. /etc/systemd/journald.d/noratelimit.conf containing

RateLimitIntervalSec=0
RateLimitBurst=0

2. /etc/NetworkManager/conf.d/debug.conf

[logging]
level=TRACE
domains=ALL

Then restart journald:

sudo systemctl restart systemd-journald

and NetworkManager:

sudo systemctl restart network-manager

Then you get the full debug log of NetworkManager via

journalctl -u NetworkManager

After all that, reboot and/or connect to your VPN and do

journalctl -u NetworkManager > log.txt

and attach the log.txt file to this bug report. Do not compress the file and do not package it together with other files.

Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

The SRU for systemd has arrived in bionic-proposed (see bug 1754671). Could you make sure that you have installed BOTH the network-manager and systemd SRUs from bionic-proposed (to make sure that I did not perhaps do something wrong with the systemd update in my PPA). Versions should be:

network-manager: 1.10.14-0ubuntu2
systemd: 237-3ubuntu10.22

Does this combination solve your problem? Please reboot to make sure that everything gets replaced by the newer versions.

If this still does not help, please follow my instructions in comment #8 and comment #10.

Revision history for this message
Mal (madmalibu) wrote :

Can only attach one file at a time, so this and the next comment are heavily linked.

- Updated to systemd 237-3ubuntu10.22 (from bionic-updates)
- Enabled debug logging for network manager and disabled flood protection on journald.
- Rebooted and successfully ran dig (second attempt)
  - First run did not return expected IP.
  - Second run onwards resolved as expected.
- Log file from reboot until after the second dig attached.

Revision history for this message
Mal (madmalibu) wrote :

Process follows directly from previous comment.

- Updated to network-manager 1.10.14-0ubuntu2 (from bionic-proposed)
- Rebooted and ran dig twice to mirror previous test, never resolved to expected result. :(
- Log file from reboot until after the second dig attached.

Ran dig a few more times over a couple of minutes, but never resolved successfully.

Mathew Hodson (mhodson) on 2019-06-12
tags: added: regression-proposed
removed: regression-update
Changed in network-manager (Ubuntu):
status: Incomplete → Triaged
Changed in network-manager (Ubuntu Bionic):
status: Incomplete → Triaged
Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Mal, thanks for the logs, could you tell me, to make it easier for me to find what went wrong, tell me which host names you queried with "dig", which IP they should return, and through which DNS? Thank you.

Revision history for this message
mzmrk (mzmrk) wrote :

I have some extra information that might lead you into the right direction.

I used the following extra line in my openvpn connection config file (/etc/NetworkManager/system-connections/vpn_connection_name):
[ipv4]
dns-priority=-1

After the update DNS received from OpenVPN connection was not respected anymore.

Today I replaced that line with the following:
[ipv4]
dns-search=~

with following versions:
network-manager: 1.10.14-0ubuntu2
systemd: 237-3ubuntu10.22

Now my DNS is correctly updated after connecting to OpenVPN.

Revision history for this message
mzmrk (mzmrk) wrote :

Some clarification:
1. dns-priority=-1 used to work (openvpn had highest DNS priority over other connections) before the regression happend.
2. I rebooted my system after changing config to dns-search=~

Revision history for this message
Mal (madmalibu) wrote :

Here you are, Till. Three sections, one for each NM version containing the two dig requests during the tests, plus one for a dig to contact the local DNS explicitly.

nm-1.10.6:

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER' # first attempt always fails
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 127.0.0.1

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER' # second onwards always works
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 172.22.0.2

nm-1.10.14:

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER'
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 127.0.0.1 # first attempt always fails

$ dig dnsmasq.local.org.com | grep -FA1 ';; ANSWER'
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 127.0.0.1 # all subsequent attempts fail too :(

Direct dig example:

$ dig dnsmasq.local.org.com @127.0.0.54 | grep -FA1 ';; ANSWER' # direct query to local dns specified in config (see main issue body)
;; ANSWER SECTION:
dnsmasq.local.org.com. 600 IN A 172.22.0.2

Changed in network-manager (Ubuntu Bionic):
assignee: nobody → Till Kamppeter (till-kamppeter)
Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

I asked on the upstream mailing list and got the following answer:

----------
Hi,

the domain specification in the configuration reported on launchpad:

 [Resolve]
 Cache=no
 DNS=127.0.0.54
 Domains=~.local.org.com

is invalid:

 systemd-resolved[13415]: Failed to add search domain '~.local.org.com', ignoring: Invalid argument

The correct form should be without the dot:

 Domains=~local.org.com

Can you ask to fix that configuration error and try again? Otherwise,
queries for the local.org.com domain are not routed to the local
dnsmasq instance.

Beniamino
----------

Could you correct your configuration and try again? Please report back here, thanks.

Changed in network-manager (Ubuntu Bionic):
status: Triaged → Incomplete
Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Mal, regarding your log files of comments #12 and #13, did you do the booting and the dig commands in less than 1 minute for each NM version? Each of the logs spans a time frame of little less than 1 minute?

Could you also repeat your tests after correcting your configuration according to comment #18? Thanks.

Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Mal, could you also provide logs of sytemd-resolved (if needed complete journal) for your tests?

Changed in network-manager (Ubuntu):
status: Triaged → Incomplete
Revision history for this message
Mal (madmalibu) wrote :

Sorry, I've been a bit swamped and this fell through the cracks, I'll try and find some time this week to try with the updated config.

In answer to your previous question: yes, the dig was the first thing I did following a reboot each time with the intention of providing the smallest number of log lines that would need to be examined but during which the problem had definitely occurred.

I'll let you know the results of trying with the tweaked config ASAP.

Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Please also run the command

systemd-resolve --status

and post the output here for each of your tests.

Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Best would even be if you could run the tests as shown in the "[Test case]" section of the description of bug 1754671, but in addition capture the full journal with all messages of systemd-resolved.

Revision history for this message
Mal (madmalibu) wrote :

Well ... that configuration correction appears to have done the trick! I'm very sorry to have inadvertently put you through this for the sake an off-by-one-character invalid config. Re-reading the documentation I can now see where the confusion occurred when crafting the config originally, and I guess until now I was just unknowingly benefiting from a non-spec compliant edge-case in the previous version or some other chance alignment of stars. Cue the facepalm.

Here are the package versions and the updated config I've just tested successfully with:

ii network-manager 1.10.14-0ubuntu2
ii systemd 237-3ubuntu10.24

/etc/systemd/resolved.conf.d/local.org.com.conf
[Resolve]
Cache=no
DNS=127.0.0.54
Domains=~local.org.com

Huge thanks for sticking with this, it's really appreciated, and happy to provide any further information if you still feel there are questions here that need answering, but if not, maybe we can call this a wrap?

Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Does this mean that for you there is now no regression in the Bionic SRU of Network Manager (1.10.14-0ubuntu2)? Can I mark this bug report as "Invalid" then?

Revision history for this message
Mal (madmalibu) wrote :

Correct, the "regression", such as it was, seemingly exists only as a change to how that invalid systemd-resolved conf was handled. I did go back and verify that the invalid config did function on the old version, but no on the new, so there is a functional change there, but given the invalidity in the config I've no objection to marking this bug as Invalid.

Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

OK, then this is actually no regression. Closing ...

Changed in network-manager (Ubuntu):
status: Incomplete → Invalid
Changed in network-manager (Ubuntu Bionic):
status: Incomplete → Invalid
tags: removed: bionic regression-proposed
Revision history for this message
Till Kamppeter (till-kamppeter) wrote :

Thank you very much for your great cooperation. I wish other bug reporters do so, too.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers