Full-tunnel VPN DNS leakage regression

[dwmw2]

[Impact]
When using a VPN the DNS requests might still be sent to a DNS server outside the VPN when they should not

[Test case]
1) Set up a VPN with split tunneling:
  a) Configure VPN normally (set up remote host, any ports and options needed for the VPN to work)
  b) Under the IPv4 tab: enable "Use this connection only for the resources on its network".
  c) Under the IPv6 tab: enable "Use this connection only for the resources on its network".

2) Connect to the VPN.

3) Run 'systemd-resolve --status'; note the DNS servers configured:
  a) For the VPN; under a separate link (probably tun0), note down the IP of the DNS server(s). Also note the name of the interface (link).
  b) For the "main" connection; under the link for your ethernet or wireless devices (wl*, en*, whatever it may be), note down the IP of the DNS server(s). Also note the name of the interface (link).

4) In a separate terminal, run 'sudo tcpdump -ni <the main interface> port 53'; let it run.

5) In a separate terminal, run 'sudo tcpdump -ni <the VPN interface> port 53'; let it run.

6) In yet another terminal, issue name resolution requests using dig:
  a) For a name known to be reachable via the public network:
     'dig www.yahoo.com'
  b) For a name known to be reachable only via the VPN:
     'dig <some DNS behind the VPN>'

7) Check the output of each terminal running tcpdump. When requesting the public name, traffic can go through either. When requesting the "private" name (behind the VPN), traffic should only be going through the interface for the VPN. Additionally, ensure the IP receiving the requests for the VPN name is indeed the IP address noted above for the VPN's DNS server.

If you see no traffic showing in tcpdump output when requesting a name, it may be because it is cached by systemd-resolved. Use a different name you have not tried before.

[Regression potential]
The code change the handling of DNS servers when using a VPN, we should check that name resolution still work whne using a VPN in different configurations

-----------------

In 16.04 the NetworkManager package used to carry this patch:
http://bazaar.launchpad.net/~network-manager/network-manager/ubuntu/view/head:/debian/patches/Filter-DNS-servers-to-add-to-dnsmasq-based-on-availa.patch

It fixed the DNS setup so that when I'm on the VPN, I am not sending unencrypted DNS queries to the (potentially hostile) local nameservers.

This patch disappeared in an update. I think it was present in 1.2.2-0ubuntu0.16.04.4 but was dropped some time later.

This security bug exists upstream too: https://bugzilla.gnome.org/show_bug.cgi?id=746422
It's not a *regression* there though, as they didn't fix it yet (unfortunately!)

[Mathieu Trudel-Lapierre (cyphermox)]

Confirming this is broken. Dropping the patch 0001-dns-use-DBus-to-make-dnsmasq-nameserver-changes.patch in network-manager (1.2.4-0ubuntu0.16.04.1) was done, but it looks like not all the code in that patch was actually upstream.

[dwmw2 (dwmw2)]

This is CVE-2018-1000135. For some reason the 'Link to CVE' option above doesn't seem to work.

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1000135

[Olivier Tilloy (osomon)]

There's active work going on upstream (see https://bugzilla.gnome.org/show_bug.cgi?id=746422 and https://cgit.freedesktop.org/NetworkManager/NetworkManager/log/?h=bg/dns-bgo746422) to fix the issue.

https://bugzilla.gnome.org/show_bug.cgi?id=746422#c36 explains how.

Once in master, it would probably be doable to backport those changes (including https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=b2f306ac3d84283fdebb225079f354afb8c2a752) to the 1.10 branch, which is what's in bionic (1.10.6-2ubuntu1). Backporting to xenial (currently 1.2.6-0ubuntu0.16.04.2) would likely be an entirely different story.

[Lubomir Rintel (lkundrak)]

bg/dns-domains-bgo746422 LGTM

[Lubomir Rintel (lkundrak)]

That is, bg/dns-bgo746422 LGTM

[Bgalvani (bgalvani)]

Merged to master:

https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=d9782589248e61c0cb5aec90e3eb62612891116b

[dwmw2 (dwmw2)]

FWIW the original situation appears to be fairly broken regardless of the security situation. I've lost count of the number of times in my recent travels that the DNS servers of the airport/hotel/etc. in which I find myself are *not* directly on the local subnet. And thus aren't accessible as soon as I join the VPN.

If that situation is going to be allowed to persist in any form (especially if it's the default and we have to take special action to give the VPN nameservers top priority), then we should fix that by explicitly adding routes to them, as I've been having to do manually:

 $ ip route
default dev vpn0 proto static scope link metric 50
default via 10.246.8.1 dev wlp2s0 proto static metric 600
8.8.4.4 via 10.246.8.1 dev wlp2s0
8.8.8.8 via 10.246.8.1 dev wlp2s0

[Olivier Tilloy (osomon)]

A fix was merged to the upstream master branch: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=d9782589248e61c0cb5aec90e3eb62612891116b

[Olivier Tilloy (osomon)]

(In reply to Beniamino Galvani from comment #45)
> (In reply to Michael Biebl from comment #42)
> > Will those changes be pulled into the nm-1-10 branch as well?
>
> Let's wait until the branch is merged to master and see if there are
> any issues. If there are no complaints, we can think about backporting
> it to a stable branch.

Now that this has been in master and 1.12 for a few months, is this something you would consider backporting to 1.10 ?

[Thomas Haller (thaller-1)]

(In reply to Olivier Tilloy from comment #51)
> (In reply to Beniamino Galvani from comment #45)
> > (In reply to Michael Biebl from comment #42)
> > > Will those changes be pulled into the nm-1-10 branch as well?
> >
> > Let's wait until the branch is merged to master and see if there are
> > any issues. If there are no complaints, we can think about backporting
> > it to a stable branch.
>
> Now that this has been in master and 1.12 for a few months, is this
> something you would consider backporting to 1.10 ?

Can you describe who would pick up the upstream patches if they get backported?
Would 1.10 branch be far enough?

Would you then rebase to a minor 1.10.z release, or would you just cherry-pick the patches?

[dwmw2 (dwmw2)]

I'd like to see them in a Fedora release, which is preferably via a 1.10.z release rather than just cherry-picking the patches.

Also Ubuntu but they basically never fix anything so I suppose that's unlikely. I'd still like it in a 1.10.z release so they don't have any *excuse* though.

[Sebastien Bacher (seb128)]

@David, I don't think that comment is either constructive or true. We have somewhat limited resources and struggle a bit sometime to keep up with network-manager bugfixes/update but we would backport a fix for this bug to our stable serie if there is one landing in the 1.10 vcs.

[Bgalvani (bgalvani)]

The backport is quite complicated due to several changes that made the
DNS code in nm-1-12 diverge from nm-1-10. The th/policy-and-mdns
branch [1] alone changed:

 src/dns/nm-dns-dnsmasq.c | 330 +++++++++++++--------------------
 src/dns/nm-dns-manager.c | 581 +++++++++++++++++++++++++++++++++-------------------------
 src/dns/nm-dns-systemd-resolved.c | 219 ++++++++++++++--------

so, unless we decide to also backport that branch, the backport of the
fix requires reimplementing several non-trivial patches on top of
1.10, with the high risk of breaking things.

If we also backport the th/policy-and-mdns branch, we would need to
backport a total of 30+ commits, including mDNS support and related
libnm API.

Any opinions? This latter approach seems doable, but it is a big
change for a minor release.

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/log/?qt=range&q=5eea9be983bd95f5a482e483a438a100b80c716f

[Thomas Haller (thaller-1)]

(In reply to Beniamino Galvani from comment #55)
>
> Any opinions? This latter approach seems doable, but it is a big
> change for a minor release.

Backporting th/policy-and-mdns too, seems favourable to me. As long, as the cherry-picks are trivial, the result will all be code which is on master and tested for a while already. I'd rather cherry-pick more patches just to make them apply, instead of reimplementing them. YMMV.

[Gijs Molenaar (gijzelaar)]

Is it possible to upload a fixed package to bionic backports?

[fessmage (fessmage)]

Same question, will it be backported to Ubuntu 18.04 ?

[Olivier Tilloy (osomon)]

See the discussion in the upstream bug report. The fix is in the master branch and needs to be backported to the 1.10 series so that we can pick it up in bionic.

[Olivier Tilloy (osomon)]

This is fixed in the 1.12 series of network-manager (1.12.0 release), so cosmic and dingo are not affected.

[Olivier Tilloy (osomon)]

I confirm that the Ubuntu desktop team would be interested in picking up the update if this fix and related branches were to be backported to the 1.10 branch.

Is this something we can reasonably expect?

[Thomas Haller (thaller-1)]

it's backported to nm-1-10 branch: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=1e486a721de1fec76c81bfc461671a7fbdae531b

[Olivier Tilloy (osomon)]

Excellent, thank you Thomas!

[Olivier Tilloy (osomon)]

The fix was backported to the upstream 1.10 series.

[Bgalvani (bgalvani)]

The fix is included also in the 1.10.14 release.

[Olivier Tilloy (osomon)]

That's handy. The backport to Ubuntu 18.04 is being tracked by https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1754671.

[Sebastien Bacher (seb128)]

I've updated the description for the SRU but if someone had a better description of a testcase that would be welcome

[Brian Murray (brian-murray)]

Hello dwmw2, or anyone else affected,

Accepted network-manager into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.10.14-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Olivier Tilloy (osomon)]

Please test and share your feedback on this new version here, but refrain from changing the verification-needed-bionic tag for now. This new version includes many changes and we want to give it an extended testing period to ensure no regressions sneak in, before it is published to bionic-updates. Thanks!

[Steve Langasek (vorlon)]

How does this proposed change relate to LP: #1726124? Are users who are currently relying on correct split DNS handling by network-manager+systemd-resolved in bionic going to see this regress and have all DNS requests now sent over the VPN when they aren't supposed to?

[fessmage (fessmage)]

I installed package of network-manager 1.10.14-0ubuntu1 from bionic-proposed, and can confirm that version fixed dns leak: now when vpn connection established it gets `DNS Domain: ~.` in systemd-resolve automatically, so no more needed to manually apply command `systemd-resolve -i tun0 --set-domain=~.`. This positively fix dns leakage, verified by dnsleaktest.com

[Olivier Tilloy (osomon)]

@Steve (sorry for the late reply): not sure how that relates to bug #1726124, but in my limited understanding of the changes, they shouldn't regress the split-DNS use case.

Some relevant pointers to better understand the fixes and their context:

 - https://bugzilla.gnome.org/show_bug.cgi?id=746422 (particularly comments 8 and 26)

 - https://wiki.gnome.org/Projects/NetworkManager/DNS

 - https://gitlab.freedesktop.org/NetworkManager/NetworkManager/blob/nm-1-10/NEWS

[dwmw2 (dwmw2)]

network-manager-1.10.14-0ubuntu1 does seem to fix the DNS problem here; thanks.

[dwmw2 (dwmw2)]

Hm, that didn't last long. Now it isn't looking up *anything* in the VPN domains. It's all going to the local VPN server. I don't know what changed.

[dwmw2 (dwmw2)]

Not sure what happened there. It was looking up *some* names in the $COMPANY.com domain on the VPN, but others not, consistently. I couldn't see a pattern.

I have manually set ipv4.dns-search="~." and ipv4.dns-priority=-1 and now it does seem to be behaving. However, this shouldn't be necessary. This VPN has non-split routing and shouldn't it have non-split DNS too, by default? I shouldn't have to change the configuration, just to get back to the secure behaviour which used to work.

[dwmw2 (dwmw2)]

I've finally managed to test the Ubuntu 18.04 backport; apologies for the delay.

At first it seemed to work. I tried an internal domain which isn't the main $COMPANY.com and isn't in the default search domains for the VPN, and it worked after the upgrade.

Some hours later, I was unable to get a Kerberos ticket because various other DNS lookups, even within the main $COMPANY.com domain, were being done on the local network and not the VPN.

I manually set ipv4.dns-priority=-1 and ipv4.dns-search=~. but this is a full-tunnel VPN; surely I shouldn't have needed to do that part manually?

(Sebastian, apologies for the somewhat grumpy comment earlier. I am very happy to be proved wrong.)

[dwmw2 (dwmw2)]

Gr, and apologies too for spelling your name wrong, Sebastien.

[fessmage (fessmage)]

@dwmw2, as far as i understand, you should configuring DNS through systemd-resolve only. Try remove your edits from `/etc/NetworkManager/system-connections`, or even delete your connections from NetworkManager interface, and create new. After that, establish vpn connection and see at `systemd-resolve --status`, you should get something like this:

```
Link 3 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: xx.xx.xx.xx
                      xx.xx.xx.xx
          DNS Domain: ~.

Link 2 (enp3s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.1
          DNS Domain: local.domain
```

Where local.domain was received from DHCP server in local network. In that case you will send DNS requests in local.domain to local DNS server, and all other DNS requests - over VPN. That is expected behaviour. If you get this, but you have needs for redirecting DNS requests for some domain through other route (let's say, requests to local2.domain2, without VPN), you can do this with next command: `systemd-resolve -i enp3s0 --set-domain=local2.domain2`

[Bgalvani (bgalvani)]

Hi,

can you please attach the log file generated in this way:

 # nmcli general logging level trace
 # journalctl -u NetworkManager -f > dns-log.txt &
 # killall -HUP NetworkManager
 # kill %1

when the DNS is wrongly configured? Thanks.

[dwmw2 (dwmw2)]

Before I do that, do we agree that I shouldn't have needed to manually set dns-priority and dns-search as described in comment 62, for a VPN connection which is full-tunnel?

[Bgalvani (bgalvani)]

(In reply to David Woodhouse from comment #65)
> Before I do that, do we agree that I shouldn't have needed to manually set
> dns-priority and dns-search as described in comment 62, for a VPN connection
> which is full-tunnel?

Yes, the full-tunnel VPN should have higher priority.

We have an automated CI test to check this scenario:

 https://github.com/NetworkManager/NetworkManager-ci/blob/master/nmcli/features/dns.feature#L403

and it seems to be working well.

[dwmw2 (dwmw2)]

Ack, then I will remove those settings and test again. With dnsmasq, OK? This can never work with plain resolv.conf anyway.

Thanks.

[Bgalvani (bgalvani)]

(In reply to David Woodhouse from comment #67)
> Ack, then I will remove those settings and test again. With dnsmasq, OK?

Yes, thanks.

[Taylor Raack (track16)]

I can also confirm that the network-manager package version 1.10.14-0ubuntu1 from bionic-proposed fixes the issue.

[dwmw2 (dwmw2)]

Is there a 16.04 package? This was a regression there caused by an earlier update.

I have users reporting the same bizarre behaviour I wasn't able to clearly describe before — essentially, DNS being sent out seemingly random interfaces (sometimes VPN, sometimes local). My advice to just install this package *and* manually set dns-priority=-1,dns-search=~. and get on with life even though you really shouldn't have to manually set the latter, doesn't work for the 16.04 users...

And yes, when other things stop being on fire I need to undo those settings and try to work out what's going wrong. We aren't using systemd-resolve here because historically it also hasn't worked right while dnsmasq did.

[Sebastien Bacher (seb128)]

@dwmw2, 'This was a regression there caused by an earlier update.' would give some details ont that? you should probably open another report specifically about that if there was a regression in a xenial update

[dwmw2 (dwmw2)]

@seb128 please see "In 16.04 the NetworkManager package used to carry this patch..." in the bug description above.

[Mathew Hodson (mhodson)]

Looking at the upstream bug, it looks like the fix relies on reworking large parts of the code and wouldn't be easy to SRU to Xenial.

[Steve Langasek (vorlon)]

Based on comment #12 I am not sure that this is considered "verification-done" by the relevant developers and there was no comment given when the tags were changed. Resetting.

I also think there should be an affirmative test as part of this SRU that the use case I described in comment #13 has not been regressed.

[Till Kamppeter (till-kamppeter)]

I have now done the test under [Test Case] in the initial description of this bug report.

I have a completely updated (including -proposed) Bionic machine (real iron, a Lenovo X1 Carbon 2nd gen from 2015) with network-manager 1.10.14-0ubuntu1

I have configured the Canonical VPN, both UK and US. I have turned on only the UK one. It is configured to be used only for the internal destinations on both IPv4 and IPv6.

The system in this configuration I have rebooted to be assure that all processes including the kernel are using the newest software.

Then I have followed the instructions of the test case.

When running "dig <a Canonical-internal host name>" I get immediately an answer with exit code 0 ("echo $?"), so the request was successful.

When I look into the "tcpdump" terminals, the host name gets polled through both interfaces, but naturally the answer only comes from the DNS of the VPN.

So to my understanding the bug is not fixed as the private host name gets also sent to the public DNS.

"systemd-resolve --status" lists the VPN DNS first, as link 4 and afterwards the public DNS as link3.

[Till Kamppeter (till-kamppeter)]

Good news, the network-manager SRU is not broken or wrong, but an additional SRU, on systemd, is needed to actually fix this bug.

I got a hint from Iain Lane (Laney, thank you very much) to the following fix in systemd upstream:

https://github.com/systemd/systemd/commit/a97a3b256

and backported it to Bionic's systemd package (debdiff attached). With the network-manager SRU from -proposed attached plus the patched systemd package installed the problem goes away. If I repeat the test of [Test Case] (after a reboot) the DNS requests to any of the VPN's domains go actually only to the VPN's DNS.

[Timo Aaltonen (tjaalton)]

Hello dwmw2, or anyone else affected,

Accepted network-manager into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.10.14-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Will be releasing network-manager without the systemd part for now as it poses no threat to the user.

[Launchpad Janitor (janitor)]

This bug was fixed in the package network-manager - 1.10.14-0ubuntu2

---------------
network-manager (1.10.14-0ubuntu2) bionic; urgency=medium

  [ Till Kamppeter ]
  * debian/tests/nm: Add gi.require_version() calls for NetworkManager
    and NMClient to avoid stderr output which fails the test.

  [ Iain Lane ]
  * debian/tests/control: The nm tests need dnsmasq-base and isc-dhcp-client
    too.

network-manager (1.10.14-0ubuntu1) bionic; urgency=medium

  * New stable version (LP: #1809132), including:
    - Support private keys encrypted with AES-{192,256}-CBC in libnm
      (LP: #942856)
    - Fix leak of DNS queries to local name servers when connecting to a
      full-tunnel VPN (CVE-2018-1000135) (LP: #1754671)
  * Dropped patch applied upstream:
    - debian/patches/CVE-2018-15688.patch
    - debian/patches/e91f1a7d2a6b8400b6b331d5b72287dcb5164a39.patch
  * Refreshed patches:
    - debian/patches/Don-t-make-NetworkManager-D-Bus-activatable.patch
    - debian/patches/Force-online-state-with-unmanaged-devices.patch
    - debian/patches/Read-system-connections-from-run.patch
    - debian/patches/Update-dnsmasq-parameters.patch
    - debian/patches/libnm-register-empty-NMClient-and-NetworkManager-when-loa.patch

 -- Till Kamppeter <email address hidden> Fri, 10 May 2019 13:34:00 +0200

[Adam Conrad (adconrad)]

The original bug report was about a regression in 16.04 with the dnsmasq integration. While I'm glad this got the ball rolling on the bionic networkd integration, let's not forget that we broke xenial? Added a xenial task for network-manager accordingly.

[dwmw2 (dwmw2)]

I am receiving reports that it isn't fixed in 18.04 either. Users are still seeing DNS lookups on the local network, until they manually edit the VPN config to include:

[ipv4]
dns-priority=-1
dns-search=~.;

I thought that wasn't going to be necessary?

[Till Kamppeter (till-kamppeter)]

dwmw2, did you apply the systemd fix from comment #27? For this bug to be fixed you need BOTRH the fixed packages of network-manager and systemd.

[dwmw2 (dwmw2)]

These systems are using dnsmasq not systemd-resolver. This was done for historical reasons; I'm not sure of the specific bug which caused that choice.

[Till Kamppeter (till-kamppeter)]

Unfortunately, the SRU for systemd did not yet get processed. Therefore I have now uploaded this version of systemd to my PPA so that you can already test/get your problem solved. Please tell here whether it actually fixes the bug.

Here is my PPA:

https://launchpad.net/~till-kamppeter/+archive/ubuntu/ppa

Please follow this link, follow the instructions in the section "Adding this PPA to your system", then update your system with the command

sudo apt dist-upgrade

This will update only systemd as I did not upload any other package for Bionic to my PPA.

Make also sure you have the update of network-manager (1.10.14-0ubuntu2) installed. Reboot and check whether everything works correctly now.

[dwmw2 (dwmw2)]

We aren't using systemd-resolver for various historical reasons; we are using dnsmasq which should be expected to work. It isn't, but we have manually added the dns-priority=-1;dns-search=~. settings which make it work, as an emergency deployment when the latest NM update broke things for everyone.

[Till Kamppeter (till-kamppeter)]

dwmw2, the systemd fix was mainly meant for people with standard configuration where this fix is actually needed and solve the problem.

You are writing that adding "dns-priority=-1;dns-search=~." solves the problem for you. Where/to which file did you add this? Do you need this already with the original network-manager version of Bionic (1.10.6) or do you only need ot after the update (1.10.14)?

[dwmw2 (dwmw2)]

This is Bionic.

After last week's update to 1.10.14-0ubuntu2 all my VPN users (who are using dnsmasq) reported that DNS supported working for them while they were on the VPN. Some internal names were looked up correctly, others weren't.

I resolved it for them as follows:

$ sudo nmcli con modify "$COMPANY VPN" ipv4.dns-priority -1 ipv4.dns-search ~.

This matches the observations I made in comment #18 on 2019-02-04.

I believe that with 1.10.6 all $company.com DNS did get sent to the VPN and it was lookups outside the company search domains which were leaked. So it was mostly functional, but insecure. Since 1.10.14 it got worse and many (but not all) of the $company.com lookups are being leaked too. Which is a functional problem.

(For Xenial, my advice to users has been the same since March 2018 when this ticket was first filed: tell apt to hold network-manager_1.2.2-0ubuntu0.16.04.4_amd64.deb and don't let it get updated until/unless the regression is fixed.)

[Steve Langasek (vorlon)]

> These systems are using dnsmasq not systemd-resolver.
> This was done for historical reasons; I'm not sure of
> the specific bug which caused that choice.

NetworkManager in Ubuntu 16.04 and earlier defaulted to integrating with dnsmasq. But on 18.04 and later, this integration has been deliberately replaced with integration with systemd-resolved. If you are overriding this default integration to force the use of dnsmasq instead of systemd-resolved, that is likely not a supportable configuration.

In contrast, any bug in the systemd-resolved integration in 18.04 that would force you to work around it by switching to dnsmasq is almost certainly an SRUable bug. If you can find the information about why you switched to dnsmasq, please report this as a bug against systemd (with 'ubuntu-bug systemd') and provide a link to the bug here.

[Steve Langasek (vorlon)]

Due to the SRU regressions reported in LP: #1829838 and LP: #1829566, I have reverted this SRU for the moment, restoring network-manager 1.10.6-2ubuntu1.1 to bionic-updates. I am marking this bug verification-failed pending resolution of the reported regressions.

[dwmw2 (dwmw2)]

Without the ipv4.dns-priority and ipv4.dns-search settings being manually set, I see the wired device still being used for its own search domain. This is what causes the problem.

It seems to work for me if I set *only* ipv4.dns-priority=-1 though.

My biggest practical problem here was that we had set ipv4.dns-priority and ipv4.dns-search options in an emergency deployment to our users after Ubuntu shipped the 1.10.14 update to 18.04 to fix this... and then when they *pulled* the update, new installations got the new config but older NM, and that didn't work correctly either.

I'm going to experiment with *just* setting ipv4.dns-priority=-1 and not ipv4.dns-search. Do we expect that to work for everyone, whether they have the updated package or not?

We should be setting ipv4.dns-priority=1 by default for full-tunnel VPNs, but at least if I can find a way to work around it that works for everyone, that'll be an improvement.

[dwmw2 (dwmw2)]

No, ipv4.dns-priority=-1 breaks older NM because there, there's no default routing domain ~. and not even any way to set it. So lookups outside the VPN domains are just refused instead of being leaked to the local nameservers as they were before.

As it happens, we configure our DHCP client to always override the discovered search domains to our corporate AD domain dom.company.com, so I can work around NM's failure to automatically set ipv4.dns-priority=-1 a different way — by manually configuring the VPN with ipv4.dns-domains=dom.company.com;company.com;. That works for both old and new NetworkManager.

[Bgalvani (bgalvani)]

> My biggest practical problem here was that we had set
> ipv4.dns-priority and ipv4.dns-search options in an emergency
> deployment to our users after Ubuntu shipped the 1.10.14 update to
> 18.04 to fix this... and then when they *pulled* the update, new
> installations got the new config but older NM, and that didn't work
> correctly either.

> I'm going to experiment with *just* setting ipv4.dns-priority=-1 and
> not ipv4.dns-search. Do we expect that to work for everyone, whether
> they have the updated package or not?

Yes, this should work the same on 1.10.14 and 1.12+ releases.

> We should be setting ipv4.dns-priority=1 by default for full-tunnel
> VPNs, but at least if I can find a way to work around it that works
> for everyone, that'll be an improvement.

Do you mean -1? Why? This will cause an opposite problem: local
queries leaked to the VPN, i.e. I look up nas.myhome and the query
goes through the VPN.

> No, ipv4.dns-priority=-1 breaks older NM because there, there's no default routing domain ~. and not even any way to set it

Which version is 'older'?

[dwmw2 (dwmw2)]

In my case, 'older' is 1.10.6. At this point I'm just trying to get things working for my users on Ubuntu 18.04. So I'm comparing 1.10.6 (the current Ubuntu package) with 1.10.14 (the update, that was briefly shipped in 18.04 updates and then withdrawn).

When the 1.10.14 update went out, it broke our users because lookups for the dom.company.com AD domain were now always going to the local network (because of our dhclient config override). We immediately rolled out a manual setting of ipv4.dns-priority=-1 (yes, not =1) to fix it.

Then the Ubuntu 1.10.14 update was withdrawn, and new installations got 1.10.6 which *doesn't* work with ipv4.dns-priority=-1, because there's no ~. routing domain and thus lookups outside the VPN domains don't work at all.

(We don't care about local lookups not working. In fact we'd prefer local lookups *not* to work, which might be part of the reason we've overridden the search domain obtained by DHCP. If we could stop *routing* to the local network from working, we would. It's an open feature request, I believe.)

But *because* I know the search domain on the local physical network will always be just 'dom.company.com', adding that explicitly in the VPN config is enough and I don't need ipv4.dns-priority=-1. Now I have a config that works with both 1.10.6 and 1.10.14. So I can wait for the regressions that caused the 1.10.14 update to be pulled, to get fixed and then maybe later I can set ipv4.dns-priority=-1 again.

Anyway, we now know that the reason my initial testing of the Ubuntu 1.10.14 backport was unsuccessful, was because of the search domain on the *local* network. I've been able to work around that, and things are now working OK.

[Dariusz Gadomski (dgadomski)]

I have backported what was listed as nm-1-10 fix for the bug in the upstream bugzilla [1].
I have also applied fixes for bug #1825946 and bug #1790098 to it.

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=1e486a721de1fec76c81bfc461671a7fbdae531b

After testing this build for some time (available at ppa:dgadomski/network-manager) I haven't found any problems.

@Till I'd appreciate you having a look at it. Thanks!

[Till Kamppeter (till-kamppeter)]

Great work, thank you very much!

It will need some testing of which I can only test the reproducer in the initial description of this bug report, not any regressions which the first attempt of upstream-update-based SRU, as I could not reproduce these by myself.

So I would say to take this as a new proposed SRU and also ask the reporters of the regressions whether this version does not cause them.

[Mathew Hodson (mhodson)]

This fix was first included in upstream 1.12.0, so this was actually fixed in Cosmic with network-manager 1.12.2-0ubuntu3

[Till Kamppeter (till-kamppeter)]

Sorry for the late reply, I was on a conference last week.

I installed the PPA now and tested with the reproducer of the initial posting. This works for me. Also the machine in general seems to work OK with this version of network-manager.

Thank you very much Dariusz for packaging this version.

So now the 1.10.14 should be removed from -proposed (to avoid need of an epoch), and the version from the PPA of Dariusz should get uploaded into -proposed, and then the reporters of the regressions in the 1.10.14 SRU informed (by comments in their bug reports) for the new SRU being verified.

Could someone from the release team initiate the process by removing 1.10.14 from -proposed? Thanks.

[Timo Aaltonen (tjaalton)]

Hello dwmw2, or anyone else affected,

Accepted network-manager into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.10.6-2ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted network-manager (1.10.6-2ubuntu1.2) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

network-manager/1.10.6-2ubuntu1.2 (amd64, arm64, i386, ppc64el)
systemd/237-3ubuntu10.31 (i386, ppc64el)
netplan.io/0.98-0ubuntu1~18.04.1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#network-manager

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Eric Desrochers (slashd)]

The netplan.io (arm64) autopkgtest failure (due to timeout) has been retried today, it passed:
http://autopkgtest.ubuntu.com/packages/n/netplan.io/bionic/arm64

No more failure reported in pending sru page.

- Eric

[Till Kamppeter (till-kamppeter)]

Now network-manager is hanging on (all autopkg tests passed):

Not touching package due to block request by freeze (contact #ubuntu-release if update is needed)

Which freeze do we currently have on Bionic?

[Till Kamppeter (till-kamppeter)]

No worries about my previous comment, it is solved.

[Dariusz Gadomski (dgadomski)]

I have just run the test case from this bug description on the bionic-proposed version 1.10.6-2ubuntu1.2.
tcpdump does not show any leak of the VPN-specific queries. I have not observed other issues in my tests.

[Launchpad Janitor (janitor)]

This bug was fixed in the package network-manager - 1.10.6-2ubuntu1.2

---------------
network-manager (1.10.6-2ubuntu1.2) bionic; urgency=medium

  [ Till Kamppeter ]
  * debian/tests/nm: Add gi.require_version() calls for NetworkManager
    and NMClient to avoid stderr output which fails the test. (LP: #1825946)

  [ Dariusz Gadomski ]
  * d/p/fix-dns-leak-lp1754671.patch: backport of DNS leak fix. (LP: #1754671)
  * d/p/lp1790098.patch: retry activating devices when the parent becomes
    managed. (LP: #1790098)

 -- Dariusz Gadomski <email address hidden> Sat, 07 Sep 2019 16:10:59 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for network-manager has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Joe Hohertz (jhohertz)]

1.10.6-2ubuntu1.2 has cause a regression in functionality.

Anyone using a "split" VPN, where there is no default route, AND wish to have DNS services supplied by the server to be honoured, via use of the ipv4.dns-priority parameter, will have this broken. This is a bit of a sore point considering the hoops one has to jump through to make this work at all.

Reverting to previous version restores functionality.

[Andre Klapper (a9016009)]

bugzilla.gnome.org is being shut down in favor of a GitLab instance.
We are closing all old bug reports and feature requests in GNOME Bugzilla which have not seen updates for a long time.

If you still use NetworkManager and if you still see this bug / want this feature in a recent and supported version of NetworkManager, then please feel free to report it at https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/

Thank you for creating this report and we are sorry it could not be implemented (workforce and time is unfortunately limited).