Full-tunnel VPN DNS leakage regression

Bug #1754671 reported by dwmw2 on 2018-03-09
326
This bug affects 10 people
Affects Status Importance Assigned to Milestone
NetworkManager
Confirmed
Medium
network-manager (Ubuntu)
High
Unassigned
Xenial
Undecided
Unassigned
Bionic
High
Till Kamppeter
Cosmic
Undecided
Unassigned
systemd (Ubuntu)
High
Unassigned
Xenial
Undecided
Unassigned
Bionic
High
Dan Streetman
Cosmic
High
Dan Streetman

Bug Description

[Impact]
When using a VPN the DNS requests might still be sent to a DNS server outside the VPN when they should not

[Test case]
1) Set up a VPN with split tunneling:
  a) Configure VPN normally (set up remote host, any ports and options needed for the VPN to work)
  b) Under the IPv4 tab: enable "Use this connection only for the resources on its network".
  c) Under the IPv6 tab: enable "Use this connection only for the resources on its network".

2) Connect to the VPN.

3) Run 'systemd-resolve --status'; note the DNS servers configured:
  a) For the VPN; under a separate link (probably tun0), note down the IP of the DNS server(s). Also note the name of the interface (link).
  b) For the "main" connection; under the link for your ethernet or wireless devices (wl*, en*, whatever it may be), note down the IP of the DNS server(s). Also note the name of the interface (link).

4) In a separate terminal, run 'sudo tcpdump -ni <the main interface> port 53'; let it run.

5) In a separate terminal, run 'sudo tcpdump -ni <the VPN interface> port 53'; let it run.

6) In yet another terminal, issue name resolution requests using dig:
  a) For a name known to be reachable via the public network:
     'dig www.yahoo.com'
  b) For a name known to be reachable only via the VPN:
     'dig <some DNS behind the VPN>'

7) Check the output of each terminal running tcpdump. When requesting the public name, traffic can go through either. When requesting the "private" name (behind the VPN), traffic should only be going through the interface for the VPN. Additionally, ensure the IP receiving the requests for the VPN name is indeed the IP address noted above for the VPN's DNS server.

If you see no traffic showing in tcpdump output when requesting a name, it may be because it is cached by systemd-resolved. Use a different name you have not tried before.

[Regression potential]
The code change the handling of DNS servers when using a VPN, we should check that name resolution still work whne using a VPN in different configurations

-----------------

In 16.04 the NetworkManager package used to carry this patch:
http://bazaar.launchpad.net/~network-manager/network-manager/ubuntu/view/head:/debian/patches/Filter-DNS-servers-to-add-to-dnsmasq-based-on-availa.patch

It fixed the DNS setup so that when I'm on the VPN, I am not sending unencrypted DNS queries to the (potentially hostile) local nameservers.

This patch disappeared in an update. I think it was present in 1.2.2-0ubuntu0.16.04.4 but was dropped some time later.

This security bug exists upstream too: https://bugzilla.gnome.org/show_bug.cgi?id=746422
It's not a *regression* there though, as they didn't fix it yet (unfortunately!)

CVE References

dwmw2 (dwmw2) on 2018-03-09
information type: Private Security → Public Security

Confirming this is broken. Dropping the patch 0001-dns-use-DBus-to-make-dnsmasq-nameserver-changes.patch in network-manager (1.2.4-0ubuntu0.16.04.1) was done, but it looks like not all the code in that patch was actually upstream.

Changed in network-manager (Ubuntu):
status: New → Confirmed
importance: Undecided → High
tags: added: regression-update
dwmw2 (dwmw2) wrote :

This is CVE-2018-1000135. For some reason the 'Link to CVE' option above doesn't seem to work.

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1000135

95 comments hidden view all 137 comments

The idea is that VPNs would automatically have the "~." default lookup domain added, according to the never-default flag.

(In reply to David Woodhouse from comment #31)
> (In reply to Beniamino Galvani from comment #29)
> > > • We might perhaps take this opportunity to kill the weirdness that we have
> > > redundant DNS information in *both* IPv4 and IPv6 configs. That was always
> > > wrong, and a clean break to make it sane might be better than letting it
> > > persist for ever.
> >
> > Changing this is a huge pain for users, and developers too (but this
> > probably doesn't matter much). I can't think of a way to achieve a smooth
> > transition from the separate DNS properties into a new common setting.
>
> It's a short-term pain, which will eventually go away.

Maybe, but existing properties are part of the API and so they will never be
dropped because we don't break API between releases. We'll have to maintain
them, together with the new properties and the code to sync them forever.

> Can't we start by shadowing the ipv?.dns-* options into a generic dns.*
> option set so that they're identical? We can give people a long time to
> start using the new location, before eventually taking away the old ones.

If you mean that ipv4.dns-*, ipv6.dns-* and dns.* should be all identical,
that is a change in behavior and would badly break users.

If by shadowing you mean keeping them is sync (with the dns.* as the union
of ipv4.dns-* and ipv6.dns-*), that is possible but would create some other
problems in my opinion.

> We definitely shouldn't be making the problem *worse* by doing things like
> the ~ prefix hack or adding any *more* fields to ipv?.dns-*. Can't we at
> least add that as dns.lookup-domain even if it's all by itself in the "DNS"
> settings for now?

Ok, we aren't going to add any new properties to ipvx.dns-*. Yes, I think we
can add a dns.lookup-domain property.

Right, the idea was the dns.* properties would be the union of the ipv4.dns-* and ipv6.dns-* (which right now are treated identically and just appended to each other, I believe?).

Note: it's "lookup domains" we should be using for the proxy setup we poke into PacRunner, not "search domains". Can we fix that too please?

Will Cooke (willcooke) on 2018-03-26
tags: added: incoming rs-bb-
tags: added: rls-bb-incoming
removed: incoming rs-bb-

I pushed branch bg/dns-bgo746422. I think it should solve the leaks in
case of full-tunnel VPNS when using dnsmasq or systemd-resolved. It
basically:

 - allows users to specify that connections get the default lookup
   domain through the special entry "~."

 - automatically adds "~." to connections with the default route

 - applies rules from comment 32 to decide which domains are used
   based on DNS priority.

Other things that I didn't do, but can be done later (if necessary)
are:

 - as noticed by Thomas, you can't override the NM decision of
   automatically adding "~." to connections with default route.
   Actually, you can add "~." to another connection with a lower
   priority value, and it will override the "~." added by NM on the
   first connection. Perhaps this is enough. Otherwise we could
   introduce another special domain.

 - I haven't added a new dns setting with duplicates of the existing
   ipvx properties. I think it's really a different issue and should
   be solved separately.

 - Also I didn't add the dns.lookup-domain property, as the "~."
   domain is sufficient and it is just another case of the
   routing-only domains we already support.

What do you think?

> dns: dnsmasq: fix adding multiple domains

can you add a "Fixes" comment?

> core: allow '~.' domain in ip4 and ip6 configs

+ if (!nm_streq (search, "~.")) {
          len = strlen (search);
          if (search[len - 1] == '.')
               search[len - 1] = 0;
+ }

this seems wrong, for example if "search" is "~..".
Either, do a full normalization and drop such entires (or clean up the duplicate dots). Or: normalize "~." to "~" too. It's odd that the special domain is treated entirely different regardingt the trailing dot.

> dns: use dns-priority to provide a preprocessed domain list to plugins

          _LOGD ("update-dns: updating plugin %s", plugin_name);
+ rebuild_domain_lists (self);
          if (!nm_dns_plugin_update (plugin,

do we need to rebuild the list every time? We know when an update changes something. Can we cache the result, generate it when needed, and clear it when something changes?

+ struct {
+ const char **search;
+ char **reverse;
+ } domains;
 } NMDnsIPConfigData;

I think these are leaked.

> dns: dnsmasq: honor dns-priority

+ int addr_family, i, j, num;

let's use guint for index variables of arrays? Also matches
  num = nm_ip_config_get_num_nameservers (ip_config);

> dns: sd-resolved: honor dns-priority

- NMIPConfig *ip_config = elem->data;
+ NMDnsIPConfigData *data = elem->data;
+ NMIPConfig *ip_config = data->ip_config;

this seems wrong.

Overall, lgtm

Changed in network-manager (Ubuntu):
assignee: nobody → Olivier Tilloy (osomon)
tags: removed: rls-bb-incoming
Changed in network-manager:
importance: Unknown → Medium
status: Unknown → Confirmed

(In reply to Thomas Haller from comment #37)
> > dns: dnsmasq: fix adding multiple domains
>
> can you add a "Fixes" comment?

Done.

> > core: allow '~.' domain in ip4 and ip6 configs
>
> + if (!nm_streq (search, "~.")) {
> len = strlen (search);
> if (search[len - 1] == '.')
> search[len - 1] = 0;
> + }
>
> this seems wrong, for example if "search" is "~..".
> Either, do a full normalization and drop such entires (or clean up the
> duplicate dots). Or: normalize "~." to "~" too. It's odd that the special
> domain is treated entirely different regardingt the trailing dot.

What do you mean by "full normalization"?

We can convert 'domain.' into 'domain' because it's equivalent, but
'domain..' or 'my..domain' are invalid and they should be dropped. I
think this would be the best approach.

If we normalize '~.' to '~', once we strip the ~ prefix the domain
would become empty, which is not desirable in my opinion.

> > dns: use dns-priority to provide a preprocessed domain list to plugins
>
> _LOGD ("update-dns: updating plugin %s", plugin_name);
> + rebuild_domain_lists (self);
> if (!nm_dns_plugin_update (plugin,
>
> do we need to rebuild the list every time? We know when an update changes
> something. Can we cache the result, generate it when needed, and clear it
> when something changes?

If nothing changes, the DNS configuration hash will be the same and
update_dns() won't be called at all by nm_dns_manager_end_updates().

Ok, nm_dns_manager_set_hostname() calls update_dns() without checking
the hash, but do we need another caching mechanism different from the
existing one just for this case?

> + struct {
> + const char **search;
> + char **reverse;
> + } domains;
> } NMDnsIPConfigData;
>
> I think these are leaked.

Ops, fixed.

> > dns: dnsmasq: honor dns-priority
>
> + int addr_family, i, j, num;
>
> let's use guint for index variables of arrays? Also matches
> num = nm_ip_config_get_num_nameservers (ip_config);

Ok.

> > dns: sd-resolved: honor dns-priority
>
> - NMIPConfig *ip_config = elem->data;
> + NMDnsIPConfigData *data = elem->data;
> + NMIPConfig *ip_config = data->ip_config;
>
> this seems wrong.

Why? Looks fine to me.

(In reply to Beniamino Galvani from comment #38)
> (In reply to Thomas Haller from comment #37)

> What do you mean by "full normalization"?
>
> We can convert 'domain.' into 'domain' because it's equivalent, but
> 'domain..' or 'my..domain' are invalid and they should be dropped. I
> think this would be the best approach.

I mean, to handle "my..domain". Either normalize the double . away, or verify it and drop it as invalid. If you don't do either, then "~.." will end up being treated like "~." which is wrong.

> If we normalize '~.' to '~', once we strip the ~ prefix the domain
> would become empty, which is not desirable in my opinion.

Yes, after dropping the default domain becomes "". It's not that you currently use the domain "." as-is. You do:

  nm_streq (domain, ".") ? NULL : domain)

that could also be:

  nm_streq (domain, "") ? NULL : domain)

or

#define DEFAULT_DOMAIN ""
  nm_streq (domain, DEFAULT_DOMAIN) ? NULL : domain)

(maybe a define is in order either way).

> > > dns: use dns-priority to provide a preprocessed domain list to plugins
> >
> > _LOGD ("update-dns: updating plugin %s", plugin_name);
> > + rebuild_domain_lists (self);
> > if (!nm_dns_plugin_update (plugin,
> >
> > do we need to rebuild the list every time? We know when an update changes
> > something. Can we cache the result, generate it when needed, and clear it
> > when something changes?
>
> If nothing changes, the DNS configuration hash will be the same and
> update_dns() won't be called at all by nm_dns_manager_end_updates().
>
> Ok, nm_dns_manager_set_hostname() calls update_dns() without checking
> the hash, but do we need another caching mechanism different from the
> existing one just for this case?

Maybe the hashing mechanism is anyway ugly, and should be dropped (in a future commit). Instead of implementing a SHA-hashing of ~some~ parameters, implement a cmp() function. It's more efficient, and easier to get right.

> > > dns: sd-resolved: honor dns-priority
> >
> > - NMIPConfig *ip_config = elem->data;
> > + NMDnsIPConfigData *data = elem->data;
> > + NMIPConfig *ip_config = data->ip_config;
> >
> > this seems wrong.
>
> Why? Looks fine to me.

you are right.

(In reply to Thomas Haller from comment #39)
> (In reply to Beniamino Galvani from comment #38)
> > (In reply to Thomas Haller from comment #37)
>
> > What do you mean by "full normalization"?
> >
> > We can convert 'domain.' into 'domain' because it's equivalent, but
> > 'domain..' or 'my..domain' are invalid and they should be dropped. I
> > think this would be the best approach.
>
> I mean, to handle "my..domain". Either normalize the double . away, or
> verify it and drop it as invalid. If you don't do either, then "~.." will
> end up being treated like "~." which is wrong.
>
> > If we normalize '~.' to '~', once we strip the ~ prefix the domain
> > would become empty, which is not desirable in my opinion.
>
> Yes, after dropping the default domain becomes "". It's not that you
> currently use the domain "." as-is. You do:
>
> nm_streq (domain, ".") ? NULL : domain)
>
> that could also be:
>
> nm_streq (domain, "") ? NULL : domain)
>
> or
>
> #define DEFAULT_DOMAIN ""
> nm_streq (domain, DEFAULT_DOMAIN) ? NULL : domain)
>
> (maybe a define is in order either way).

Yeah, considering "" internally as the wildcard domain simplifies things a bit. I've updated the branch.

> Maybe the hashing mechanism is anyway ugly, and should be dropped (in a
> future commit). Instead of implementing a SHA-hashing of ~some~ parameters,
> implement a cmp() function. It's more efficient, and easier to get right.

A agree, this could be an improvement (for the future).

101 comments hidden view all 137 comments
Olivier Tilloy (osomon) wrote :

There's active work going on upstream (see https://bugzilla.gnome.org/show_bug.cgi?id=746422 and https://cgit.freedesktop.org/NetworkManager/NetworkManager/log/?h=bg/dns-bgo746422) to fix the issue.

https://bugzilla.gnome.org/show_bug.cgi?id=746422#c36 explains how.

Once in master, it would probably be doable to backport those changes (including https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=b2f306ac3d84283fdebb225079f354afb8c2a752) to the 1.10 branch, which is what's in bionic (1.10.6-2ubuntu1). Backporting to xenial (currently 1.2.6-0ubuntu0.16.04.2) would likely be an entirely different story.

102 comments hidden view all 137 comments

> core: allow '~.' domain in ip4 and ip6 configs

commit bebafff7844228d018d17f0d11c5d6035dbf6a91,
lgtm, but the commit message no longer seams to match, does it?

> dns: use dns-priority to provide a preprocessed domain list to plugins

+ c_list_for_each_entry (ip_data, _ip_config_lst_head (self), ip_config_lst) {

the list head is accessed on every iteration. Could we pre-cache the value like:

    CList *ip_config_lst_head;

    ip_config_lst_head = _ip_config_lst_head (self);

    c_list_for_each_entry (ip_data, ip_config_lst_head, ip_config_lst) {

    (and below again)

+ int i, n, n_domains = 0;

these variables are used to iterate over a number of guint values. Like

  n = nm_ip_config_get_num_searches (ip_config);

where get_num_searches() returns a guint. Could we consistently use the correct type? (yes, I know that C might optimize signed for loops better, because it assumes that signed cannot overflow. But IMO consistent use of types is more important than what the compiler might do).

+ priority = nm_ip_config_get_dns_priority (ip_config);
+ nm_assert (priority != 0);

this invariant is not enforced by NMIPxConfig, so, you basically assert that all callers that create an NMIPxConfig properly initialize priority to a non-zero value. It asserts something that is two layers of code away. That is non-obvious. Not sure what to do about this. Ok, fine as is :)

add the wilcard domain to all non-VPN
        ^^^^^^^

+ parent_priority = GPOINTER_TO_INT (g_hash_table_lookup (ht, "."));
+ if (parent_priority < 0 && parent_priority < priority) {
+ *out_parent = ".";
+ *out_parent_priority = parent_priority;

is "." still right? Should this be "" to mean the wildcard domain?

could you add a
   nm_assert (!g_hash_table_contains (ht, domain));
to domain_is_shadowed() -- because that one is already checked before calling domain_is_shadowed().

         g_free (ip_data->domains.search);

I always like avoiding unnecessary copies. But in this case, "search" array will point inside strings owned by NMIPxConfig. That seems quite fragile to me. Should we not clone them? In practice it's not necessary, but it feels fragile. What can we do about that?

»···/* Free the array and return NULL if the only element was the ending NULL */
»···strv = (char **) g_ptr_array_free (domains, (domains->len == 1));

»···return _nm_utils_strv_cleanup (strv, FALSE, FALSE, TRUE);

skip_repeated=TRUE likely won't help much, because duplicated domains are probably not sorted after each other. Maybe sort them first? Maybe by cherry-picking "shared: add nm_utils_strv_sort() helper" from https://github.com/NetworkManager/NetworkManager/pull/91

Will those changes be pulled into the nm-1-10 branch as well?

(In reply to Thomas Haller from comment #41)
> > core: allow '~.' domain in ip4 and ip6 configs
>
> commit bebafff7844228d018d17f0d11c5d6035dbf6a91,
> lgtm, but the commit message no longer seams to match, does it?

> the list head is accessed on every iteration. Could we pre-cache the value
> like:

> + int i, n, n_domains = 0;
>
> these variables are used to iterate over a number of guint values. Like
>
> n = nm_ip_config_get_num_searches (ip_config);
>
> where get_num_searches() returns a guint. Could we consistently use the
> correct type?

> add the wilcard domain to all non-VPN

> is "." still right? Should this be "" to mean the wildcard domain?

> could you add a
> nm_assert (!g_hash_table_contains (ht, domain));
> to domain_is_shadowed() -- because that one is already checked before
> calling domain_is_shadowed().

Fixed all the above.

> I always like avoiding unnecessary copies. But in this case, "search" array
> will point inside strings owned by NMIPxConfig. That seems quite fragile to
> me. Should we not clone them? In practice it's not necessary, but it feels
> fragile. What can we do about that?

Now the list is cleared just after nm_dns_plugin_update() returns to ensure
that elements in the list don't become stale. What do you think?

> »···/* Free the array and return NULL if the only element was the ending
> NULL */
> »···strv = (char **) g_ptr_array_free (domains, (domains->len == 1));
>
> »···return _nm_utils_strv_cleanup (strv, FALSE, FALSE, TRUE);
>
> skip_repeated=TRUE likely won't help much, because duplicated domains are
> probably not sorted after each other. Maybe sort them first? Maybe by
> cherry-picking "shared: add nm_utils_strv_sort() helper" from
> https://github.com/NetworkManager/NetworkManager/pull/91

I don't understand, duplicate domains don't have to be consecutive with
current _nm_utils_strv_cleanup() implementation.

I've added some NM-CI tests to ensure the behavior is the one we expect:

https://github.com/NetworkManager/NetworkManager-ci/pull/180

(In reply to Michael Biebl from comment #42)
> Will those changes be pulled into the nm-1-10 branch as well?

Let's wait until the branch is merged to master and see if there are
any issues. If there are no complaints, we can think about backporting
it to a stable branch.

> I don't understand, duplicate domains don't have to be consecutive with
> current _nm_utils_strv_cleanup() implementation.

How do you mean?

calling "_nm_utils_strv_cleanup (strv, FALSE, FALSE, TRUE);" with skip_repeated=TRUE will remove "consecutive duplicates". That doesn't seem useful, because most of the time (given how the array is constructed), duplicates are not consecutive.
so either: do not drop any duplicates and do skip_repeated=FALSE
or: drop all duplicates, by sorting the array first, followed by skip_repeated=TRUE.

bg/dns-domains-bgo746422 LGTM

That is, bg/dns-bgo746422 LGTM

FWIW the original situation appears to be fairly broken regardless of the security situation. I've lost count of the number of times in my recent travels that the DNS servers of the airport/hotel/etc. in which I find myself are *not* directly on the local subnet. And thus aren't accessible as soon as I join the VPN.

If that situation is going to be allowed to persist in any form (especially if it's the default and we have to take special action to give the VPN nameservers top priority), then we should fix that by explicitly adding routes to them, as I've been having to do manually:

 $ ip route
default dev vpn0 proto static scope link metric 50
default via 10.246.8.1 dev wlp2s0 proto static metric 600
8.8.4.4 via 10.246.8.1 dev wlp2s0
8.8.8.8 via 10.246.8.1 dev wlp2s0

110 comments hidden view all 137 comments
Changed in network-manager:
status: Confirmed → Fix Released
111 comments hidden view all 137 comments

(In reply to Beniamino Galvani from comment #45)
> (In reply to Michael Biebl from comment #42)
> > Will those changes be pulled into the nm-1-10 branch as well?
>
> Let's wait until the branch is merged to master and see if there are
> any issues. If there are no complaints, we can think about backporting
> it to a stable branch.

Now that this has been in master and 1.12 for a few months, is this something you would consider backporting to 1.10 ?

(In reply to Olivier Tilloy from comment #51)
> (In reply to Beniamino Galvani from comment #45)
> > (In reply to Michael Biebl from comment #42)
> > > Will those changes be pulled into the nm-1-10 branch as well?
> >
> > Let's wait until the branch is merged to master and see if there are
> > any issues. If there are no complaints, we can think about backporting
> > it to a stable branch.
>
> Now that this has been in master and 1.12 for a few months, is this
> something you would consider backporting to 1.10 ?

Can you describe who would pick up the upstream patches if they get backported?
Would 1.10 branch be far enough?

Would you then rebase to a minor 1.10.z release, or would you just cherry-pick the patches?

I'd like to see them in a Fedora release, which is preferably via a 1.10.z release rather than just cherry-picking the patches.

Also Ubuntu but they basically never fix anything so I suppose that's unlikely. I'd still like it in a 1.10.z release so they don't have any *excuse* though.

@David, I don't think that comment is either constructive or true. We have somewhat limited resources and struggle a bit sometime to keep up with network-manager bugfixes/update but we would backport a fix for this bug to our stable serie if there is one landing in the 1.10 vcs.

The backport is quite complicated due to several changes that made the
DNS code in nm-1-12 diverge from nm-1-10. The th/policy-and-mdns
branch [1] alone changed:

 src/dns/nm-dns-dnsmasq.c | 330 +++++++++++++--------------------
 src/dns/nm-dns-manager.c | 581 +++++++++++++++++++++++++++++++++-------------------------
 src/dns/nm-dns-systemd-resolved.c | 219 ++++++++++++++--------

so, unless we decide to also backport that branch, the backport of the
fix requires reimplementing several non-trivial patches on top of
1.10, with the high risk of breaking things.

If we also backport the th/policy-and-mdns branch, we would need to
backport a total of 30+ commits, including mDNS support and related
libnm API.

Any opinions? This latter approach seems doable, but it is a big
change for a minor release.

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/log/?qt=range&q=5eea9be983bd95f5a482e483a438a100b80c716f

(In reply to Beniamino Galvani from comment #55)
>
> Any opinions? This latter approach seems doable, but it is a big
> change for a minor release.

Backporting th/policy-and-mdns too, seems favourable to me. As long, as the cherry-picks are trivial, the result will all be code which is on master and tested for a while already. I'd rather cherry-pick more patches just to make them apply, instead of reimplementing them. YMMV.

115 comments hidden view all 137 comments
Gijs Molenaar (gijzelaar) wrote :

Is it possible to upload a fixed package to bionic backports?

fessmage (fessmage) wrote :

Same question, will it be backported to Ubuntu 18.04 ?

Olivier Tilloy (osomon) wrote :

See the discussion in the upstream bug report. The fix is in the master branch and needs to be backported to the 1.10 series so that we can pick it up in bionic.

Olivier Tilloy (osomon) wrote :

This is fixed in the 1.12 series of network-manager (1.12.0 release), so cosmic and dingo are not affected.

Changed in network-manager (Ubuntu):
status: Confirmed → Fix Released
assignee: Olivier Tilloy (osomon) → nobody
113 comments hidden view all 137 comments

I confirm that the Ubuntu desktop team would be interested in picking up the update if this fix and related branches were to be backported to the 1.10 branch.

Is this something we can reasonably expect?

Excellent, thank you Thomas!

114 comments hidden view all 137 comments
Olivier Tilloy (osomon) wrote :

The fix was backported to the upstream 1.10 series.

115 comments hidden view all 137 comments

The fix is included also in the 1.10.14 release.

That's handy. The backport to Ubuntu 18.04 is being tracked by https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1754671.

115 comments hidden view all 137 comments
Sebastien Bacher (seb128) wrote :

I've updated the description for the SRU but if someone had a better description of a testcase that would be welcome

description: updated

Hello dwmw2, or anyone else affected,

Accepted network-manager into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.10.14-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in network-manager (Ubuntu Bionic):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-bionic
Olivier Tilloy (osomon) wrote :

Please test and share your feedback on this new version here, but refrain from changing the verification-needed-bionic tag for now. This new version includes many changes and we want to give it an extended testing period to ensure no regressions sneak in, before it is published to bionic-updates. Thanks!

Steve Langasek (vorlon) wrote :

How does this proposed change relate to LP: #1726124? Are users who are currently relying on correct split DNS handling by network-manager+systemd-resolved in bionic going to see this regress and have all DNS requests now sent over the VPN when they aren't supposed to?

fessmage (fessmage) wrote :

I installed package of network-manager 1.10.14-0ubuntu1 from bionic-proposed, and can confirm that version fixed dns leak: now when vpn connection established it gets `DNS Domain: ~.` in systemd-resolve automatically, so no more needed to manually apply command `systemd-resolve -i tun0 --set-domain=~.`. This positively fix dns leakage, verified by dnsleaktest.com

Olivier Tilloy (osomon) wrote :

@Steve (sorry for the late reply): not sure how that relates to bug #1726124, but in my limited understanding of the changes, they shouldn't regress the split-DNS use case.

Some relevant pointers to better understand the fixes and their context:

 - https://bugzilla.gnome.org/show_bug.cgi?id=746422 (particularly comments 8 and 26)

 - https://wiki.gnome.org/Projects/NetworkManager/DNS

 - https://gitlab.freedesktop.org/NetworkManager/NetworkManager/blob/nm-1-10/NEWS

dwmw2 (dwmw2) wrote :

network-manager-1.10.14-0ubuntu1 does seem to fix the DNS problem here; thanks.

dwmw2 (dwmw2) wrote :

Hm, that didn't last long. Now it isn't looking up *anything* in the VPN domains. It's all going to the local VPN server. I don't know what changed.

dwmw2 (dwmw2) wrote :

Not sure what happened there. It was looking up *some* names in the $COMPANY.com domain on the VPN, but others not, consistently. I couldn't see a pattern.

I have manually set ipv4.dns-search="~." and ipv4.dns-priority=-1 and now it does seem to be behaving. However, this shouldn't be necessary. This VPN has non-split routing and shouldn't it have non-split DNS too, by default? I shouldn't have to change the configuration, just to get back to the secure behaviour which used to work.

108 comments hidden view all 137 comments

I've finally managed to test the Ubuntu 18.04 backport; apologies for the delay.

At first it seemed to work. I tried an internal domain which isn't the main $COMPANY.com and isn't in the default search domains for the VPN, and it worked after the upgrade.

Some hours later, I was unable to get a Kerberos ticket because various other DNS lookups, even within the main $COMPANY.com domain, were being done on the local network and not the VPN.

I manually set ipv4.dns-priority=-1 and ipv4.dns-search=~. but this is a full-tunnel VPN; surely I shouldn't have needed to do that part manually?

(Sebastian, apologies for the somewhat grumpy comment earlier. I am very happy to be proved wrong.)

Gr, and apologies too for spelling your name wrong, Sebastien.

108 comments hidden view all 137 comments
fessmage (fessmage) wrote :

@dwmw2, as far as i understand, you should configuring DNS through systemd-resolve only. Try remove your edits from `/etc/NetworkManager/system-connections`, or even delete your connections from NetworkManager interface, and create new. After that, establish vpn connection and see at `systemd-resolve --status`, you should get something like this:

```
Link 3 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: xx.xx.xx.xx
                      xx.xx.xx.xx
          DNS Domain: ~.

Link 2 (enp3s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.1
          DNS Domain: local.domain
```

Where local.domain was received from DHCP server in local network. In that case you will send DNS requests in local.domain to local DNS server, and all other DNS requests - over VPN. That is expected behaviour. If you get this, but you have needs for redirecting DNS requests for some domain through other route (let's say, requests to local2.domain2, without VPN), you can do this with next command: `systemd-resolve -i enp3s0 --set-domain=local2.domain2`

109 comments hidden view all 137 comments

Hi,

can you please attach the log file generated in this way:

 # nmcli general logging level trace
 # journalctl -u NetworkManager -f > dns-log.txt &
 # killall -HUP NetworkManager
 # kill %1

when the DNS is wrongly configured? Thanks.

Before I do that, do we agree that I shouldn't have needed to manually set dns-priority and dns-search as described in comment 62, for a VPN connection which is full-tunnel?

(In reply to David Woodhouse from comment #65)
> Before I do that, do we agree that I shouldn't have needed to manually set
> dns-priority and dns-search as described in comment 62, for a VPN connection
> which is full-tunnel?

Yes, the full-tunnel VPN should have higher priority.

We have an automated CI test to check this scenario:

 https://github.com/NetworkManager/NetworkManager-ci/blob/master/nmcli/features/dns.feature#L403

and it seems to be working well.

Ack, then I will remove those settings and test again. With dnsmasq, OK? This can never work with plain resolv.conf anyway.

Thanks.

(In reply to David Woodhouse from comment #67)
> Ack, then I will remove those settings and test again. With dnsmasq, OK?

Yes, thanks.

112 comments hidden view all 137 comments
Taylor Raack (track16) wrote :

I can also confirm that the network-manager package version 1.10.14-0ubuntu1 from bionic-proposed fixes the issue.

dwmw2 (dwmw2) wrote :

Is there a 16.04 package? This was a regression there caused by an earlier update.

I have users reporting the same bizarre behaviour I wasn't able to clearly describe before — essentially, DNS being sent out seemingly random interfaces (sometimes VPN, sometimes local). My advice to just install this package *and* manually set dns-priority=-1,dns-search=~. and get on with life even though you really shouldn't have to manually set the latter, doesn't work for the 16.04 users...

And yes, when other things stop being on fire I need to undo those settings and try to work out what's going wrong. We aren't using systemd-resolve here because historically it also hasn't worked right while dnsmasq did.

Sebastien Bacher (seb128) wrote :

@dwmw2, 'This was a regression there caused by an earlier update.' would give some details ont that? you should probably open another report specifically about that if there was a regression in a xenial update

dwmw2 (dwmw2) wrote :

@seb128 please see "In 16.04 the NetworkManager package used to carry this patch..." in the bug description above.

Mathew Hodson (mathew-hodson) wrote :

Looking at the upstream bug, it looks like the fix relies on reworking large parts of the code and wouldn't be easy to SRU to Xenial.

tags: added: verification-done-bionic
removed: verification-needed verification-needed-bionic
Steve Langasek (vorlon) wrote :

Based on comment #12 I am not sure that this is considered "verification-done" by the relevant developers and there was no comment given when the tags were changed. Resetting.

I also think there should be an affirmative test as part of this SRU that the use case I described in comment #13 has not been regressed.

tags: added: verification-needed verification-needed-bionic
removed: verification-done-bionic
description: updated

I have now done the test under [Test Case] in the initial description of this bug report.

I have a completely updated (including -proposed) Bionic machine (real iron, a Lenovo X1 Carbon 2nd gen from 2015) with network-manager 1.10.14-0ubuntu1

I have configured the Canonical VPN, both UK and US. I have turned on only the UK one. It is configured to be used only for the internal destinations on both IPv4 and IPv6.

The system in this configuration I have rebooted to be assure that all processes including the kernel are using the newest software.

Then I have followed the instructions of the test case.

When running "dig <a Canonical-internal host name>" I get immediately an answer with exit code 0 ("echo $?"), so the request was successful.

When I look into the "tcpdump" terminals, the host name gets polled through both interfaces, but naturally the answer only comes from the DNS of the VPN.

So to my understanding the bug is not fixed as the private host name gets also sent to the public DNS.

"systemd-resolve --status" lists the VPN DNS first, as link 4 and afterwards the public DNS as link3.

Good news, the network-manager SRU is not broken or wrong, but an additional SRU, on systemd, is needed to actually fix this bug.

I got a hint from Iain Lane (Laney, thank you very much) to the following fix in systemd upstream:

https://github.com/systemd/systemd/commit/a97a3b256

and backported it to Bionic's systemd package (debdiff attached). With the network-manager SRU from -proposed attached plus the patched systemd package installed the problem goes away. If I repeat the test of [Test Case] (after a reboot) the DNS requests to any of the VPN's domains go actually only to the VPN's DNS.

Changed in systemd (Ubuntu):
status: New → Fix Released
Changed in systemd (Ubuntu Bionic):
status: New → Triaged
Changed in systemd (Ubuntu):
importance: Undecided → High
Changed in systemd (Ubuntu Bionic):
importance: Undecided → High
Timo Aaltonen (tjaalton) wrote :

Hello dwmw2, or anyone else affected,

Accepted network-manager into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.10.14-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Łukasz Zemczak (sil2100) wrote :

Will be releasing network-manager without the systemd part for now as it poses no threat to the user.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package network-manager - 1.10.14-0ubuntu2

---------------
network-manager (1.10.14-0ubuntu2) bionic; urgency=medium

  [ Till Kamppeter ]
  * debian/tests/nm: Add gi.require_version() calls for NetworkManager
    and NMClient to avoid stderr output which fails the test.

  [ Iain Lane ]
  * debian/tests/control: The nm tests need dnsmasq-base and isc-dhcp-client
    too.

network-manager (1.10.14-0ubuntu1) bionic; urgency=medium

  * New stable version (LP: #1809132), including:
    - Support private keys encrypted with AES-{192,256}-CBC in libnm
      (LP: #942856)
    - Fix leak of DNS queries to local name servers when connecting to a
      full-tunnel VPN (CVE-2018-1000135) (LP: #1754671)
  * Dropped patch applied upstream:
    - debian/patches/CVE-2018-15688.patch
    - debian/patches/e91f1a7d2a6b8400b6b331d5b72287dcb5164a39.patch
  * Refreshed patches:
    - debian/patches/Don-t-make-NetworkManager-D-Bus-activatable.patch
    - debian/patches/Force-online-state-with-unmanaged-devices.patch
    - debian/patches/Read-system-connections-from-run.patch
    - debian/patches/Update-dnsmasq-parameters.patch
    - debian/patches/libnm-register-empty-NMClient-and-NetworkManager-when-loa.patch

 -- Till Kamppeter <email address hidden> Fri, 10 May 2019 13:34:00 +0200

Changed in network-manager (Ubuntu Bionic):
status: Fix Committed → Fix Released
Adam Conrad (adconrad) wrote :

The original bug report was about a regression in 16.04 with the dnsmasq integration. While I'm glad this got the ball rolling on the bionic networkd integration, let's not forget that we broke xenial? Added a xenial task for network-manager accordingly.

Changed in systemd (Ubuntu Xenial):
status: New → Invalid
dwmw2 (dwmw2) wrote :

I am receiving reports that it isn't fixed in 18.04 either. Users are still seeing DNS lookups on the local network, until they manually edit the VPN config to include:

[ipv4]
dns-priority=-1
dns-search=~.;

I thought that wasn't going to be necessary?

dwmw2, did you apply the systemd fix from comment #27? For this bug to be fixed you need BOTRH the fixed packages of network-manager and systemd.

dwmw2 (dwmw2) wrote :

These systems are using dnsmasq not systemd-resolver. This was done for historical reasons; I'm not sure of the specific bug which caused that choice.

Unfortunately, the SRU for systemd did not yet get processed. Therefore I have now uploaded this version of systemd to my PPA so that you can already test/get your problem solved. Please tell here whether it actually fixes the bug.

Here is my PPA:

https://launchpad.net/~till-kamppeter/+archive/ubuntu/ppa

Please follow this link, follow the instructions in the section "Adding this PPA to your system", then update your system with the command

sudo apt dist-upgrade

This will update only systemd as I did not upload any other package for Bionic to my PPA.

Make also sure you have the update of network-manager (1.10.14-0ubuntu2) installed. Reboot and check whether everything works correctly now.

dwmw2 (dwmw2) wrote :

We aren't using systemd-resolver for various historical reasons; we are using dnsmasq which should be expected to work. It isn't, but we have manually added the dns-priority=-1;dns-search=~. settings which make it work, as an emergency deployment when the latest NM update broke things for everyone.

dwmw2, the systemd fix was mainly meant for people with standard configuration where this fix is actually needed and solve the problem.

You are writing that adding "dns-priority=-1;dns-search=~." solves the problem for you. Where/to which file did you add this? Do you need this already with the original network-manager version of Bionic (1.10.6) or do you only need ot after the update (1.10.14)?

dwmw2 (dwmw2) wrote :

This is Bionic.

After last week's update to 1.10.14-0ubuntu2 all my VPN users (who are using dnsmasq) reported that DNS supported working for them while they were on the VPN. Some internal names were looked up correctly, others weren't.

I resolved it for them as follows:

$ sudo nmcli con modify "$COMPANY VPN" ipv4.dns-priority -1 ipv4.dns-search ~.

This matches the observations I made in comment #18 on 2019-02-04.

I believe that with 1.10.6 all $company.com DNS did get sent to the VPN and it was lookups outside the company search domains which were leaked. So it was mostly functional, but insecure. Since 1.10.14 it got worse and many (but not all) of the $company.com lookups are being leaked too. Which is a functional problem.

(For Xenial, my advice to users has been the same since March 2018 when this ticket was first filed: tell apt to hold network-manager_1.2.2-0ubuntu0.16.04.4_amd64.deb and don't let it get updated until/unless the regression is fixed.)

Steve Langasek (vorlon) wrote :

> These systems are using dnsmasq not systemd-resolver.
> This was done for historical reasons; I'm not sure of
> the specific bug which caused that choice.

NetworkManager in Ubuntu 16.04 and earlier defaulted to integrating with dnsmasq. But on 18.04 and later, this integration has been deliberately replaced with integration with systemd-resolved. If you are overriding this default integration to force the use of dnsmasq instead of systemd-resolved, that is likely not a supportable configuration.

In contrast, any bug in the systemd-resolved integration in 18.04 that would force you to work around it by switching to dnsmasq is almost certainly an SRUable bug. If you can find the information about why you switched to dnsmasq, please report this as a bug against systemd (with 'ubuntu-bug systemd') and provide a link to the bug here.

Steve Langasek (vorlon) wrote :

Due to the SRU regressions reported in LP: #1829838 and LP: #1829566, I have reverted this SRU for the moment, restoring network-manager 1.10.6-2ubuntu1.1 to bionic-updates. I am marking this bug verification-failed pending resolution of the reported regressions.

Changed in network-manager (Ubuntu Bionic):
status: Fix Released → In Progress
tags: added: verification-failed verification-failed-bionic
removed: verification-needed verification-needed-bionic
Dan Streetman (ddstreet) on 2019-05-29
Changed in systemd (Ubuntu Cosmic):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Xenial):
assignee: nobody → Dan Streetman (ddstreet)
assignee: Dan Streetman (ddstreet) → nobody
Changed in systemd (Ubuntu Cosmic):
importance: Undecided → High
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
status: Triaged → In Progress
Dan Streetman (ddstreet) on 2019-05-29
tags: added: ddstreet-next
Timo Aaltonen (tjaalton) on 2019-05-31
tags: added: verification-needed verification-needed-bionic verification-needed-cosmic
removed: verification-failed verification-failed-bionic
Changed in systemd (Ubuntu Cosmic):
status: In Progress → Fix Committed
Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
Changed in network-manager (Ubuntu Bionic):
assignee: Olivier Tilloy (osomon) → Till Kamppeter (till-kamppeter)
Changed in network-manager (Ubuntu Cosmic):
status: New → Won't Fix
tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic verification-needed-cosmic
Dan Streetman (ddstreet) on 2019-06-10
Changed in systemd (Ubuntu Bionic):
status: Fix Committed → Fix Released
Changed in systemd (Ubuntu Cosmic):
status: Fix Committed → Fix Released
tags: removed: ddstreet-next
Changed in network-manager (Ubuntu Xenial):
status: New → Confirmed
93 comments hidden view all 137 comments

Without the ipv4.dns-priority and ipv4.dns-search settings being manually set, I see the wired device still being used for its own search domain. This is what causes the problem.

It seems to work for me if I set *only* ipv4.dns-priority=-1 though.

My biggest practical problem here was that we had set ipv4.dns-priority and ipv4.dns-search options in an emergency deployment to our users after Ubuntu shipped the 1.10.14 update to 18.04 to fix this... and then when they *pulled* the update, new installations got the new config but older NM, and that didn't work correctly either.

I'm going to experiment with *just* setting ipv4.dns-priority=-1 and not ipv4.dns-search. Do we expect that to work for everyone, whether they have the updated package or not?

We should be setting ipv4.dns-priority=1 by default for full-tunnel VPNs, but at least if I can find a way to work around it that works for everyone, that'll be an improvement.

No, ipv4.dns-priority=-1 breaks older NM because there, there's no default routing domain ~. and not even any way to set it. So lookups outside the VPN domains are just refused instead of being leaked to the local nameservers as they were before.

As it happens, we configure our DHCP client to always override the discovered search domains to our corporate AD domain dom.company.com, so I can work around NM's failure to automatically set ipv4.dns-priority=-1 a different way — by manually configuring the VPN with ipv4.dns-domains=dom.company.com;company.com;. That works for both old and new NetworkManager.

> My biggest practical problem here was that we had set
> ipv4.dns-priority and ipv4.dns-search options in an emergency
> deployment to our users after Ubuntu shipped the 1.10.14 update to
> 18.04 to fix this... and then when they *pulled* the update, new
> installations got the new config but older NM, and that didn't work
> correctly either.

> I'm going to experiment with *just* setting ipv4.dns-priority=-1 and
> not ipv4.dns-search. Do we expect that to work for everyone, whether
> they have the updated package or not?

Yes, this should work the same on 1.10.14 and 1.12+ releases.

> We should be setting ipv4.dns-priority=1 by default for full-tunnel
> VPNs, but at least if I can find a way to work around it that works
> for everyone, that'll be an improvement.

Do you mean -1? Why? This will cause an opposite problem: local
queries leaked to the VPN, i.e. I look up nas.myhome and the query
goes through the VPN.

> No, ipv4.dns-priority=-1 breaks older NM because there, there's no default routing domain ~. and not even any way to set it

Which version is 'older'?

In my case, 'older' is 1.10.6. At this point I'm just trying to get things working for my users on Ubuntu 18.04. So I'm comparing 1.10.6 (the current Ubuntu package) with 1.10.14 (the update, that was briefly shipped in 18.04 updates and then withdrawn).

When the 1.10.14 update went out, it broke our users because lookups for the dom.company.com AD domain were now always going to the local network (because of our dhclient config override). We immediately rolled out a manual setting of ipv4.dns-priority=-1 (yes, not =1) to fix it.

Then the Ubuntu 1.10.14 update was withdrawn, and new installations got 1.10.6 which *doesn't* work with ipv4.dns-priority=-1, because there's no ~. routing domain and thus lookups outside the VPN domains don't work at all.

(We don't care about local lookups not working. In fact we'd prefer local lookups *not* to work, which might be part of the reason we've overridden the search domain obtained by DHCP. If we could stop *routing* to the local network from working, we would. It's an open feature request, I believe.)

But *because* I know the search domain on the local physical network will always be just 'dom.company.com', adding that explicitly in the VPN config is enough and I don't need ipv4.dns-priority=-1. Now I have a config that works with both 1.10.6 and 1.10.14. So I can wait for the regressions that caused the 1.10.14 update to be pulled, to get fixed and then maybe later I can set ipv4.dns-priority=-1 again.

Anyway, we now know that the reason my initial testing of the Ubuntu 1.10.14 backport was unsuccessful, was because of the search domain on the *local* network. I've been able to work around that, and things are now working OK.

Changed in network-manager:
status: Fix Released → Confirmed
Displaying first 40 and last 40 comments. View all 137 comments or add a comment.
This report contains Public Security information  Edit
Everyone can see this security related information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.