1.10.14-0ubuntu2 breaks DNS propagation from VPN

[Jean]

I have an OpenVPN connection, and I use the update-systemd-resolved to correctly fetch the DNS from the VPN. The issue arose updating my Ubuntu 18.04: the script is no longer invoked, and even invoking it manually (sudo openvpn myfile.ovpn) does not work anymore, even if in that case systemd-resolve --status shows the new DNS on the interface correctly.

As suggested in https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1211110/comments/99, reverting to 1.10.6-2ubuntu1.1 fixes the issue.

[Sebastien Bacher (seb128)]

Thanks for new report and confirming that downgrading fixes the issue

[Till Kamppeter (till-kamppeter)]

In bug 1211110 you say:

I had this same issue and I circumvented it installing the update-systemd-resolved script through the openvpn-systemd-resolved package.

Does this mean that if you install the openvpn-systemd-resolved package your problem gets solved without downgrading network-manager? Perhaps we could add a dependency which would install this package automatically.

[Steve Langasek (vorlon)]

> Does this mean that if you install the openvpn-systemd-resolved package
> your problem gets solved without downgrading network-manager? Perhaps we
> could add a dependency which would install this package automatically.

This is a universe package which NetworkManager should not require in order to properly configure resolved.

[Till Kamppeter (till-kamppeter)]

In the initial posting you say that "systemd-resolve --status" shows the new DNS but does not get used, could you also try out the systemd fix of bug 1754671?

[Jean (alessandro-lai85)]

Sorry, what fix are you referring to? Can you point me to a specific comment?

> Does this mean that if you install the openvpn-systemd-resolved package your problem gets solved without downgrading network-manager?

Nope, I need that package from the start anyway to make everything work, I installed it when I installed my laptop with 18.04 when it was hot from the press last year.

[Till Kamppeter (till-kamppeter)]

Unfortunately, the SRU for systemd did not yet get processed. Therefore I have now uploaded this version of systemd to my PPA:

https://launchpad.net/~till-kamppeter/+archive/ubuntu/ppa

Please follow this link, follow the instructions in the section "Adding this PPA to your system", then update your system with the command

sudo apt dist-upgrade

This will update only systemd as I did not upload any other package for Bionic to my PPA.

Make also sure you have the update of network-manager (1.10.14-0ubuntu2) installed. Reboot and check whether everything works correctly now.

[Steve Langasek (vorlon)]

Due to the SRU regressions reported in LP: #1829838 and LP: #1829566, I have reverted this SRU for the moment, restoring network-manager 1.10.6-2ubuntu1.1 to bionic-updates.

network-manager 1.10.14-0ubuntu2 remains available in bionic-proposed for testing purposes.

[Till Kamppeter (till-kamppeter)]

I am rather new to network-manager internals, but could you try the command

sudo nmcli con modify "$COMPANY VPN" ipv4.dns-priority -1 ipv4.dns-search ~.

Does this solve your problem?

[Till Kamppeter (till-kamppeter)]

1. /etc/systemd/journald.d/noratelimit.conf containing

RateLimitIntervalSec=0
RateLimitBurst=0

2. /etc/NetworkManager/conf.d/debug.conf

[logging]
level=TRACE
domains=ALL

Then restart journald:

sudo systemctl restart systemd-journald

and NetworkManager:

sudo systemctl restart network-manager

Then you get the full debug log of NetworkManager via

journalctl -u NetworkManager

After all that, reboot and/or connect to your VPN and do

journalctl -u NetworkManager > log.txt

and attach the log.txt file to this bug report. Do not compress the file and do not package it together with other files.

[Till Kamppeter (till-kamppeter)]

Sorry there was a part missing. Let us try again:

Please create the following files (and directories if needed for them):

1. /etc/systemd/journald.d/noratelimit.conf containing

RateLimitIntervalSec=0
RateLimitBurst=0

2. /etc/NetworkManager/conf.d/debug.conf

[logging]
level=TRACE
domains=ALL

Then restart journald:

sudo systemctl restart systemd-journald

and NetworkManager:

sudo systemctl restart network-manager

Then you get the full debug log of NetworkManager via

journalctl -u NetworkManager

After all that, reboot and/or connect to your VPN and do

journalctl -u NetworkManager > log.txt

and attach the log.txt file to this bug report. Do not compress the file and do not package it together with other files.

[Till Kamppeter (till-kamppeter)]

The SRU for systemd has arrived in bionic-proposed (see bug 1754671). Could you make sure that you have installed BOTH the network-manager and systemd SRUs from bionic-proposed (to make sure that I did not perhaps do something wrong with the systemd update in my PPA). Versions should be:

network-manager: 1.10.14-0ubuntu2
systemd: 237-3ubuntu10.22

Does this combination solve your problem? Please reboot to make sure that everything gets replaced by the newer versions.

If this still does not help, please follow my instructions in comment #8 and comment #10.

[Till Kamppeter (till-kamppeter)]

Jean, we need your cooperation to find out whether our SRU of Network Manager for Bionic (NM 1.10.14) actually has a regression or whether your problem was caused by the missing update of systemd. With this information we can help many other users of Bionic who suffer other bugs and for which we have introduced this SRU.

So I want to ask you whether you could install BOTH the SRUs for systemd and network-manager as I have described in comment #11 and if after installing these packages and rebooting the problem still persists follow my instructions of comment #10 to provide us additional information to solve the problem.

Thank you very much in advance.

[Till Kamppeter (till-kamppeter)]

Jean, note that if you test the two SRUs and confirm us that they solve the problem for you, you unblock this SRU and give us way to provide further SRUs on Network Manager in the future, as Bionic has still some years to go.

[Till Kamppeter (till-kamppeter)]

Jean, could you for your tests in addition to installing the two SRUs uninstall the openvpn-systemd-resolved package with

sudo apt purge openvpn-systemd-resolved

and then reboot?

The openvpn-systemd-resolved is actually not needed for network-manager with systemd-resolved in Ubuntu and I have verified the SRUs (in bug 1754671) with openvpn-systemd-resolved NOT installed.

[Jean (alessandro-lai85)]

Can you please explain step-by-step what I need to do to test? I understand purging the script, but I don't know what should I do in regard to the Network package. I'm still on 18.04.

[Till Kamppeter (till-kamppeter)]

Please make sure your system is fully up-to-date. Especially make sure your systemd package is 237-3ubuntu10.22 or newer. Now check your network-manager package. If it is not 1.10.14-0ubuntu2 (probably it is 1.10.6) update it from bionic-proposed as described in comment #11 of bug #1754671. Then reboot. Now check whether the problem does not occur any more for you, for example by doing the steps of the section "[Test case]" in the description of bug #1754671.

If the problem still persists, follow the instructions of my comment #9. Also post the output of

systemd-resolve --status

here.

[Jean (alessandro-lai85)]

My systemd is newer (237-3ubuntu10.24), instead network manager is locked at 1.10.6 (1.10.6-2ubuntu1.1). I've upgraded network-manager as suggested to 1.10.14-0ubuntu2 and rebooted.

I've shut down both my Wifi and LAN since I'm at the office, and hooked up my phone with USB tethering to use an external connection and use the VPN to my office.

First test: it works. I get my DNS from the VPN:

Link 61 (tun2)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 172.27.0.42
                      172.27.0.33
          DNS Domain: ~.

Following the test case in #1754671: when I enable the split connection, this is the full result of systemd-resolve --status, where I do not see any DNS coming from the VPN:

Global
          DNS Domain: mycompany.com
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 48 (tun1)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 43 (tun0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 42 (enp0s20f0u2)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.42.129
          DNS Domain: ~.

Link 11 (br-b2c3b7f9b208)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 10 (br-775248335fa8)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 9 (br-6671ba352ece)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 8 (br-642ed19f0ac7)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 7 (br-5699f03e12bc)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 6 (br-1f97d7363f0e)
      Current Scopes: none
       L...

[Till Kamppeter (till-kamppeter)]

Jean, thank you for the input, could you retest these two cases (with and without split connection) but create debug logs of Network Manager and systemd-resolved, following the instructions on https://wiki.ubuntu.com/DebuggingNetworkManager (and/or of my comment #10)?

[Till Kamppeter (till-kamppeter)]

Sorru for the late reply, I had a lot to do.

I am not sure whether you have correctly taken the logs.

Both are taken on July 25 but when you look into them, they span the following time frames:

log-network-manager-with-segmentation.txt: Jan 31 - July 1
log-network-manager-without-segmentation.txt: Jan 31 - July 25

A "diff" shows also that all content of of the first is also in the second. It seems that the first got cut off during the upload.

As they contain all history back to January, is it possible that the longer one (log-network-manager-without-segmentation.txt) contains both tests? Could you tell in which order you did the tests and which parts of the log reflect which test?

Or could you re-test but this time reading the time stamps in the log before and after each test and upload only the logs of these time spans?

[Till Kamppeter (till-kamppeter)]

The longer of your two attached log files, log-network-manager-without-segmentation.txt, seems to contain at least one of your two tests, so I extracted the part from your test starting (last start of network-manager, with log verbosity set to contain <trace> and <debug> messages) up to the end. I have attached this part to this posting as log-network-manager-without-segmentation-2.txt.

When one searches for the string "current configuration:" one finds where DNS configuration is changed due to network interfaces being added or removed. The last of these occurrences is the only place where your VPN (with your office's DNSes 172.27.0.42 and 172.27.0.33) gets added to your phone's internet access (with DNS 192.168.42.129). Only in this acase you get an error with pacrunner (PAC = Proxy Auto Configuration, see https://wiki.gnome.org/Projects/NetworkManager/Proxies) but seems to succeed later. There are also some other errors visible. Here are some lines of the log:

----------
lug 25 16:53:11 CLIFMI085 NetworkManager[21584]: <trace> [1564066391.4886] dns-mgr: current configuration: [{'nameservers': <['172.27.0.42', '172.27.0.33']>, 'interface': <'tun2'>, 'priority': <50>, 'vpn': <true>}, {'nameservers': <['192.168.42.129']>, 'interface': <'enp0s20f0u2'>, 'priority': <100>, 'vpn': <false>}]
lug 25 16:53:11 CLIFMI085 NetworkManager[21584]: <debug> [1564066391.4895] dispatcher: (32) (tun2) dispatching action 'vpn-up'
lug 25 16:53:11 CLIFMI085 NetworkManager[21584]: <debug> [1564066391.4900] pacrunner: call[0x563b6899bc30]: send: new config ({'Interface': <'tun2'>, 'Method': <'direct'>, 'BrowserOnly': <false>, 'Domains': <['172.31.252.244/24', '172.27.0.0/24', '172.27.240.0/20', '192.168.222.0/24', '185.48.32.186/32', '80.247.66.0/24', '87.248.32.84/32', '178.239.180.149/32', '80.247.66.96/32', '87.248.32.223/32', '10.6.0.0/16', '172.31.252.0/24']>},)
lug 25 16:53:11 CLIFMI085 NetworkManager[21584]: <trace> [1564066391.4900] pacrunner: call[0x563b6899bc30]: sending...
lug 25 16:53:11 CLIFMI085 NetworkManager[21584]: <debug> [1564066391.4907] pacrunner: call[0x563b6899bc30]: sending failed: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.pacrunner" does not exist
lug 25 16:53:11 CLIFMI085 NetworkManager[21584]: <debug> [1564066391.4907] prope
rties-changed[0x563b688b9c10]: type NMDeviceTun, iface NMDBusDeviceStatisticsSke
leton: {'TxBytes': <uint64 48>}
lug 25 16:53:11 CLIFMI085 NetworkManager[21584]: <debug> [1564066391.4908] prope
rties-changed[0x563b688b9c10]: type NMDeviceTun, iface NMDBusDeviceTunSkeleton:
{'Dhcp4Config': <objectpath '/'>, 'Dhcp6Config': <objectpath '/'>, 'Ip4Address':
 <uint32 4110163884>, 'Ip4Config': <objectpath '/'>, 'Ip6Config': <objectpath '/
'>, 'IpInterface': <''>, 'Managed': <true>, 'State': <uint32 20>, 'StateReason': <(uint32 20, uint32 41)>}
lug 25 16:53:11 CLIFMI085 NetworkManager[21584]: <debug> [1564066391.4909] device[0x563b688b9c10] (tun2): ip4-config: update (commit=0, new-config=0x563b688ab4a0)
lug 25 16:53:11 CLIFMI085 NetworkManager[21584]: <debug> [1564066391.4910] device[0x563b688b9c10] (tun2): ip6-config: update (commit=0, new-config=0x563b688b4940)
lug 25 16:53:11 CLIFMI085 NetworkMana...

[Jean (alessandro-lai85)]

I did the tests in succession, obviously the longer one is the second in the sequence. Sorry for not noticing that the logs had the whole history.

So you can basically take the last timestamp from the shorter log and use it to cut the longer one. The test spans just a couple of minutes before and after the cut, so you can restrict your search a lot. I've tried to not do anything else while recording.

[Till Kamppeter (till-kamppeter)]

The shorter one ends at Jul 1, long before I asked you for the tests, and also its last line looks broken. It is mot probably incomplete, somehow a part of it got lost during upload.
In the longer one I have found one of your two tests. I have cut it at the beginning of that test and uploaded the resulting log to this bug report as attachment of comment #22.

[Jean (alessandro-lai85)]

Ouch, it go truncated locally, not during upload :( The local copy of that file is truncated too.

Since you got the one without segmentation, you only got the logs from the one that is currently working.

I re-did the experiment, and using the `--since` option of journalctl I've cut the logs to the bare minimum. This is the log with segmentation, where the DNS are not getting set from the VPN.

[Till Kamppeter (till-kamppeter)]

Jean, a new Bionic SRU for bug 1754671 got issued, now much less invasive simply backporting the fixes and not being a full upstream update. Please follow the instructions in bug 1754671. If you are still running 1.10.14 (the old SRU) please downgrade to the old network-manager and then do the updsate to get the new SRU.

Please check and report here whether the regressions you observed in 1.10.14 do not occur in the new SRU. Thanks.

[Jean (alessandro-lai85)]

Hi Till, I've installed network-manager 1.10.6-2ubuntu1.2 from proposed as suggested in https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1754671/comments/143 and no, the issue is still there: if I use the split VPN connection I do not get the DNS under systemd-resolve --status; if I route all traffic through the VPN instead it works.

[Till Kamppeter (till-kamppeter)]

Jean, does this mean that 1.10.6-2ubuntu1.1 works perfectly for you and 1.10.6-2ubuntu1.2 shows the problem you mention?

[Jean (alessandro-lai85)]

Nope, behavior is the same between 1.10.6-2ubuntu1.1 and 1.10.6-2ubuntu1.2: broken if using split traffic between VPN and normal connection.

[DKA (kopax)]

`sudo /usr/sbin/ModemManager --debug`:

```
ModemManager[14619]: <info> [1574911308.752025] ModemManager (version 1.10.0) starting in system bus...
ModemManager[14619]: <debug> [1574911308.753455] create MMSleepMonitor singleton (0x5578a9df8ec0)
ModemManager[14619]: <debug> [1574911308.756022] Bus acquired, creating manager...
ModemManager[14619]: <debug> [1574911308.759953] [filter] created
ModemManager[14619]: <debug> [1574911308.760000] [filter] explicit whitelist: yes
ModemManager[14619]: <debug> [1574911308.760015] [filter] virtual devices forbidden: yes
ModemManager[14619]: <debug> [1574911308.760030] [filter] net devices allowed: yes
ModemManager[14619]: <debug> [1574911308.760044] [filter] cdc-wdm devices allowed: yes
ModemManager[14619]: <debug> [1574911308.760056] [filter] tty devices:
ModemManager[14619]: <debug> [1574911308.760069] [filter] blacklist applied: yes
ModemManager[14619]: <debug> [1574911308.760082] [filter] manual scan only applied: yes
ModemManager[14619]: <debug> [1574911308.760094] [filter] platform driver check: yes
ModemManager[14619]: <debug> [1574911308.760105] [filter] driver check: no
ModemManager[14619]: <debug> [1574911308.760118] [filter] cdc-acm interface check: no
ModemManager[14619]: <debug> [1574911308.760131] [filter] with net check: no
ModemManager[14619]: <debug> [1574911308.760143] [filter] default: allowed
ModemManager[14619]: <debug> [1574911308.760239] [plugin manager] looking for plugins in '/usr/lib/x86_64-linux-gnu/ModemManager'
ModemManager[14619]: <debug> [1574911308.760922] [plugin manager] loaded plugin 'Linktop'
ModemManager[14619]: <debug> [1574911308.761178] [plugin manager] loaded plugin 'Via CBP7'
ModemManager[14619]: <debug> [1574911308.761397] [plugin manager] loaded plugin 'MTK'
ModemManager[14619]: <debug> [1574911308.761616] [plugin manager] loaded plugin 'Pantech'
ModemManager[14619]: <debug> [1574911308.761973] [plugin manager] loaded plugin 'u-blox'
ModemManager[14619]: <debug> [1574911308.762191] [plugin manager] loaded plugin 'Quectel'
ModemManager[14619]: <debug> [1574911308.762390] [plugin manager] loaded plugin 'Haier'
ModemManager[14619]: <debug> [1574911308.762716] [plugin manager] loaded plugin 'Novatel'
ModemManager[14619]: <debug> [1574911308.763106] [plugin manager] loaded plugin 'Fibocom'
ModemManager[14619]: <debug> [1574911308.763427] [plugin manager] loaded plugin 'Thuraya'
ModemManager[14619]: <debug> [1574911308.763640] [plugin manager] loaded plugin 'Sierra'
ModemManager[14619]: <debug> [1574911308.763965] [plugin manager] loaded plugin 'Telit'
ModemManager[14619]: <debug> [1574911308.764286] [plugin manager] loaded plugin 'Samsung'
ModemManager[14619]: <debug> [1574911308.764616] [plugin manager] loaded plugin 'Ericsson MBM'
ModemManager[14619]: <debug> [1574911308.764838] [plugin manager] loaded plugin 'AnyDATA'
ModemManager[14619]: <debug> [1574911308.765051] [plugin manager] loaded plugin 'Motorola'
ModemManager[14619]: <debug> [1574911308.765269] [plugin manager] loaded plugin 'Nokia'
ModemManager[14619]: <debug> [1574911308.765609] ...

[DKA (kopax)]

Sorry for that long post, I am having an issue I can't send SMS and it seems to be related to a certain org.pacmac , any clue?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jean (alessandro-lai85)]

Changing status to this bug since:
 - it affects multiple colleagues of mine with the same setup
 - it affects me under Ubuntu 20.04 too, fully updated with the following versions:
   - systemd 245.4-4ubuntu3.1
   - network-manager 1.22.10-1ubuntu2.1
   - network-manager-openvpn 1.8.12-1
   - openvpn-systemd-resolved 1.3.0-3

I also tried with and without openvpn-systemd-resolved, no change.

[Jean (alessandro-lai85)]

To confirm again:

$ systemd-resolve --status | grep "Current DNS" -A 2 -B 1
    DNSSEC supported: no
  Current DNS Server: 192.168.1.1
         DNS Servers: 192.168.1.1
          DNS Domain: ~.

This remains true even if I manually force the right DNS in the VPN settings.

Attaching syslog of a split connection being established, with correctly reports receiving the right DNS from the VPN server, but still obtains the same result at systemd level.

[Jean (alessandro-lai85)]

I've found a workaround for the issue: using `nm-connection-editor` I've put my company's domains into the list of additional domain searches, and this makes the split tunnel works with the domains propagating correctly.