systemd-resolved does not reset DNS server and search domain list properly after VPN disconnect

Bug #1975667 reported by stan383
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Fix Released
Medium
Unassigned
Jammy
Fix Released
Medium
Unassigned

Bug Description

[Impact]

Networking components such as VPNs that rely on systemd-resolved's API to configure search domains may inadvertently leave the network configuration in a bad state. This is a result of a broken systemd-resolved API.

[Test Plan]
* On a jammy host, configure a couple search domains with resolvectl:

$ resolvectl domain <network interface> search1.internal search2.internal
$ resolvectl domain <network interface>

* In any case, both domains should be displayed. Then, attempt to clear the configured domains:

$ resolvectl domain <network interface> ""
$ resolvectl domain <network interface>

* On a patched system, the two domains should no longer be displayed. On an un-patched system, one of the domains will still be configured.

[Where problems could occur]
This patch touches the logic that configures search domains in systemd-resolved. If the patch caused regressions, it would be related to the set of configured search domains.

[Original Description]

Hi,
in Ubuntu 21.10 I am facing a problem with DNS server list and search domain list is not properly reset back to the previous values after a VPN is disconnected. I reproduced this in Ubuntu 21.10 instance which was upgraded from the older version of Ubuntu as well as in Live USB Ubuntu 21.10 so it is not an "upgrade issue".

I use this resolv.conf symlink:
/etc/resolv.conf -> ../run/systemd/resolve/resolv.conf

Actual behavior:
VPN connect will add VPN's DNS servers and search domains into /etc/resolv.conf. When VPN is disconnected there are some of the VPN's DNS server and search domain entries left there, so it is not reset back properly.

Desired behavior:
VPN connect will add VPN's DNS servers and search domains into /etc/resolv.conf. When VPN is disconnected DNS servers and search domain list is restored to exactly the same state as was prior to the VPN connection.

Steps for reproducing:
1. Before VPN is connected this is the DNS server and search domain list in /etc/resolv.conf:

nameserver 192.168.122.1
search .

2. Once the VPN is connected, we see there were VPN's DNS server and serach domain list entries added:

nameserver 2xx.xx.xx.x0
nameserver 2xx.xx.xx.x1
nameserver 192.168.122.1
search domain1.local domain2.internal domain3.internal

3. After VPN disconnection, we see the DNS server and search domain list in /etc/resolv.conf is not restored to the state at point (1.) and some entries from VPN is being kept there:

nameserver 2xx.xx.xx.x1
nameserver 192.168.122.1
search domain2.internal domain3.internal

ProblemType: Bug
DistroRelease: Ubuntu 21.10
Package: systemd 248.3-1ubuntu8
ProcVersionSignature: Ubuntu 5.13.0-19.19-generic 5.13.14
Uname: Linux 5.13.0-19-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu70
Architecture: amd64
CasperMD5CheckResult: pass
CasperVersion: 1.465
CurrentDesktop: ubuntu:GNOME
Date: Wed May 25 06:06:05 2022
LiveMediaBuild: Ubuntu 21.10 "Impish Indri" - Release amd64 (20211012)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU USB Tablet
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
 /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/15p, 5000M
 /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/15p, 480M
     |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 480M
MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/casper/vmlinuz file=/cdrom/preseed/username.seed maybe-ubiquity quiet splash ---
SourcePackage: systemd
SystemdDelta:
 [EXTENDED] /usr/lib/systemd/system/rc-local.service → /usr/lib/systemd/system/rc-local.service.d/debian.conf
 [EXTENDED] /usr/lib/systemd/system/systemd-localed.service → /usr/lib/systemd/system/systemd-localed.service.d/locale-gen.conf
 [EXTENDED] /usr/lib/systemd/system/user@.service → /usr/lib/systemd/system/user@.service.d/timeout.conf

 3 overridden configuration files found.
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.release: 0.0
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.14.0-2
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-6.0
dmi.modalias: dmi:bvnSeaBIOS:bvr1.14.0-2:bd04/01/2014:br0.0:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-6.0:sku:cvnQEMU:ct1:cvrpc-q35-6.0:
dmi.product.name: Standard PC (Q35 + ICH9, 2009)
dmi.product.version: pc-q35-6.0
dmi.sys.vendor: QEMU

Related branches

Revision history for this message
stan383 (stan383) wrote :
Revision history for this message
Nick Rosbrook (enr0n) wrote :

I think there are actually two similar bugs here. The first I think is caused by [1], which I have confirmed is present in impish, but not focal or jammy. This can be demonstrated by the following:

$ resolvectl dns eth0 8.8.8.8 8.8.4.4 1.1.1.1
$ resolvectl dns eth0
Link 110 (eth0): 8.8.8.8 8.8.4.4 1.1.1.1
$ resolvectl dns eth0 "" # This SHOULD clear all DNS servers
$ resolvectl dns eth0
Link 110 (eth0): 8.8.4.4 1.1.1.1 # Only 8.8.8.8 was removed.

I think the second issue is caused by [2], which I have confirmed is present in impish and jammy, but not focal. This can be demonstrated by the following:

$ resolvectl domain eth0 search1.internal search2.internal search3.internal
$ resolvectl domain eth0
Link 2 (ens3): search1.internal search2.internal search3.internal
$ resolvectl domain eth0 "" # This SHOULD clear all search domains
$ resolvectl domain eth0
Link 2 (ens3): search2.internal search3.internal # Only search1.internal was removed

[1] https://github.com/systemd/systemd/issues/19651
[2] https://github.com/systemd/systemd/issues/23027

tags: added: rls-ii-incoming rls-jj-incoming
tags: added: fr-2418
Changed in systemd (Ubuntu):
importance: Undecided → Medium
Changed in systemd (Ubuntu Jammy):
importance: Undecided → Medium
Changed in systemd (Ubuntu):
status: New → Confirmed
Changed in systemd (Ubuntu Jammy):
status: New → Confirmed
tags: added: rls-ii-notfixing
removed: rls-ii-incoming rls-jj-incoming
Nick Rosbrook (enr0n)
description: updated
Revision history for this message
Lukas Märdian (slyon) wrote :

The fix is already included in upstream v251 (Kinetic+)

Changed in systemd (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello stan383, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Jammy):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/249.11-0ubuntu3.5)

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.5) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.48.2-0ubuntu1 (ppc64el)
systemd/249.11-0ubuntu3.5 (ppc64el)
udisks2/2.9.4-1ubuntu2 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

tags: added: foundations-todo
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello stan383, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Nick Rosbrook (enr0n) wrote :

I have verified this fix using systemd 249.11-0ubuntu3.6 from jammy-proposed:

nr@clean-jammy-amd64:~$ apt-cache policy systemd
systemd:
  Installed: 249.11-0ubuntu3.6
  Candidate: 249.11-0ubuntu3.6
  Version table:
 *** 249.11-0ubuntu3.6 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.4 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
nr@clean-jammy-amd64:~$ sudo resolvectl domain ens3 search1.internal search2.internal
nr@clean-jammy-amd64:~$ sudo resolvectl domain ens3
Link 2 (ens3): search1.internal search2.internal
nr@clean-jammy-amd64:~$ sudo resolvectl domain ens3 ""
nr@clean-jammy-amd64:~$ sudo resolvectl domain ens3
Link 2 (ens3):

tags: added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/249.11-0ubuntu3.6)

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.6) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

stunnel4/3:5.63-1build1 (amd64)
munin/2.0.57-1ubuntu2 (amd64)
corosync/unknown (s390x)
conntrack-tools/unknown (s390x)
exim4/4.95-4ubuntu2.1 (ppc64el)
umockdev/0.17.7-1 (armhf)
netplan.io/0.104-0ubuntu2.1 (amd64)
initramfs-tools/0.140ubuntu13 (amd64)
dovecot/unknown (s390x)
network-manager/1.36.6-0ubuntu2 (amd64)
cups/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Nick Rosbrook (enr0n) wrote :

The autopkgtest regressions for systemd 249.11-0ubuntu3.6 in jammy-proposed were all resolved with retries.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.9 KiB)

This bug was fixed in the package systemd - 249.11-0ubuntu3.6

---------------
systemd (249.11-0ubuntu3.6) jammy; urgency=medium

  * Deny-list TEST-58-REPART on ppc64el (LP: #1988994)
    File: debian/patches/lp1988994-Deny-list-TEST-58-REPART-on-ppc64el.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d2ed3cc1d223bf35015b15ff83b50156b58f0f38

systemd (249.11-0ubuntu3.5) jammy; urgency=medium

  [ Nick Rosbrook ]
  * Ensure dns_search_domain_unlink_marked removes all marked domains (LP: #1975667)
    File: debian/patches/lp1975667-Ensure-dns_search_domain_unlink_marked-removes-all-marked.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=919d5ddedd5bb8b45ab9437bf42d66c2821bb074
  * core,firstboot: workaround timezone issues on Ubuntu Core (LP: #1981042)
    Thanks to Robert Ancell for preparing the patch.
    File: debian/patches/lp1981042-core-firstboot-workaround-timezone-issues-caused-by-Ubunt.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b15546361b549217908fb6ca5d473be23d7fa757
  * network: do not remove localhost address (LP: #1979951)
    File: debian/patches/lp1979951-network-do-not-remove-localhost-address.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=2cd88391cce9fe95a486ae6dd214c12f236f3881
  * units: remove the restart limit on the modprobe@.service (LP: #1982462)
    File: debian/patches/lp1982462-units-remove-the-restart-limit-on-the-modprobe-.service.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8f0acd1b2fbb8eed1259c34963e5e9b201bef900
  * pstore: do not try to load mtdpstore (LP: #1981622)
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=15225032c3657f5906ee49d48929f9295a8664a0
  * core/mount: downgrade log level about several mkdir failures (LP: #1979952)
    Files:
    - debian/patches/lp1979952-Revert-core-mount-fail-early-if-directory-cannot-be-creat.patch
    - debian/patches/lp1979952-core-mount-downgrade-log-level-about-several-mkdir-failur.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ee8cfcf500698fab2e990de291ecf4c3ab87a4ae
  * debian/control: add Recommends: systemd-hwe-hwdb to udev.
    The systemd-hwe-hwdb brings in additional hwdb rules for HWE, so we want
    those installed with udev by default.
    File: debian/control
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4a7a3258c33201cca305956820fcc6bcd6052d76
  * hwdb: implement --root option for systemd-hwdb query (LP: #1988078)
    Files:
    - debian/libsystemd0.symbols
    - debian/patches/lp1988078-hwdb-implement-root-option-for-systemd-hwdb-query.patch
    - debian/patches/lp1988078-sd-hwdb-add-sd_hwdb_new_from_path.patch
    - debian/patches/lp1988078-sd-hwdb-include-sys-stat.h-in-hwdb-internal.h.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=937fef96c858f2f2042bf71032f315647c14add0

  [ Luca Boccassi ]
  * Enable systemd-repart and ship it in a new s...

Read more...

Changed in systemd (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.