Issue with detecting new state in dhclient-enter-hooks.d/resolved

Bug #1850009 reported by Ivan Brawley on 2019-10-27
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Undecided
Unassigned

Bug Description

After upgrading to 19.10 I noticed that resolving hostnames stopped working.

Quick release, which package and version info:

root@caret:~# lsb_release -r
Release: 19.10
root@caret:~# dpkg -S /etc/dhcp/dhclient-enter-hooks.d/resolved
systemd: /etc/dhcp/dhclient-enter-hooks.d/resolved
root@caret:~# apt-cache policy systemd
systemd:
  Installed: 242-7ubuntu3
  Candidate: 242-7ubuntu3
  Version table:
 *** 242-7ubuntu3 500
        500 http://au.archive.ubuntu.com/ubuntu eoan/main amd64 Packages
        100 /var/lib/dpkg/status

The file /etc/resolv.conf was pointing to the local instance of systemd-resolved however there was no search line, indicating something wasn't quite right.

Restarting the systemd-resolved service did recreate /etc/resolv.conf with expected search line.

After much debugging, I've worked out what was going wrong.

The dhcp hook file /etc/dhcp/dhclient-enter-hooks.d/resolved has a bug in the resolver state change logic.

This may also impact on tickets 1805183, 1832050, 1832053 and maybe others.

When doing the md5sum of the old and new state, the command is wrong. It isn't sending STDOUT (and optionally STDERR) to the temp file but rather putting the md5sum command into the background and then creating an empty file for both states. Which obviously isn't different, so systemd-resolved doesn't restart.

Quick diff of a working version (I'm not capturing STDERR but one can with "2>&1")

======= BEGIN PATCH =======
*** .brk-resolved-20191027 2019-10-27 11:19:56.210069032 +1030
--- resolved 2019-10-27 11:30:17.156381734 +1030
***************
*** 30,36 ****
                mkdir -p $statedir

                oldstate="$(mktemp)"
! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf &> $oldstate
                if [ -n "$new_domain_name_servers" ] ; then
                    cat <<EOF >$statedir/isc-dhcp-v4-$interface.conf
  [Resolve]
--- 30,36 ----
                mkdir -p $statedir

                oldstate="$(mktemp)"
! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf > $oldstate
                if [ -n "$new_domain_name_servers" ] ; then
                    cat <<EOF >$statedir/isc-dhcp-v4-$interface.conf
  [Resolve]
***************
*** 55,61 ****
                fi

                newstate="$(mktemp)"
! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf &> $newstate
                if ! cmp $oldstate $newstate; then
                    systemctl try-reload-or-restart systemd-resolved.service
                fi
--- 55,61 ----
                fi

                newstate="$(mktemp)"
! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf > $newstate
                if ! cmp $oldstate $newstate; then
                    systemctl try-reload-or-restart systemd-resolved.service
                fi
======= END PATCH =======

Also, the script removes the script removes the $oldstate file but doesn't remove the $newstate file.

Not sure how this made it through QA testing... But breaking DHCP client shouldn't happen like this.

ivan.

ProblemType: Bug
DistroRelease: Ubuntu 19.10
Package: network-manager 1.20.4-2ubuntu2
ProcVersionSignature: Ubuntu 5.3.0-19.20-generic 5.3.1
Uname: Linux 5.3.0-19-generic x86_64
ApportVersion: 2.20.11-0ubuntu8
Architecture: amd64
Date: Sun Oct 27 11:48:33 2019
InstallationDate: Installed on 2017-04-21 (918 days ago)
InstallationMedia: Ubuntu-Server 17.04 "Zesty Zapus" - Release amd64 (20170412)
IpRoute:
 default via 172.16.1.14 dev enp0s5
 150.101.89.32/28 dev enp0s5 scope link
 169.254.0.0/16 dev enp0s5 scope link metric 1000
 172.16.1.0/24 dev enp0s5 proto kernel scope link src 172.16.1.120
IwConfig:
 lo no wireless extensions.

 enp0s5 no wireless extensions.
NetworkManager.state:
 [main]
 NetworkingEnabled=false
 WirelessEnabled=true
 WWANEnabled=true
ProcEnviron:
 LANGUAGE=en_AU:en
 TERM=vt100
 PATH=(custom, no user)
 LANG=en_AU.UTF-8
 SHELL=/bin/bash
RfKill:

SourcePackage: network-manager
UpgradeStatus: Upgraded to eoan on 2019-10-26 (0 days ago)
nmcli-con:

nmcli-dev:
 DEVICE TYPE STATE IP4-CONNECTIVITY IP6-CONNECTIVITY DBUS-PATH CONNECTION CON-UUID CON-PATH
 enp0s5 ethernet unmanaged unknown unknown /org/freedesktop/NetworkManager/Devices/2 -- -- --
 lo loopback unmanaged unknown unknown /org/freedesktop/NetworkManager/Devices/1 -- -- --
nmcli-nm:
 RUNNING VERSION STATE STARTUP CONNECTIVITY NETWORKING WIFI-HW WIFI WWAN-HW WWAN
 running 1.20.4 connected started unknown disabled enabled enabled enabled enabled

Ivan Brawley (brawley) wrote :
affects: network-manager (Ubuntu) → systemd (Ubuntu)
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
Peteris Dzenis (peterdze) wrote :

Hello,

I can confirm this issue: name resolution was not working after upgrading to Ubuntu 19.10. It was possible to fix it until the next reboot (or next networking service restart) by restarting systemd-resolved service. Located the same issue and temporary resolved it by commenting out `if' condition, forcing systemd-resolved service to always reload/restart as it was in previous Ubuntu versions.

These are the changes to /etc/dhcp/dhclient-enter-hooks/resolved file:

--- resolved.original 2019-10-29 15:43:51.940561949 +0200
+++ resolved 2019-10-29 15:44:24.804941675 +0200
@@ -56,9 +56,9 @@

               newstate="$(mktemp)"
               md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf &> $newstate
- if ! cmp $oldstate $newstate; then
+ #if ! cmp $oldstate $newstate; then
                   systemctl try-reload-or-restart systemd-resolved.service
- fi
+ #fi

               rm $oldstate
           }

Best regards,
Peteris

Ivan Brawley (brawley) wrote :

This bug is also being handled in #1805183. Comment #9 has a much better patch (along the same lines as mine but also captures STDERR from md5sum). Though it doesn't have Eoan included in the list of affected versions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers