Bogus routes after DHCP lease change

[Ante Karamatić]

[impact]

networkd does not remove old route(s) after DHCP address change

[test case]

on a system using networkd, that is connected to a network where you can control the addresses that the DHCP server provides, setup system with networkd to get address via DHCP, e.g.

[Match]
Name=ens3

[Network]
DHCP=ipv4

(re)start networkd or reboot, so the system gets an ipv4 DHCP address, and corresponding route to the gateway.

Then on the dhcp server, change the subnet to a different subnet. On the client, once its renews its DHCP address, the server will provide a new address in the new subnet, and the client will add a new default route to the new gateway address. However, the old default route to the old gateway address isn't removed.

Note this also happens without changing the entire subnet, but is more subtle as shown in the original description.

[regression potential]

this affects how networkd handles routes, so has the potential to leave a system with partial or incorrect networking, or no networking at all. Any regression would most likely occur during networkd (re)start or during renewal of a DHCP lease, or when an interface is brought up.

[other info]

original description:
---

Netplan config:

network:
  version: 2
  renderer: networkd
  ethernets:
    eno4:
      dhcp4: no
    eno1np0:
      dhcp4: no
      addresses:
        - 172.16.0.2/24
  bridges:
    br0:
      dhcp4: yes
      interfaces:
        - eno4

On initial boot, machine got 10.0.15.109 IP address:

May 03 13:09:41 ceph2 systemd-networkd[29349]: br0: Configured
May 03 13:09:41 ceph2 systemd-networkd[29349]: br0: DHCPv4 address 10.0.15.109/23 via 10.0.15.253

At one point, DHCP server reserver this IP address and client eventually picked up new IP address:

May 03 15:01:12 ceph2 systemd-networkd[1137]: br0: DHCPv4 address 10.0.15.128/23 via 10.0.15.253

This resulted in IP addresses:

# ip -o a
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
1: lo inet6 ::1/128 scope host \ valid_lft forever preferred_lft forever
2: eno1np0 inet 172.16.0.2/24 brd 172.16.0.255 scope global eno1np0\ valid_lft forever preferred_lft forever
2: eno1np0 inet6 fe80::b226:28ff:fe53:56be/64 scope link \ valid_lft forever preferred_lft forever
6: br0 inet 10.0.15.128/23 brd 10.0.15.255 scope global dynamic br0\ valid_lft 503sec preferred_lft 503sec
6: br0 inet6 fe80::b8d7:5eff:fe6b:62a/64 scope link \ valid_lft forever preferred_lft forever

So far, everything is fine. But, the routes on the machine are bogus:

# ip r
default via 10.0.15.253 dev br0 proto dhcp src 10.0.15.109 metric 100
default via 10.0.15.253 dev br0 proto dhcp src 10.0.15.128 metric 100
10.0.14.0/23 dev br0 proto kernel scope link src 10.0.15.128
10.0.15.253 dev br0 proto dhcp scope link src 10.0.15.109 metric 100
10.0.15.253 dev br0 proto dhcp scope link src 10.0.15.128 metric 100
172.16.0.0/24 dev eno1np0 proto kernel scope link src 172.16.0.2

routes with src 10.0.15.109 should have been removed when lease was renewed. I'm not sure if this is a bug in netplan or systemd. This is 18.04, systemd 37-3ubuntu10.21, netplan 0.40.1~18.04.4.

[Ante Karamatić (ivoks)]

I forgot to mention; this puts machine in a weird state. It can be connected to, via 10.0.15.128, but it can't initiate connections to outside. By removing routes:

# ip r d default via 10.0.15.253 dev br0 proto dhcp src 10.0.15.109 metric 100
# ip r d 10.0.15.253 dev br0 proto dhcp scope link src 10.0.15.109 metric 100

machine is usable again.

[Ryan Harper (raharper)]

Can you provide the journal? At least all of:

journalctl -o short-monotonic -u systemd-networkd.service ?

[Ryan Harper (raharper)]

Looks like this issue, I think:

https://github.com/systemd/systemd/issues/12490

[Ante Karamatić (ivoks)]

It's exactly the same as that systemd issue.

[Łukasz Zemczak (sil2100)]

Hello Ante, or anyone else affected,

Accepted systemd into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/242-7ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (242-7ubuntu3.2) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.42.1-1ubuntu1 (amd64)
systemd/242-7ubuntu3.2 (ppc64el)
ndctl/unknown (armhf)
casper/1.427 (amd64)
netplan.io/0.98-0ubuntu1 (ppc64el)
munin/unknown (armhf)
linux-oem-osp1/5.0.0-1026.29 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

with dnsmasq server setup:

$ ip -4 a show ens8
3: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 1.2.3.1/24 scope global ens8
       valid_lft forever preferred_lft forever
$ cat /etc/dnsmasq.d/test

interface=ens8
bind-dynamic

no-resolv
no-poll

domain=test,1.2.3.4/24
dhcp-range=test,1.2.3.100,1.2.3.199,1m
dhcp-host=52:54:00:f7:b2:99,1.2.3.50,1m

on the test system:

ubuntu@lp1831787-e:~$ dpkg -l systemd | grep ii
ii systemd 242-7ubuntu3.2 amd64 system and service manager
ubuntu@lp1831787-e:~$ ip -4 a show dev ens8
3: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 1.2.3.50/24 brd 1.2.3.255 scope global dynamic ens8
       valid_lft 106sec preferred_lft 106sec
ubuntu@lp1831787-e:~$ ip -4 r
default via 1.2.3.1 dev ens8 proto dhcp src 1.2.3.50 metric 1024
1.2.3.0/24 dev ens8 proto kernel scope link src 1.2.3.50
1.2.3.1 dev ens8 proto dhcp scope link src 1.2.3.50 metric 1024

then on dnsmasq server, change the test system addr to .60:

$ cat /etc/dnsmasq.d/test

interface=ens8
bind-dynamic

no-resolv
no-poll

domain=test,1.2.3.4/24
dhcp-range=test,1.2.3.100,1.2.3.199,1m
dhcp-host=52:54:00:f7:b2:99,1.2.3.60,1m

ubuntu@dhcp-test:~$ sudo systemctl restart dnsmasq

on test system, wait for dhcp lease to timeout:

ubuntu@lp1831787-e:~$ ip -4 a show dev ens8
3: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 1.2.3.60/24 brd 1.2.3.255 scope global dynamic ens8
       valid_lft 114sec preferred_lft 114sec
ubuntu@lp1831787-e:~$ ip -4 r
default via 1.2.3.1 dev ens8 proto dhcp src 1.2.3.60 metric 1024
1.2.3.0/24 dev ens8 proto kernel scope link src 1.2.3.60
1.2.3.1 dev ens8 proto dhcp scope link src 1.2.3.60 metric 1024

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 242-7ubuntu3.2

---------------
systemd (242-7ubuntu3.2) eoan; urgency=medium

  [ Dan Streetman ]
  * d/extra/dhclient-enter-resolved-hook:
    - Replace use of bash-only &> with > and 2> (LP: #1849608)
  * d/p/lp1849658-resolved-set-stream-type-during-DnsStream-creation.patch:
    - Fix bug in refcounting TCP stream types (LP: #1849658)
  * d/extra/dhclient-enter-resolved-hook: cleanup temp $newstate file

  [ Rafael David Tinoco ]
  * Add support to KeepConfiguration= fixing behaviour for HA (LP: #1815101)
    - d/p/lp1815101-01-networkd-add-support-to-keep-configuration.patch
    - d/p/lp1815101-02-networkd-stop-clients-when-networkd-shuts-down.patch
    - d/p/lp1815101-03-network-add-KeepConfiguration-dhcp-on-stop.patch
    - d/p/lp1815101-04-network-make-KeepConfiguration-static-drop-DHCP-addr.patch
    - d/p/lp1815101-05-man-add-documentation-about-KeepConfiguration.patch

systemd (242-7ubuntu3.1) eoan; urgency=medium

  [ Balint Reczey ]
  * Fix shutdown and related actions from the login screen (LP: #1847896)
    File: debian/patches/logind-consider-greeter-sessions-suitable-as-display-sess.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b407dfd8c9dc81594553c27467c35b383333d74c
  * debian/gbp.conf: Set debian-branch to ubuntu-eoan
    File: debian/gbp.conf
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f399ce2cf4701a2dbb4b3505d2dd17a210d62f5c

  [ Dan Streetman ]
  * Fix bogus routes after DHCP lease change (LP: #1831787)
    Files:
    - debian/patches/lp1831787/0001-networkd-Add-back-static-routes-after-DHCPv4-lease-e.patch
    - debian/patches/lp1831787/0002-network-set-preferred-source-in-removing-route-entry.patch
    - debian/patches/lp1831787/0003-network-lower-log-level-about-critical-connection.patch
    - debian/patches/lp1831787/0004-network-reset-Link-dhcp4_configured-flag-earlier.patch
    - debian/patches/lp1831787/0005-network-split-dhcp_lease_lost-into-small-pieces.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ced3f5c2f619083f7beb164d94d4ccfe52222fe8
  * Set src address for dhcp 'classless' routes (LP: #1835581)
    File: debian/patches/lp1835581-src-network-networkd-dhcp4.c-set-prefsrc-for-classle.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6a7ef370fb1335548448920be4ae6176b67044a8
  * Allows cache=no-negative option to be set, ignoring negative answers to
    be cached (LP: #1668771)
    File: debian/patches/lp1668771-resolved-switch-cache-option-to-a-tri-state-option-s.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=27def26f5b1d1b8ba314c4a925fc1b7c43837f86

 -- Dan Streetman <email address hidden> Fri, 01 Nov 2019 16:33:08 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Dan Streetman (ddstreet)]

as disco is EOL next week, marking this won't fix for disco.

[Dan Streetman (ddstreet)]

hmm, i can't reproduce this anymore, it seems one of my other patches to bionic fixed the problem.

@ivoks, or anyone else, if you still see this with the latest systemd in bionic, please let me know.