systemd fails to setup static routes at boot when using DHCP

[Alexey Zagarin]

[impact]

a systemd-networkd configuration that uses ipv4 dhcp but ignores the dhcp-provided route, and instead sets up a static route, and also does not include a static ipv4 address, fails to actually create the static route.

This is due to networkd attempting to set up the static route before the dhcp ipv4 address is assigned, and the kernel does not allow creation of the route before setting up corresponding ipv4 address.

this results in a network that does have the dhcp-provided ipv4 address, but is missing both its dhcp-provided route (because networkd is configured to ignore it) and the static route (because networkd was not able to create it).

[test case]

(remove or unconfigure netplan, so it will not conflict with this manual networkd configuration)

create a networkd config file, e.g.:

$ cat /etc/systemd/network/10-eth0.network
[Match]
Name=eth0

[Network]
DHCP=ipv4

[Route]
Destination=10.0.0.0/8
Gateway=10.202.51.1

[DHCP]
UseRoutes=false

then *reboot* the system, and check that the static route was not applied:

$ ip r
10.202.51.0/24 dev eth0 proto kernel scope link src 10.202.51.254

note that because networkd does not remove ipv4 addresses that it manages (including dhcpv4 addresses), restarting networkd after the initial boot
will correctly create the static route, e.g.:

ubuntu@lp1836695-b:~$ ip r
10.202.51.0/24 dev eth0 proto kernel scope link src 10.202.51.254
ubuntu@lp1836695-b:~$ sudo systemctl restart systemd-networkd
ubuntu@lp1836695-b:~$ ip r
10.0.0.0/8 via 10.202.51.1 dev eth0 proto static
10.202.51.0/24 dev eth0 proto kernel scope link src 10.202.51.254

[regression potential]

adjusting how networkd works always carries the risk of breaking networking.

TBD detailed regression potential after analyzing fix.

[other info]

original description:

--

Consider the following setup:

network:
  version: 2
  renderer: networkd
  ethernets:
    ens4:
      dhcp-identifier: mac
      dhcp4: yes
      dhcp4-overrides:
        use-dns: no
        use-ntp: no
        send-hostname: no
        use-hostname: no
        use-routes: no
      routes:
      - to: 10.0.0.0/8
        via: 10.50.0.1
      optional: true

Thus I only need to get the IP address by DHCP, then add some static routes. This setup doesn't work. Apparently `routes` keyword only works when using static addresses.

[Lars Erik Pedersen (pedersen-larserik)]

I am also having this issue.
netplan.io: 0.97-0ubuntu1~18.04.1

[Lars Erik Pedersen (pedersen-larserik)]

My config:
network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      match:
        macaddress: <removed>
      dhcp4: true
      dhcp4-overrides:
        use-routes: false
      routes:
        - to: 10.212.132.0/24
          table: 1
          scope: link
        - to: 0.0.0.0/0
          via: 10.212.132.1
          table: 1
      routing-policy:
        - to: 0.0.0.0/0
          from: 10.212.132.0/24
          table: 1
    ens9:
      match:
        macaddress: <removed>
      dhcp4: true
      dhcp4-overrides:
        use-routes: false
      routes:
        - to: 10.212.134.0/25
          table: 2
          scope: link
        - to: 0.0.0.0/0
          via: 10.212.134.1
          table: 2
      routing-policy:
        - to: 0.0.0.0/0
          from: 10.212.134.0/25
          table: 2

The two routing tables will not have the static routes added.

[Mathieu Trudel-Lapierre (cyphermox)]

Since netplan only does writing configuration to be consumed by the backends like systemd, this would actually be a systemd bug; reassigning.

I thought that worked though, in some setups, especially with use-routes: false as it was being done in the config above.

Nevertheless, it needs investigation. I expect we could see the routes are being installed, then ripped out after systemd-networkd gets an address from DHCP.

[Lars Erik Pedersen (pedersen-larserik)]

Hi. I did som further investigation here, and I think I've figured out what happens in networkd here.

Aug 21 10:54:45 nettest1 systemd-networkd[5463]: ens3: Removing address: 2001:700:1d00:ec00:5054:ff:fe49:4bc9/64 (valid for 4w 2d)
Aug 21 10:54:46 nettest1 systemd-networkd[5463]: ens3: Removing address: 10.212.128.84/24 (valid for 11h 57min 54s)
Aug 21 10:54:46 nettest1 systemd-networkd[5463]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_32 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=17 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Aug 21 10:54:46 nettest1 systemd-networkd[5463]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=18 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Aug 21 10:54:46 nettest1 systemd-networkd[5463]: ens3: Routing policy rule configured
Aug 21 10:54:46 nettest1 systemd-networkd[5463]: ens3: Could not set route: Network is unreachable
Aug 21 10:54:46 nettest1 systemd-networkd[5463]: ens3: Routes set
Aug 21 10:54:46 nettest1 systemd-networkd[5463]: ens3: Adding address: 10.212.128.84/24 (valid for 12h)
Aug 21 10:54:46 nettest1 systemd-networkd[5463]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_32 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=19 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Aug 21 10:54:46 nettest1 systemd-networkd[5463]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=20 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Aug 21 10:54:46 nettest1 systemd-networkd[5463]: ens3: Configured

Basically it seems like networkd does the following on "netplan apply":
1) Remove every address
2) Add routes - which of course will fail with "network is unreachable" because the interfaces don't have an address yet
3) Adding the addresses

[Dan Streetman (ddstreet)]

@pedersen-larserik, @zagarin, can either of you could test your netplan config on 19.10 (Eoan) to verify things work for you there?

[Lars Erik Pedersen (pedersen-larserik)]

@ddstreet It seems to work as expected in Eoan!
netplan.io 0.98-0ubuntu1

[Dan Streetman (ddstreet)]

> @ddstreet It seems to work as expected in Eoan!

excellent! now I need to figure out (again) which upstream commit fixes this, as I know I found it before but apparently forgot to include the info in this bug...

[Dan Streetman (ddstreet)]

as disco is EOL next week, marking this won't fix for disco.

[Steve Langasek (vorlon)]

Hello Alexey, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.34 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (237-3ubuntu10.34) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

php7.2/7.2.24-0ubuntu0.18.04.2 (armhf)
openssh/1:7.6p1-4ubuntu0.3 (arm64, armhf, ppc64el, amd64, s390x, i386)
dovecot/1:2.2.33.2-1ubuntu4.5 (armhf)
gvfs/1.36.1-0ubuntu1.3.3 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Steve Langasek (vorlon)]

Hello Alexey, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.39 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (237-3ubuntu10.39) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

php7.2/7.2.24-0ubuntu0.18.04.2 (armhf)
gvfs/1.36.1-0ubuntu1.3.3 (ppc64el)
lxc/3.0.3-0ubuntu1~18.04.1 (amd64)
systemd/237-3ubuntu10.39 (i386)
netplan.io/0.98-0ubuntu1~18.04.1 (i386, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

verified on bionic

ubuntu@lp1836695-b:~$ dpkg -l systemd|grep ii
ii systemd 237-3ubuntu10.39 amd64 system and service manager
ubuntu@lp1836695-b:~$ ip r
10.0.0.0/8 via 192.168.122.1 dev ens3 proto static
192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.35

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 237-3ubuntu10.39

---------------
systemd (237-3ubuntu10.39) bionic; urgency=medium

  [ Dariusz Gadomski ]
  * d/p/lp1762391/0001-Call-getgroups-to-know-size-of-supplementary-groups-.patch,
    d/p/lp1762391/0002-user-util-tweak-to-in_gid.patch,
    d/p/lp1762391/0003-user-util-Add-helper-functions-for-gid-lists-operati.patch,
    d/p/lp1762391/0004-execute-Restore-call-to-pam_setcred.patch,
    d/p/lp1762391/0005-execute-Detect-groups-added-by-PAM-and-merge-them-wi.patch,
    d/p/lp1762391/0006-test-Add-tests-for-gid-list-ops.patch,
    d/p/lp1762391/0007-execute-add-const-to-array-parameters-where-possible.patch,
    d/p/lp1762391/0008-execute-allow-pam_setcred-to-fail-ignore-errors.patch:
    - Restore call to pam_setcred (LP: #1762391)

  [ Ioanna Alifieraki ]
  * d/p/lp1860548/0001-Revert-Replace-use-of-snprintf-with-xsprintf.patch,
    d/p/lp1860548/0002-job-truncate-unit-description.patch:
    - use snprintf instead of xsprintf (LP: #1860548)

  [ Dan Streetman ]
  * d/p/lp1833193-network-update-address-when-static-address-was-alrea.patch:
    - Update lft when static addr was cfg by dhcp (LP: #1833193)
  * d/p/lp1849261/0001-core-when-we-can-t-enqueue-OnFailure-job-show-full-e.patch,
    d/p/lp1849261/0002-core-don-t-trigger-OnFailure-deps-when-a-unit-is-goi.patch:
    - Only trigger OnFailure= if Restart= is not in effect (LP: #1849261)
  * d/p/lp1671951-network-set-ipv6-mtu-after-link-up-or-device-mtu-cha.patch:
    - set ipv6 mtu at correct time (LP: #1671951)
  * d/p/lp1845909/0001-networkd-honour-LinkLocalAddressing.patch,
    d/p/lp1845909/0002-networkd-fix-link_up-12505.patch,
    d/p/lp1845909/0003-network-do-not-send-ipv6-token-to-kernel.patch,
    d/p/lp1845909/0004-network-rename-linux_configure_after_setting_mtu-to-linux.patch,
    d/p/lp1845909/0005-network-add-link-setting_genmode-flag.patch,
    d/p/lp1845909/0006-network-if-ipv6ll-is-disabled-enumerate-tentative-ipv6-ad.patch,
    d/p/lp1845909/0007-network-drop-foreign-config-after-addr_gen_mode-has-been-.patch,
    d/p/lp1845909/0008-network-drop-IPv6LL-address-when-LinkLocalAddressing.patch:
    - if LinkLocalAddressing=no prevent creation of ipv6ll (LP: #1845909)
  * d/p/lp1859862-network-Do-not-disable-IPv6-by-writing-to-sysctl.patch:
    - enable ipv6 when needed (LP: #1859862)
  * d/p/lp1836695-networkd-Add-back-static-routes-after-DHCPv4-lease-e.patch:
    - (re)add static routes after getting dhcp4 addr (LP: #1836695)
  * d/t/storage:
    - fix buggy test (LP: #1831459)
    - without scsi_debug, skip test (LP: #1847816)

 -- Dan Streetman <email address hidden> Thu, 06 Feb 2020 10:00:49 -0500