systemd-networkd fails to apply static IPv4 when the static IP is the same as previously configured by DHCP

Bug #1833193 reported by Lars Erik Pedersen on 2019-06-18
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd
Fix Released
Unknown
systemd (Ubuntu)
Medium
Dan Streetman
Bionic
Medium
Dan Streetman
Disco
Medium
Dan Streetman

Bug Description

[impact]

if networkd is configured with a static address, and the address has already been assigned by dhcp (so that it has finite valid_lft), networkd will not change the valid_lft to 'forever' during startup, which leaves networkd thinking the address is static (meaning, networkd will not attempt to renew the dhcp lease) but the address valid_lft counting down.

Once the valid_lft expires, the kernel will remove the address from the interface, breaking the system's networking.

[test case]

configure a system with networkd similar to:

[Match]
Name=ens3

[Network]
DHCP=ipv4

restart systemd-networkd, and ens3 should get a DHCP address, e.g.:

$ ip -4 a show ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.122.147/24 brd 192.168.122.255 scope global dynamic ens3
       valid_lft 2295sec preferred_lft 2295sec

then, edit the networkd config to use a static address, e.g.:

[Match]
Name=ens3

[Network]
Address=192.168.122.147/24
Gateway=192.168.122.1
DNS=192.168.122.1
Domains=vm

make sure to use the exact same address that was provided by dhcp.

Then, without rebooting, just restart networkd and check the address lft:

$ sudo systemctl restart systemd-networkd

$ ip -4 a show ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.122.147/24 brd 192.168.122.255 scope global dynamic ens3
       valid_lft 2992sec preferred_lft 2992sec

the address should have 'forever' valid_lft, but it doesn't.

[regression potential]

this changes the details of how networkd updates static addresses during startup, so any regression would occur at networkd startup when using static addresses.

[scope]

this is needed only in Bionic and Disco.

this is fixed by upstream:
https://github.com/systemd/systemd/pull/11460/commits
specifically commit a47a6daebe0af26c07d8d423225eb4add110793c
which is included in Eoan and later.

In Xenial using networkd, testing shows this problem does not occur, because this older version of networkd removes the 'foreign' dhcp address before re-assigning it as a static address.

[other info]

original description:
---

In bionic, running systemd 237-3ubuntu10.22 and netplan.io 0.97-0ubuntu1~18.04.1, systemd-networkd fails to configure an interface with a static IPv4 address if the statically confiugred address is the same as the interface already has gotten from DHCP.

This will cause the interface to loose its IP address when the DHCP lease exires, even though you've told netplan to configure it as static.

I expect systemd-networkd to actually configure an IP address as static, regardless of what address the interface has before from DHCP.

# lsb_release -rd
Description: Ubuntu 18.04.2 LTS
Release: 18.04

# apt-cache policy systemd
systemd:
  Installed: 237-3ubuntu10.22

# apt-cache policy netplan.io
netplan.io:
  Installed: 0.97-0ubuntu1~18.04.1

A paste of systemd-networkd's debug log when I run "netplan apply" and the interface already has the static IP configured from DHCP.

It seems like upon a restart, systemd-networkd will allways add whatever IP config it had before the service stopped, and then apply changes (if any). Since my new config has the same IP as it already had, it does nothing even though the new config has static configuration.

summary: - systemd-network fails to apply static IPv4 when the static IP is the
+ systemd-networkd fails to apply static IPv4 when the static IP is the
same as previously configured by DHCP

I should add that if I set "dhcp4: no" explicit when confiugring static addressing, it behaves as expected, but according to the current documentation, this parameter should default to "no", so this should not be neccescary.

Dan Streetman (ddstreet) wrote :

Can you paste an example networkd config file, and repro instructions, that reproduces this problem?

Repro instructions:
- (Enable systemd-networkd debug logging)
- Start a server with its interface configured with DHCP
- In the netplan config, configure the interface wtih static addressing, using the exact same address as it already has gotten from DHCP
- Either wait for the initial DHCP lease time to expire, and observe that the interface loses its address; OR observe in the debug log that networkd applied the address with a lease time, even though you told netplan to use static addressing

Networkd config with static addressing:
cat /run/systemd/network/10-netplan-ens3.network
[Match]
MACAddress=<redacted>

[Network]
LinkLocalAddressing=ipv6
Address=10.212.132.107/24
Gateway=10.212.132.1
DNS=8.8.8.8
DNS=8.8.4.4
Domains=<redacted>

Networkd config with DHCP:
cat /run/systemd/network/10-netplan-ens3.network
[Match]
MACAddress=<redacted>
[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6

[DHCP]
RouteMetric=100
UseMTU=true

Debug log output from systemd-networkd when netplan tells it to apply static addressing when the interface already has an address from DHCP:
Sep 30 10:36:15 nettest1 systemd[1]: Stopped Network Service.
Sep 30 10:36:16 nettest1 systemd[1]: Starting Network Service...
<snip>
Sep 30 10:36:16 nettest1 systemd-networkd[6086]: ens3: Adding address: 10.212.132.107/24 (valid for 11h 52min 55s)

Dan Streetman (ddstreet) wrote :

Thanks! that clarifies it - I think this is a bug upstream with systemd.

> if I set "dhcp4: no" explicit when confiugring static addressing, it behaves as expected

are you sure? that isn't what I would expect, it seems the problem is networkd doesn't adjust the address lifetime when networkd is restarted and the address changes from dhcp->static; i would not expect the DHCP=no param it fix that. Although that's a netplan config, so possibly netplan does something additional under the covers, other than what systemd-networkd does.

> It seems like upon a restart, systemd-networkd will allways add whatever IP config it had before the service stopped, and then apply changes

not exactly, what's happening is when you stop networkd, it does not remove network configuration that it manages, so in this case it would leave the dhcp address on the interface. then when you (re)start networkd, it does not change existing addresses that it manages, so it sees that the interface already has the address that it's been configured to use as a static address, and does not change it. It looks like networkd needs to also check the address's lifetime to make sure static addrs have their lifetime set to 'forever' (I have not actually checked the code yet to verify this).

If you reboot, or manually remove the address, or just bring the interface down and back up, networkd will re-add the address with 'forever' lft, so this should only be a problem when you have manually changed the config for an address from dhcp->static using the same ip address, and then restart networkd without rebooting, at least IIUC.

Changed in systemd:
status: Unknown → Fix Released

Yes that's the same issue. And it seems to be fixed in some version as well? I just tested it on Eoan, using systemd 242-6, and the valid__lft and preferred_lft did indeed change to 'forever'. Don't know which version it was fixed in though. Do you think it kan be backported to Bionic?

The discussion in the GitHub issue was rather interesting as well. I kind of "agree" that this is a strange thing to do, but I don't think that should be any of systemd's business..

Dan Streetman (ddstreet) wrote :

> I just tested it on Eoan, using systemd 242-6, and the valid__lft and preferred_lft did indeed change to 'forever'.

well that's interesting...I tested with the latest upstream systemd code and i *was* able to reproduce the problem...maybe something reverted the fix since 242. Unless, of course, I messed up my test :-) I'll backport the fix for that and see if it fixes bionic, then retest the latest upstream to make sure it hasn't reverted.

> The discussion in the GitHub issue was rather interesting as well. I kind of "agree" that this is a strange thing to do, but I don't think that should be any of systemd's business..

yes, I agree, and it is an entertaining discussion on what systemd should prevent users from doing ;)

Dan Streetman (ddstreet) wrote :

Sorry for the delay; I need to re-test this again with upstream code to verify it does reproduce there (or not) before working to backport any fix.

Changed in systemd (Ubuntu):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Medium
status: New → In Progress
Dan Streetman (ddstreet) on 2019-12-12
tags: added: bionic ddstreet disco eoan focal sts systemd
Dan Streetman (ddstreet) wrote :

Update: I retested today, and I must have made a mistake in my previous upstream tests; it does appear fixed with upstream systemd, as well as being fixed in Ubuntu E/F.

Dan Streetman (ddstreet) on 2019-12-12
Changed in systemd (Ubuntu Bionic):
status: New → In Progress
Changed in systemd (Ubuntu Disco):
status: New → In Progress
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Disco):
importance: Undecided → Medium
Changed in systemd (Ubuntu Bionic):
importance: Undecided → Medium
Changed in systemd (Ubuntu):
status: In Progress → Fix Released
Dan Streetman (ddstreet) on 2019-12-12
description: updated
description: updated
Dan Streetman (ddstreet) on 2019-12-12
tags: removed: eoan focal
Dan Streetman (ddstreet) wrote :

as disco is EOL next week, marking this won't fix for disco.

Changed in systemd (Ubuntu Disco):
status: In Progress → Won't Fix

Hello Lars, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.34 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic

All autopkgtests for the newly accepted systemd (237-3ubuntu10.34) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

php7.2/7.2.24-0ubuntu0.18.04.2 (armhf)
openssh/1:7.6p1-4ubuntu0.3 (arm64, armhf, ppc64el, amd64, s390x, i386)
dovecot/1:2.2.33.2-1ubuntu4.5 (armhf)
gvfs/1.36.1-0ubuntu1.3.3 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hi!

Tested with systemd=237-3ubuntu10.34 from proposed (and libsystemd0=237-3ubuntu10.34) and it seems to fix the problem.

What's interesting though, is that a fresh install of bionic came with a newer version (systemd=237-3ubuntu10.38) - which has the bug. This test effectively ended up in a downgrade.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Dan Streetman (ddstreet) wrote :

> What's interesting though, is that a fresh install of bionic came with a newer version (systemd=237-3ubuntu10.38) - which has the bug. This test effectively ended up in a downgrade.

yes, a security release for systemd overrode the version with this fix in -proposed, so I have to rebase these patches and re-upload. These things sometimes happen, unfortunately.

Dimitri John Ledkov (xnox) wrote :

This SRU needs to be reuploaded, due to security update that trumped this in progress SRU.

Changed in systemd (Ubuntu Bionic):
status: Fix Committed → In Progress
tags: added: verification-failed verification-failed-bionic
removed: verification-done-bionic verification-needed

Hello Lars, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.39 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-failed verification-failed-bionic

All autopkgtests for the newly accepted systemd (237-3ubuntu10.39) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

php7.2/7.2.24-0ubuntu0.18.04.2 (armhf)
gvfs/1.36.1-0ubuntu1.3.3 (ppc64el)
lxc/3.0.3-0ubuntu1~18.04.1 (amd64)
systemd/237-3ubuntu10.39 (i386)
netplan.io/0.98-0ubuntu1~18.04.1 (i386, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hi!

Re-tested with systemd=237-3ubuntu10.39 from proposed (and libsystemd0=237-3ubuntu10.39), and it works.

tags: added: verification-done-bionic
removed: verification-needed-bionic

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 237-3ubuntu10.39

---------------
systemd (237-3ubuntu10.39) bionic; urgency=medium

  [ Dariusz Gadomski ]
  * d/p/lp1762391/0001-Call-getgroups-to-know-size-of-supplementary-groups-.patch,
    d/p/lp1762391/0002-user-util-tweak-to-in_gid.patch,
    d/p/lp1762391/0003-user-util-Add-helper-functions-for-gid-lists-operati.patch,
    d/p/lp1762391/0004-execute-Restore-call-to-pam_setcred.patch,
    d/p/lp1762391/0005-execute-Detect-groups-added-by-PAM-and-merge-them-wi.patch,
    d/p/lp1762391/0006-test-Add-tests-for-gid-list-ops.patch,
    d/p/lp1762391/0007-execute-add-const-to-array-parameters-where-possible.patch,
    d/p/lp1762391/0008-execute-allow-pam_setcred-to-fail-ignore-errors.patch:
    - Restore call to pam_setcred (LP: #1762391)

  [ Ioanna Alifieraki ]
  * d/p/lp1860548/0001-Revert-Replace-use-of-snprintf-with-xsprintf.patch,
    d/p/lp1860548/0002-job-truncate-unit-description.patch:
    - use snprintf instead of xsprintf (LP: #1860548)

  [ Dan Streetman ]
  * d/p/lp1833193-network-update-address-when-static-address-was-alrea.patch:
    - Update lft when static addr was cfg by dhcp (LP: #1833193)
  * d/p/lp1849261/0001-core-when-we-can-t-enqueue-OnFailure-job-show-full-e.patch,
    d/p/lp1849261/0002-core-don-t-trigger-OnFailure-deps-when-a-unit-is-goi.patch:
    - Only trigger OnFailure= if Restart= is not in effect (LP: #1849261)
  * d/p/lp1671951-network-set-ipv6-mtu-after-link-up-or-device-mtu-cha.patch:
    - set ipv6 mtu at correct time (LP: #1671951)
  * d/p/lp1845909/0001-networkd-honour-LinkLocalAddressing.patch,
    d/p/lp1845909/0002-networkd-fix-link_up-12505.patch,
    d/p/lp1845909/0003-network-do-not-send-ipv6-token-to-kernel.patch,
    d/p/lp1845909/0004-network-rename-linux_configure_after_setting_mtu-to-linux.patch,
    d/p/lp1845909/0005-network-add-link-setting_genmode-flag.patch,
    d/p/lp1845909/0006-network-if-ipv6ll-is-disabled-enumerate-tentative-ipv6-ad.patch,
    d/p/lp1845909/0007-network-drop-foreign-config-after-addr_gen_mode-has-been-.patch,
    d/p/lp1845909/0008-network-drop-IPv6LL-address-when-LinkLocalAddressing.patch:
    - if LinkLocalAddressing=no prevent creation of ipv6ll (LP: #1845909)
  * d/p/lp1859862-network-Do-not-disable-IPv6-by-writing-to-sysctl.patch:
    - enable ipv6 when needed (LP: #1859862)
  * d/p/lp1836695-networkd-Add-back-static-routes-after-DHCPv4-lease-e.patch:
    - (re)add static routes after getting dhcp4 addr (LP: #1836695)
  * d/t/storage:
    - fix buggy test (LP: #1831459)
    - without scsi_debug, skip test (LP: #1847816)

 -- Dan Streetman <email address hidden> Thu, 06 Feb 2020 10:00:49 -0500

Changed in systemd (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.