127.0.0.1/::1 removed from loopback interface if you configure extra IPs on lo (r104 regression/behaviour change)

Bug #1979951 reported by Trent Lloyd
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
netplan.io (Ubuntu)
Confirmed
High
Unassigned
Focal
New
Undecided
Unassigned
Jammy
New
Undecided
Unassigned
systemd (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned

Bug Description

[Impact]

Users who wish to configure additional addresses on lo may inadvertently remove the 127.0.0.1/8 address from the interface. And, because this only happens on the second invocation of `netplan apply`, it can create confusing breakages for users. Although this is not a recommended configuration, it was previously supported and the current behavior is a regression.

[Test Plan]

* Configure additional addresses on the lo interface. I am testing in a LXD container, so I added the following to /etc/netplan/10-lxc.yaml under `ethernets:`

    lo:
      addresses: ["10.10.10.17/24","10.10.10.19/24"]

* Apply the config, and then inspect the lo interface's addresses:

 $ netplan apply
 $ ip addr show dev lo

* Observe that the 127.0.0.1/8 address is still present. Now, run the same commands again:

 $ netplan apply
 $ ip addr show dev lo

* On an affected system, observe that the 127.0.0.1/8 address is now gone. On a patched system, the address should still be present.

[Where problems could occur]

The patch removes cases where systemd-networkd will drop foreign addresses from a managed link. If problems were to occur, it would be related to the addresses configured on a managed link.

[Original Description]

If you configure the loopback interface using netplan without listing the standard loopback addresses (127.0.0.1 / ::1) that are auto-configured on boot then they are removed by netplan in r104+ but were not in r103.

Confusingly this only happens the second time "netplan apply" is invoked and not the first time.

There is some argument this is not a valid configuration, however
- It worked on r103 and broke in the r104 SRU for 20.04 and breaks on upgrade
- Since it only occurs on the second "netplan apply" it is likely people will accidentally create a configuration that on first verification works but breaks later
- Adding addresses to a loopback or dummy interface is a common need
- There is no dummy interface support in Netplan
- Removing the loopback IP doesn't immediately cause obvious breakage in an SSH session but causes all sorts of hard to diagnose issues with applications

Thus I consider this a high priority regression because of the multiple ways this can break a system at an unexpected/unrelated time to when the original change was made.

As a workaround you can add 127.0.0.1/::1 to the file and this seems to generally work and product almost the same configuration with the exception that "brd 127.255.255.255" is added to the "ip addr" output compared to the auto-created configuration. All of the different routing tables seem otherwise the tame.

= Test Case =
(1) Add IP configuration for lo to /etc/netplan/lo.yaml

lo:
  match:
    name: lo
  addresses:
  - 10.10.10.17/24
  - 10.10.10.19/24

(2) Run "netplan apply" and observe the "ip addr show dev lo" output. Expected: 127.0.0.1/::1 are still there.
(3) Run "netplan apply" a second time and observe the "ip addr show dev lo" output. Expected: 127.0.0.1/::1 are now removed.
(4) Repeat the same test under netplan r103. Expected: 127.0.0.1/::1 are still there in both cases.

= Observations =

I found this change of behaviour happens only in r104. It is suspected but not clear that this is due to the configuraiton diffing behaviour introduced in r104. It's not clear to me why it doesn't happen on the first "netplan apply".

I also found a recent upstream systemd commit to prevent networkd removing 127.0.0.1/::1 which solves this issue on r104 as well. It's possible this is a better fix to backport to solve this:
https://github.com/systemd/systemd/commit/b69bfa43052586cd6894c32d7d3d447195539ec5

Reverting to r103 *or* running a newer systemd with that patch resolves the issue on 20.04 and 22.04 in my testing.

Related branches

Revision history for this message
Lukas Märdian (slyon) wrote :

This is probably a side-effect of netplan switching over to using "networkctl reload/reconfigure" in 0.104 (it was introduced in 0.103 already, but reverted for various reasons. Re-appeared in 0.104). Should probably be fixed in systemd-networkd.

tags: added: rls-ff-incoming rls-jj-incoming
tags: added: fr-2514
Revision history for this message
Lukas Märdian (slyon) wrote :

That systemd commit is included in systemd v251, which landed in Kinetic today.

tags: removed: rls-ff-incoming rls-jj-incoming
Changed in systemd (Ubuntu):
status: New → Fix Released
Revision history for this message
Trent Lloyd (lathiat) wrote :

OK thanks for the note.

Lukas: For systemd do you want me to prepare an SRU debdiff or do you have other SRUs inflight and could just pluck this for backport to 18.04/20.04 directly?

It's a pretty tiny patch (https://github.com/systemd/systemd/commit/b69bfa43052586cd6894c32d7d3d447195539ec5)

Revision history for this message
Lukas Märdian (slyon) wrote :

Yes, feel free to create a SRU for this, updating the bug description according to https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template (adding test-case/impact/description).

We have systemd SRUs in progress right now, but those are already released to -proposed. So we can only stage this into git and have it released with the next SRU upload.

Please create git Merge Proposals on Launchpad against those branches, instead of creating a debdiff (and link them to this bug):
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/log/?h=ubuntu-jammy
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/log/?h=ubuntu-focal

Nick Rosbrook (enr0n)
description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Trent, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Jammy):
status: New → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Trent, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/249.11-0ubuntu3.5)

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.5) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.48.2-0ubuntu1 (ppc64el)
systemd/249.11-0ubuntu3.5 (ppc64el)
udisks2/2.9.4-1ubuntu2 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/245.4-4ubuntu3.18)

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.18) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-gke-5.15/5.15.0-1015.18~20.04.1 (arm64, amd64)
netplan.io/0.104-0ubuntu2~20.04.2 (armhf)
linux-hwe-5.15/5.15.0-48.54~20.04.1 (armhf, amd64)
gvfs/1.44.1-1ubuntu1.1 (ppc64el)
linux-oracle-5.15/5.15.0-1016.20~20.04.1 (arm64)
linux-oracle-5.13/5.13.0-1036.43~20.04.1 (arm64)
linux-hwe-5.11/5.11.0-61.61 (arm64)
systemd/245.4-4ubuntu3.18 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Nick Rosbrook (enr0n) wrote :

On focal, the autopkgtest regressions were all resolved with retries. I verified this fix for focal using LXD container:

root@focal:~# apt-cache policy systemd
systemd:
  Installed: 245.4-4ubuntu3.18
  Candidate: 245.4-4ubuntu3.18
  Version table:
 *** 245.4-4ubuntu3.18 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     245.4-4ubuntu3.17 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     245.4-4ubuntu3.15 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     245.4-4ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
root@focal:~# cat /etc/netplan/10-lxc.yaml
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
      dhcp-identifier: mac
    lo:
      addresses: ["10.10.10.17/24","10.10.10.19/24"]
root@focal:~# netplan apply
root@focal:~# ip addr show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.10.10.17/24 brd 10.10.10.255 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.10.10.19/24 brd 10.10.10.255 scope global secondary lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
root@focal:~# netplan apply
root@focal:~# ip addr show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.10.10.17/24 brd 10.10.10.255 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.10.10.19/24 brd 10.10.10.255 scope global secondary lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

tags: added: verification-done-focal
removed: verification-needed-focal
tags: added: foundations-todo
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Trent, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Nick Rosbrook (enr0n) wrote :

I have verified the fix on jammy using systemd 249.11-0ubuntu3.6 from jammy-proposed:

root@jammy:~# apt-cache policy systemd
systemd:
  Installed: 249.11-0ubuntu3.6
  Candidate: 249.11-0ubuntu3.6
  Version table:
 *** 249.11-0ubuntu3.6 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.4 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
root@jammy:~# cat /etc/netplan/10-lxc.yaml
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
      dhcp-identifier: mac
    lo:
      addresses: ["10.10.10.17/24","10.10.10.19/24"]
root@jammy:~# netplan apply
root@jammy:~# ip addr show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.10.10.17/24 brd 10.10.10.255 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.10.10.19/24 brd 10.10.10.255 scope global secondary lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
root@jammy:~# netplan apply
root@jammy:~# ip addr show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.10.10.17/24 brd 10.10.10.255 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.10.10.19/24 brd 10.10.10.255 scope global secondary lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

tags: added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/249.11-0ubuntu3.6)

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.6) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

stunnel4/3:5.63-1build1 (amd64)
munin/2.0.57-1ubuntu2 (amd64)
corosync/unknown (s390x)
conntrack-tools/unknown (s390x)
exim4/4.95-4ubuntu2.1 (ppc64el)
umockdev/0.17.7-1 (armhf)
netplan.io/0.104-0ubuntu2.1 (amd64)
initramfs-tools/0.140ubuntu13 (amd64)
dovecot/unknown (s390x)
network-manager/1.36.6-0ubuntu2 (amd64)
cups/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Nick Rosbrook (enr0n) wrote :

The autopkgtest regressions for systemd 249.11-0ubuntu3.6 in jammy-proposed were all resolved with retries.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.9 KiB)

This bug was fixed in the package systemd - 249.11-0ubuntu3.6

---------------
systemd (249.11-0ubuntu3.6) jammy; urgency=medium

  * Deny-list TEST-58-REPART on ppc64el (LP: #1988994)
    File: debian/patches/lp1988994-Deny-list-TEST-58-REPART-on-ppc64el.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d2ed3cc1d223bf35015b15ff83b50156b58f0f38

systemd (249.11-0ubuntu3.5) jammy; urgency=medium

  [ Nick Rosbrook ]
  * Ensure dns_search_domain_unlink_marked removes all marked domains (LP: #1975667)
    File: debian/patches/lp1975667-Ensure-dns_search_domain_unlink_marked-removes-all-marked.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=919d5ddedd5bb8b45ab9437bf42d66c2821bb074
  * core,firstboot: workaround timezone issues on Ubuntu Core (LP: #1981042)
    Thanks to Robert Ancell for preparing the patch.
    File: debian/patches/lp1981042-core-firstboot-workaround-timezone-issues-caused-by-Ubunt.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b15546361b549217908fb6ca5d473be23d7fa757
  * network: do not remove localhost address (LP: #1979951)
    File: debian/patches/lp1979951-network-do-not-remove-localhost-address.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=2cd88391cce9fe95a486ae6dd214c12f236f3881
  * units: remove the restart limit on the modprobe@.service (LP: #1982462)
    File: debian/patches/lp1982462-units-remove-the-restart-limit-on-the-modprobe-.service.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8f0acd1b2fbb8eed1259c34963e5e9b201bef900
  * pstore: do not try to load mtdpstore (LP: #1981622)
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=15225032c3657f5906ee49d48929f9295a8664a0
  * core/mount: downgrade log level about several mkdir failures (LP: #1979952)
    Files:
    - debian/patches/lp1979952-Revert-core-mount-fail-early-if-directory-cannot-be-creat.patch
    - debian/patches/lp1979952-core-mount-downgrade-log-level-about-several-mkdir-failur.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ee8cfcf500698fab2e990de291ecf4c3ab87a4ae
  * debian/control: add Recommends: systemd-hwe-hwdb to udev.
    The systemd-hwe-hwdb brings in additional hwdb rules for HWE, so we want
    those installed with udev by default.
    File: debian/control
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4a7a3258c33201cca305956820fcc6bcd6052d76
  * hwdb: implement --root option for systemd-hwdb query (LP: #1988078)
    Files:
    - debian/libsystemd0.symbols
    - debian/patches/lp1988078-hwdb-implement-root-option-for-systemd-hwdb-query.patch
    - debian/patches/lp1988078-sd-hwdb-add-sd_hwdb_new_from_path.patch
    - debian/patches/lp1988078-sd-hwdb-include-sys-stat.h-in-hwdb-internal.h.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=937fef96c858f2f2042bf71032f315647c14add0

  [ Luca Boccassi ]
  * Enable systemd-repart and ship it in a new s...

Read more...

Changed in systemd (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 245.4-4ubuntu3.18

---------------
systemd (245.4-4ubuntu3.18) focal; urgency=medium

  [ Nick Rosbrook ]
  * core: make sure we don't get confused when setting TERM for a tty fd
    (LP: #1959475)
    File: debian/patches/lp1959475-core-make-sure-we-don-t-get-confused-when-setting-TERM-fo.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b10c6853050dde26665caf3b15444d768d2bc498
  * shared/calendarspec: when mktime() moves us backwards, jump forward
    (LP: #1966800)
    File: debian/patches/lp1966800-shared-calendarspec-when-mktime-moves-us-backwards-jump-f.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1f063541e44f6ff1a6904676d4264a2e49a09594
  * network: do not remove localhost address (LP: #1979951)
    File: debian/patches/lp1979951-network-do-not-remove-localhost-address.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=87f872b8c5451f353601fb606e7fd7a479217cef
  * units: remove the restart limit on the modprobe@.service (LP: #1982462)
    File: debian/patches/lp1982462-units-remove-the-restart-limit-on-the-modprobe-.service.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=68353ffaf3539e6a58ef62a8b50850f56eae29ea

  [ Mustafa Kemal Gilor ]
  * d/p/lp1978079-efi-pstore-not-cleared-on-boot.patch: pstore: Run after
    modules are loaded. Thanks to Alexander Graf <email address hidden>.
    (LP: #1978079)
    Author: Mustafa Kemal Gilor
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6e60756f2079d6408abdb967127a1d9b9a0eba8c

 -- Nick Rosbrook <email address hidden> Wed, 31 Aug 2022 11:27:33 -0400

Changed in systemd (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers