NetworkManager 1.36.6 no longer prefers DHCPv6 addresses over SLAAC

[Kevin]

* Impact
The recent SRU created a regression in IPv6 routing order

* Test Case

1. Connect to a network where the router sends "A" and "M" bits in the RA's and has a DHCPv6 server running (e.g. any OpenWrt router).

2. When running `ip -6 a`, the list now sorts SLAAC addresses above DHCPv6 addresses. With NetworkManager 1.36.4 and earlier, this was not the case. (The Linux kernel uses the address highest in the list as preferred.)

3. When running something like `curl ifconfig.co`, the SLAAC address is being returned, which makes sense as that is now preferred by the kernel. (But it shouldn't be.)

Desired behaviour:

NetworkManager should always sort DHCPv6 addresses above SLAAC addresses, as is the case for all versions prior to 1.36.6 and also corrected again in 1.38.0. In case static addresses are manually set, those should take first priority, with DHCPv6 second and SLAAC/autoconf last.

* Regression potential

The patch change the IPv6 addresses order to match the behaviour from the previous version. If that was buggy the routing order would still be wrong, so we should check that IPv6 setups behave as expected.

---------------------------------------

Situation:

My network has both DHCPv6 and SLAAC (autoconf) for IPv6. From a privacy perspective, for readability reasons and for network management policies, DHCPv6 should *always* be preferred over SLAAC addresses when available. And according to RFC 6724, the smaller /128 scope of the DHCPv6 address should be chosen over the larger /64 scope of the SLAAC address.

NetworkManager has always been able to adhere to that by simply setting ip6.privacy=0 for the connection (in nm-connection-editor *not* selecting "Prefer temporary address" for IPv6 privacy extensions). Then it would use the DHCPv6 address as the source for all outgoing traffic.

So if you would - for instance - run `curl ifconfig.co`, the DHCPv6 address would be used to connect to the outside world and be echoed back.

Regression:

Since the update to 1.36.6, this is no longer the case. NetworkManager now routes outgoing traffic through the SLAAC address, even if ip6.privacy=0 is set for the connection.

Constantly removing the SLAAC addresses with `ip addr del` or disabling SLAAC RA's on the router are now the only ways to stop NetworkManager from preferring SLAAC over DHCPv6. None of the local options in NetworkManager 1.36.6 are able to restore the previous, desired and correct way of working: the SLAAC address should never be used as the preferred address if a DHCPv6 lease is given.

Looking at the changelog of NetworkManager 1.36.6, multiple things regarding IP address order and temporary addresses have been changed in that release, any of them (or a combination) introducing this bug:

* Fix a bug in synchronization of IP addresses with kernel that could lead to a wrong address order.
* Ignore addresses from DHCPv6 when the Otherconf router advertisement flag is set.
* Ensure temporary IPv6 addresses are removed on disconnect and reapply.

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/nm-1-36/NEWS

Steps to reproduce:

1. Connect to a network where the router sends "A" and "M" bits in the RA's and has a DHCPv6 server running (e.g. any OpenWrt router).

2. When running `ip -6 a`, the list now sorts SLAAC addresses above DHCPv6 addresses. With NetworkManager 1.36.4 and earlier, this was not the case. (The Linux kernel uses the address highest in the list as preferred.)

3. When running something like `curl ifconfig.co`, the SLAAC address is being returned, which makes sense as that is now preferred by the kernel. (But it shouldn't be.)

Desired behaviour:

NetworkManager should always sort DHCPv6 addresses above SLAAC addresses, as is the case for all versions prior to 1.36.6 and also corrected again in 1.38.0. In case static addresses are manually set, those should take first priority, with DHCPv6 second and SLAAC/autoconf last.

Implications:

This can break many real-life use cases. For instance, my router gives out static leases to my machines. Those addresses are whitelisted in all kinds of firewalls to allow me to access servers for my work. Now that the "wrong" address is being preferred for outgoing traffic (a SLAAC address that I have no influence on and cannot centrally configure), I am being locked out of the servers in question unless I forcefully remove the addresses or disable SLAAC on my router, so my outgoing traffic is being routed through the DHCPv6 address again.

Note that "just disabling SLAAC RA's on the router" is not something everybody can do, as it requires root access to the device. Moreover, it would break IPv6 connectivity entirely for devices that don't support DHCPv6 (read: Android).

Conclusion:

So this update introduces a very breaking change in IPv6 source address selection to an LTS release, while LTS releases should be stable.

I should note that the bug is not present in NetworkManager 1.38.0 on Debian sid. That just prefers DHCPv6 addresses when available, like it should. As that version is also used in Ubuntu kinetic, most likely this bug is not present there.

Looking at the changelog of 1.38.0:

* Fix bug setting priority for IP addresses.
* Static IPv6 addresses from "ipv6.addresses" are now preferred over addresses from DHCPv6, which are preferred over addresses from autoconf. This affects IPv6 source address selection, if the rules from RFC 6724, section 5 don't give a exhaustive match.

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/nm-1-38/NEWS

It looks like Ubuntu just introduced that bug by upgrading to 1.36.6, while a proper fix has only landed in 1.38.0, leaving the 1.36.x series now broken. Please either backport the fix from 1.38.0 or revert to 1.36.4, which was working fine.

System information:

/etc/os-release:

PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

apt info network-manager:

Package: network-manager
Version: 1.36.6-0ubuntu1

More background here: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1974428/comments/11

[Kevin (the7thstranger)]

Comparing the output of `ip -6 a`, you can see that the dynamic addresses are no longer at the top of the list, where they should be.

Before (network-manager 1.36.4):

2: eno0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2a10:3781:xxxx::bd0/128 scope global dynamic noprefixroute
       valid_lft 43193sec preferred_lft 43194sec
    inet6 fd10:3781:xxxx::bd0/128 scope global dynamic noprefixroute
       valid_lft 43193sec preferred_lft 43194sec
    inet6 fd10:3781:xxxx:0:9875:4dec:b9f9:e768/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 2a10:3781:xxxx:0:f802:2428:9af1:dcb3/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::36e2:ec4c:fddb:3c89/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

After (network-manager 1.36.6):

2: eno0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fd10:3781:xxxx:0:9875:4dec:b9f9:e768/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 2a10:3781:xxxx:0:f802:2428:9af1:dcb3/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 2a10:3781:xxxx::bd0/128 scope global dynamic noprefixroute
       valid_lft 43194sec preferred_lft 43194sec
    inet6 fd10:3781:xxxx::bd0/128 scope global dynamic noprefixroute
       valid_lft 43194sec preferred_lft 43194sec
    inet6 fe80::36e2:ec4c:fddb:3c89/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

On another machine with Debian sid and network-manager 1.38.0, it looks the way it should be again (dynamic addresses at the top of the list, preferred to autoconfigured / temporary addresses):

2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fd10:3781:xxxx::5bf/128 scope global dynamic noprefixroute
       valid_lft 43193sec preferred_lft 43193sec
    inet6 2a10:3781:xxxx::5bf/128 scope global dynamic noprefixroute
       valid_lft 43193sec preferred_lft 43193sec
    inet6 2a10:3781:xxxx:0:42cd:4f1b:89b8:77fd/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fd10:3781:xxxx:0:8a59:df52:9ea1:e7c8/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::a738:71dc:f10e:924e/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

So this bug was not present in NetworkManager 1.36.4, introduced in 1.36.6, and then fixed in 1.38.0.

The correct order for IPv6 addresses should be:

1. Static addresses (if present)
2. DHCPv6 addresses (shown as 'dynamic' with ip -6 a)
3. Temporary SLAAC addresses (in case IPv6 Privacy Extensions are enabled, otherwise not present)
4. Global SLAAC addresses (the long ones with :0: after the network part ending with /64)

With NetworkManager 1.36.6 the order is now 3 > 4 > 1 > 2.

I guess these commits are relevant:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/c631aa48f034ade2b5cb97ccc4462d56d80174e7

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/257221d1986b56cbb2e329fcc74a2daca145b7aa

Bottom line: addresses are now...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sebastien Bacher (seb128)]

Thanks for the detail, you might want consider to report the issue directly upstream next time which is the right place to fix a regression due to a version update,, I've forwarded it now to https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/1021

[Kevin (the7thstranger)]

@seb128 What is the plan for the network-manager package in Ubuntu jammy-updates? Is it to be left broken until upstream fixes this bug?

Because I think that the problem this last version introduces is a lot bigger than the problem(s) it solves. Therefore I would say that it can be defended to upload a new version that restores 1.36.4-2ubuntu1 for the time being, and then re-sync with upstream when (or if) this bug is fixed.

As far as I'm aware, the update from 1.36.4 to 1.36.6 did not close any outstanding Launchpad bugs, so it didn't fix anything from Ubuntu's perspective. It just introduced this regression.

[Sebastien Bacher (seb128)]

@Kevin. Thanks, it's a bit of an annoying situation. The current update is not only in proposed but was moving to updates before you reported the regression, which means we can't simply delete it from proposed, we need a fix or revert. I was hoping upstream would figure out a fix quickly enough that we could do another bugfix upload. If that doesn't play out this way we need to figure out what to do.

The initial reason for SRUing the update, out of keeping up with upstream fixes, was that it fixes some hotspot issues which is a topic we are getting complain about (https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/6a82dd18)

We could revert back the version, including other fixes. Or revert the change that created the regression you reported. Note that while it seems an annoying issue to you that's the only report we got so far which suggest it's not a common configuration for our users, which is also a reason why I decided to wait a bit to see if upstream was reacting before reverting.

[Kevin (the7thstranger)]

Unfortunately I was not tracking jammy-proposed, so I only found out about this bug when it entered jammy-updates.

I don't think my configuration is that exotic to be honest. Many companies use DHCPv6 servers giving out static leases, and many companies use firewalls to restrict incoming (SSH) connections to their servers. And not many network administrators disable SLAAC as that would break IPv6 for all Android users. So corporate and home networks with DHCPv6 and SLAAC enabled are quite common.

How viable would it be to create something like 1.36.7~really1.36.4 or something like that, which would be the previous version with only commit 6a82dd18 cherry picked to fix the hotspot issue and not introduce the source address selection bug?

I suppose upstream would at some point release 1.36.9, which will hopefully have this bug fixed, and also have commit 6a82dd18 included, so Ubuntu can go back to the upstream version then.

[Justin (vissenvijver)]

@seb128 I am also affected by this bug and privacy-wise every other Ubuntu 22.04 user as well, but I didn´t respond, trusting the process that this bug would be fixed. So I am supporting your efforts getting an upstream fix shortly or alternatively reverting the change that created the regression.

[Sebastien Bacher (seb128)]

Upstream provided me a patch to try which I uploaded to a ppa now,
https://launchpad.net/~ubuntu-desktop/+archive/ubuntu/ppa/+build/23833629

Could you install it and see if that fixes the issue?

[Kevin (the7thstranger)]

This seems to restore previous behaviour yes.

[Justin (vissenvijver)]

Thank you, this solution fixes the bug for me as well.

[Sebastien Bacher (seb128)]

Thanks for the testing and for following up on the issue. I've uploaded that patch to the SRU queue now

[Brian Murray (brian-murray)]

Hello Kevin, or anyone else affected,

Accepted network-manager into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.36.6-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Kevin (the7thstranger)]

kevin@arcadia:~$ apt info network-manager
Package: network-manager
Version: 1.36.6-0ubuntu2

kevin@arcadia:~$ apt info libnm0
Package: libnm0
Version: 1.36.6-0ubuntu2

I have installed network-manager and libnm0 version 1.36.6-0ubuntu2 from jammy-proposed. After connecting to a network with both SLAAC and DHCPv6 enabled, the DHCPv6 address was selected as the source address again, like it was in version 1.36.4, prior to the update introducing this bug.

This restores the original and desired behaviour.

[Kevin (the7thstranger)]

It's fixed upstream as well now: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/6920b6ceb1c2e7856ad76e118ee5b4dd36130735

[Kevin (the7thstranger)]

Any updates regarding the release to jammy-updates?

[Launchpad Janitor (janitor)]

This bug was fixed in the package network-manager - 1.36.6-0ubuntu2

---------------
network-manager (1.36.6-0ubuntu2) jammy; urgency=medium

  * debian/patches/fix_addresses_order.patch:
    - platform: workaround for preserving IPv6 address order
      (lp: #1977619)

 -- Sebastien Bacher <email address hidden> Thu, 09 Jun 2022 11:27:56 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for network-manager has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.