Backport iproute2 6.8.0 to noble

[Matthieu Baerts]

[Impact]
Several networking features introduced in the upstream 6.2->6.8 kernels are not accessible to noble users because noble lacks the corresponding iproute2 update. This includes support for new hardware features such as per-VF offload settings (see bug 2060969), but many others that you can see in the attached changelogs.

Normally iproute2 is updated during the devel cycle to align with the kernel version, but it was missed this cycle and discovered too late for an FFe. We request an exception from the SRU team to do this as an SRU. This includes dropping the Ubuntu Fan patches, which adds features to iproute2 that no longer work since noble's kernels no longer support Ubuntu Fan.

[Test Case]
We'll run the 6.8 kernel self tests, which make use of a number of iproute2 features (see comment #17). NVIDIA's networking organization (Mellanox) has offered to put this through their QA process (details TBD). The upstream test suite will also run in the autopkgtests, though that suite is fairly stale.

[What Could Go Wrong]
Users who may be running noble userspace with a non-noble kernel that supports Ubuntu Fan would lose support for configuring it. If the SRU team considers this to be a regression we should avoid, then we can add the Ubuntu Fan patches back.

iproute2 6.8 upstream retains backwards compatibility with earlier releases (see comment #17). There are no commits with a Fixes: annotation since the v6.8.0 tag was applied upstream.

iproute2 is obviously a key package, and any upstream version bump carries risk. This includes the possibility of breaking networking for users of an Ubuntu LTS release.

[Matthieu Baerts (matttbe)]

I hope that's OK if I put the 24.04's feature freeze as milestone.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matthieu Baerts (matttbe)]

Hello,

I was quickly checking if I could help preparing the upgrade of IPRoute2 packages for Noble (before the final freeze).

In short, IPRoute2 has Ubuntu specific modifications. The merge was not difficult, the code is there:

https://code.launchpad.net/~matttbe/ubuntu/+source/iproute2/+git/iproute2/+ref/merge-lp2051672-noble

But then, I realised that the 6.8 Linux kernel in Noble no longer includes Ubuntu FAN support. We could then drop these Ubuntu specific patches and sync with Debian.

Andrea Righi mentioned this on IRC:

> ubuntu-fan has been temporarily dropped from the noble kernel (see the discourse post https://discourse.ubuntu.com/t/introducing-kernel-6-8-for-the-24-04-noble-numbat-release), we're still trying to figure out if there are still some users that rely on this feature, if not we can definitely sync with Debian & upgrade

Because the situation is still unknown, it might be good to continue to merge IPRoute2 with the version from Debian. In this case, please use the version I shared here above. I understand that we might be short on time, but it might be good if someone from the Kernel Packages team could have a look at this :)
That would help to support networking features added in Kernel >6.1!

[dann frazier (dannf)]

It seems like this bug should now be a FFe (see https://wiki.ubuntu.com/FreezeExceptionProcess) - so converting...

[Matthieu Baerts (matttbe)]

Hi Dann,

Thank you for doing that!

Is there something I can do to help?

Note that I published the modifications I did [1] in a dedicated PPA [2]. This was a merge with Debian.

If it is no longer needed to support Ubuntu FAN -- see my comment in #3 -- all Ubuntu-specific patches can be removed, and a simple "sync" with Debian can be done instead.

[1] https://code.launchpad.net/~matttbe/ubuntu/+source/iproute2/+git/iproute2/+ref/merge-lp2051672-noble
[2] https://launchpad.net/~matttbe/+archive/ubuntu/iproute2-lp2051672

[dann frazier (dannf)]

Thanks Matthieu. I spoke to the kernel team and I think they will handle the update. I'm just trying to help unblock them by getting this reviewed/agreed by our release team ahead of time.

[Andrea Righi (arighi)]

According to the information that I collected (asking around, investigating, etc.), it seems that there are no critical users of ubuntu-fan.

Moreover, considering the impact of the ubuntu-fan kernel patches (that would require a major refactoring to avoid breaking the network eBPF ABI), we decided to proceed with the plan to deprecate this feature in all the noble kernels.

Therefore, all the user-space components that rely on such feature can follow the same plan and drop their corresponding ubuntu-fan support (iproute2 included).

[Utkarsh Gupta (utkarsh)]

Hello,

After looking at it and discussing this with Dann Frazier yesterday (via IRC on #ubuntu-release), we discussed that iproute2 is an important package, it's in main, and has a huge bunch of reverse dependencies. Bumping from 6.1.x to 6.8.x is a bit too much atm, especially given how much is already happening and this being an LTS release.

So with that, I'm going to NACK it for FFe and ask this to be uploaded via 0-day SRU or a regular SRU. :)

Sorry but I hope you understand.

[dann frazier (dannf)]

Thank Utkarsh. Yeah, we just dropped the ball on not getting this done sooner. I'd like to see if we can make the case for SRU'ing 6.8 in, so let's convert the bug to that.

I don't think we can justify dropping FAN support in an SRU, so let's add that port back in.

We'll need to make a convincing case to the SRU team that the risk of regression is minimal. Our partners at NVIDIA/Mellanox have offered to put this release through their testing. I'll see if they can provide details on what their tests will cover.

@matttbe - is there any documentation about upstream compatibility guarantees and regression testing done that you're aware of, or any tests that we could run beyond those in our autopkgtests?

[Matthieu Baerts (matttbe)]

> I don't think we can justify dropping FAN support in an SRU, so let's add that port back in.

Even if it will not be used in Ubuntu 24.04, and dropping these patches would stop the diversion with Debian?

> @matttbe - is there any documentation about upstream compatibility guarantees and regression testing done that you're aware of, or any tests that we could run beyond those in our autopkgtests?

@dannf: IPRoute2 is following the kernel development, and it is not supposed to break the compatibility with older kernel versions (also because the kernel's user API is not supposed to break).

There is a test suite included in the source code, but it looks like it is not tested when building the .deb package, check the 'debian/rules' file:

    override_dh_auto_test:
            # upstream test suite needs root and leaves machine unclean, skip it

I never used it, (and it looks like it is not often updated), but IPRoute2 (ip, tc, ss, etc.) is also heavily used in the kernel selftests:

    $ git grep -cw -e ip -e tc -e ss -- tools/testing/selftests/ | wc -l
    635

Maybe these tests are already being validated when building a new kernel for Ubuntu? That's another way to validate IPRoute2.

Other than that, Debian is using IPRoute2 versions > 6.1 for a long time now.

[dann frazier (dannf)]

On Mon, Apr 22, 2024 at 11:07:54AM -0000, Matthieu Baerts wrote:
> > I don't think we can justify dropping FAN support in an SRU, so let's
> add that port back in.
>
> Even if it will not be used in Ubuntu 24.04,

I'm happy to drop it if the stable release team would accept it.

Here's the policy:
  https://wiki.ubuntu.com/StableReleaseUpdates

However, I don't see how we can argue that dropping FAN support is
*not* a regression, even if the kernel we ship does not support it.

> and dropping these patches would stop the diversion with Debian?

That's not a goal of our SRU process. We can always realign with
Debian during the next devel cycle.

> > @matttbe - is there any documentation about upstream compatibility guarantees and regression testing done that you're aware of, or any tests that we could run beyond those in our autopkgtests?
>
> @dannf: IPRoute2 is following the kernel development, and it is not
> supposed to break the compatibility with older kernel versions (also
> because the kernel's user API is not supposed to break).
>
> There is a test suite included in the source code, but it looks like it
> is not tested when building the .deb package, check the 'debian/rules'
> file:
>
> override_dh_auto_test:
> # upstream test suite needs root and leaves machine unclean, skip it

The package does seem to include this in its autopkgtests
(debian/tests/testsuite.sh), so I think we have that coverage.

>
> I never used it, (and it looks like it is not often updated), but IPRoute2 (ip, tc, ss, etc.) is also heavily used in the kernel selftests:
>
> $ git grep -cw -e ip -e tc -e ss -- tools/testing/selftests/ | wc -l
> 635
>
> Maybe these tests are already being validated when building a new kernel for Ubuntu? That's another way to validate IPRoute2.

Thanks for the pointer!

  -dann

> Other than that, Debian is using IPRoute2 versions > 6.1 for a long time
> now.
>

[Matthieu Baerts (matttbe)]

> However, I don't see how we can argue that dropping FAN support is
> *not* a regression, even if the kernel we ship does not support it.

I understand.

I guess the situation is a bit particular here. If the official kernel doesn't support it -- something Ubuntu-specific, not upstreamed -- it means Ubuntu FAN cannot be used: there is no way to even validate that these patches work on Ubuntu 24.04. The regression would be more on the kernel side :)

>> and dropping these patches would stop the diversion with Debian?
>
> That's not a goal of our SRU process. We can always realign with
> Debian during the next devel cycle.

Good point. I just wanted to suggest that dropping these patches could be seen as a "safer approach" as it is closer to the upstream version, what is on Debian and other distributions.

> The package does seem to include this in its autopkgtests
> (debian/tests/testsuite.sh), so I think we have that coverage.

Oh, I missed that, thank you for having checked! Good news then!

[Andrea Righi (arighi)]

Hello, any progress on this? Now that ubuntu-fan is officially deprecated in Noble can we simply sync iproute2 with Debian? Is there any pending activity / requirement that are preventing this? Thanks!

[dann frazier (dannf)]

Hi Andrea - since this is now a stable release update, I *think* the next steps should be:

1) Edit this bug to use the SRU Template
  https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template
(now done)
Since your team owns this package, please review that and let me know if you have any concerns with that text.

2) Wait for 24.10 ('oracular') to open for devel and forcesync iproute2 w/ Debian (https://wiki.ubuntu.com/StableReleaseUpdates#Development_Release_Fixed_First).

3) Upload the SRU, and wait for feedback from the sru team. If approved, we'll need to run the tests in the [Test Case] section and make sure there are no regressions from 6.1.

[Andrea Righi (arighi)]

Thanks @dannf for updating the bug! The SRU description looks good to me and everything seems reasonable, same with the plan.

I'll keep monitoring this tracker and we'll proceed once oracular will be open.

[Danilo Egea Gondolfo (danilogondolfo)]

Hi folks, do you think we are good to sync iproute2 from Debian to Oracular at this point?

[dann frazier (dannf)]

I've now sync'd it to Oracular.

[Andrea Righi (arighi)]

I have an update on this. I've talked with the LXD team during the engineering sprint in Madrid and at the moment we can't break ubuntu-fan or we could potentially break some running LXD installations.

I asked the LXD team to start working as soon as possible to remove the support for this feature, that at this point is going happen in Oracular.

Therefore, good job with the iproute2 sync in Oracular @dannf. Unfortunately in Noble we still need to wait for now and support ubuntu-fan... In the meantime I've sent an updated patch set to re-enable ubuntu-fan in the kernel (with a small redesign to prevent potential incompatibilities with the upstream vxlan_policy rules):

https://lists.ubuntu.com/archives/kernel-team/2024-May/150684.html

[Matthieu Baerts (matttbe)]

Hello,

For Noble, do not hesitate to look at what I did a few weeks ago:

  https://code.launchpad.net/~matttbe/ubuntu/+source/iproute2/+git/iproute2/+ref/merge-lp2051672-noble

It is a merge with Debian (IPRoute2 6.8.0-1) where I refreshed the patches to support Ubuntu FAN. Note that I was not able to test this feature (the kernel in Noble didn't support it).

[Matthieu Baerts (matttbe)]

IPRoute2 6.9.0-1 is in Oracular: Fix Released