Ubuntu 20.04 and 20.04.1 images use linux-generic-hwe-20.04 instead of linux-generic

[Erick Brunzell]

Beginning about January 8th Focal users that installed Focal using either the Ubuntu 20.04 or 20.04.1 installation media began getting updated to the 5.8 series linux kernel. Those who had installed using the Ubuntu Focal Beta were not effected. This is because Focal Beta was properly built with ‘linux-generic’ but the 20.04 and 20.04.1 images were built with ‘linux-generic-hwe-20.04‘. This can be easily verified by looking at the iso manifest:

https://releases.ubuntu.com/20.04.1/ubuntu-20.04.1-desktop-amd64.manifest

Note: old-release manifests (where 20.04 Beta and 20.04 reside) must be downloaded here:

https://old-releases.ubuntu.com/releases/20.04/

The expected behavior is for Ubuntu Focal users to remain on the 5.4 series kernel unless they install with 20.04.2 or later images as indicated here:

https://wiki.ubuntu.com/Kernel/LTSEnablementStack

While not entirely updated it says “if one wants to remain on the original GA (General Availability) stacks, the options [include]; Install from a previous 12.04.0/12.04.1/14.04.0/14.04.1/16.04.0/16.04.1/18.04.0/18.04.1 point release and update”.

Jump just below that to the Focal specific notes and it says “The 20.04.2 and newer point releases will ship with an updated kernel and X stack by default for the desktop”. Since 20.04.2 is not even due for release until next month it makes no sense for updates to the 5.8 series kernel to be occurring unless the focal-proposed repos are enabled even if HWE protocols had been changed.

It’s also worth noting that thankfully no accompanying HWE X stack is yet available in the repos. I say thankfully because my efforts at downgrading the X stack in 16.04 and 18.04 were never successful! But the lack of a matching X stack may explain the many complaints on the forums about broken graphics???

It’s also worth noting that even the release notes say that “Ubuntu 20.04 LTS is based on the long-term supported Linux release series 5.4“.

https://wiki.ubuntu.com/FocalFossa/ReleaseNotes#Linux_Kernel

Scroll to the bottom here and you’ll also see that 20.04 and 20.04.1 are supposed to be supported with the 5.4 series kernel for the entire 5 year life span:

https://ubuntu.com/kernel/lifecycle

The other flavors’ 20.04 and 20.04.1 images were all built correctly with ‘linux-generic’ with the possible exception of Ubuntu Studio which uses a kernel stack I’m not familiar with. I’m also uncertain about Ubuntu server because I’m simply not familiar with it.

Original bug description below:

##################################################################

Just as the title says, the 20.04 and 20.04.1 images use the HWE version of linux-generic resulting in upgrades to the 5.8 series kernel. This seems to effect Ubuntu only, or I should say I checked the Kubuntu, Ubuntu Mate, Xubuntu, and Lubuntu iso manifests which all correctly list linux-generic. Also HWE is not truly complete without the accompanying HWE Xstack.

I checked all the documentation I could find and this was clearly not intentional. In fact the manifest for Ubuntu Focal Beta still used linux-generic, so this got messed up some time between April 3, 2020 and April 23, 2020. Here's just one example of the documentation for kernel support in Focal LTS:

https://ubuntu.com/about/release-cycle#ubuntu-kernel-release-cycle

Installations performed with 20.04 and 20.04.1 media were supposed to remain on the 5.4 series kernel throughout Focal's 5 year lifespan as had been the case since HWE was introduced. The first step in fixing this needs to be stopping fresh installs of Focal using the 20.04 and 20.04.1 media from immediately upgrading to the 5.8 series kernel.

It might be a little tricky to revert users from 5.8 to 5.4, but there have been reports of breakage, particularly concerning Broadcom wifi drivers and some graphics problems. But the graphics problems could also be exacerbated by the lack of the matching HWE Xstack? At any rate it's a bug.

[Chris Guiver (guiverc)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

If you look at the Ubuntu 20.04 LTS (focal fossa) release notes you'll see

https://wiki.ubuntu.com/FocalFossa/ReleaseNotes

"Ubuntu Desktop flavour now always tracks HWE kernel (hardware enablement). It means that from 20.04.2 release Ubuntu Desktop will gain new major kernel versions every 6 months through to summer of 2022."

You haven't indicated if you're talking Ubuntu Desktop, if so that was announced & is expected behavior (at least as I understand it).

Please execute the following command only once, as it will automatically gather debugging information, in a terminal:

apport-collect 1911614

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[Erick Brunzell (lbsolost)]

Unfortunately apport-collect just says "no additional info collected" (see pic added). I presume because 'ubuntu-meta' is not an installed package????? I struggled a bit trying to decide what the best way was to report a bug like this because it's not actually a bug in the kernel itself, it's just that I believe the 5.8 series kernel should not be installed without specific user action to do so.

That had been the case in the past. LTS installs using the original iso or the first point release iso would remain on the original GA (General Availability) throughout the duration of the 5 year support cycle. In fact I see the HWE Wiki has been updated to say "The 20.04.2 and newer point releases will ship with an updated kernel and X stack by default for the desktop":

https://wiki.ubuntu.com/Kernel/LTSEnablementStack

And 20.04.2 is not due for release until early February. It's worth mentioning that LTS point releases are frequently delayed by a week or two if bugs are encountered. Anyway I would expect an installation performed with 20.04.2 media to include the HWE kernel. You'll also notice that they do say "kernel and X stack" and as mentioned above the HWE X stack is not yet even in the repos!

I will probably be testing the 20.04.2 daily build very soon and I'll then report a bug against 'linux-generic-hwe-20.04' making note that the accompanying X stack is missing. And yes, this does effect only Ubuntu Desktop because as mentioned above I checked some of the other flavors manifests. Only the Ubuntu isos have the HWE version of kinux-generic installed.

[Chris Guiver (guiverc)]

Thanks for the update, and attempt to `apport-collect`. I too was unsure what package to re-file against, and had forgotten it was against the meta package when I pasted the apport-collect request.

Ubuntu 20.04.2 release date (https://wiki.ubuntu.com/FocalFossa/ReleaseSchedule) refers to the ISO release date. Installed systems will upgrade to it before the scheduled date (usually a few days, however if the ISO release date is pushed back/delayed, the installed systems upgrades do not get delayed except for extraordinary circumstances) as is normal.

The 5.8 kernel has been the -edge option for some time now in focal repositories, so people using 20.04 that have installed packages that have dependencies that bring in the latest -edge kernel will have 5.8 already which is expected; any who remained on the the stable kernel still use 5.4

I suspect the issue is understanding, or you've added additional packages added to your system (graphics drivers, 3rd party etc) that have pulled in the -edge kernel option and you off the default stable kernel path. I hoped to gain more detail from the apport-collect, but the package name needs correction first, and I don't know what it should be against (if it's a bug, If there is a bug, I suspect it's only documentation).

I've assumed you're using Ubuntu 20.04 Desktop, thus the 5.8 kernel is expected due to announced change; if you're on Ubuntu Server, then this is more of an issue as that doesn't default to HWE like Desktop does. The different path now (from 20.04) between Desktop & Server does make the HWE documents a little harder to write for (both were the same up to and including 18.04/bionic).

FYI: Today's Lubuntu 20.04.2 daily (2021-01-14) still has the 5.4 kernel, but it's the only current focal daily I've used today.

[Dave Korman (david-korman)]

This affected me too - it broke the Ubuntu version of Virtualbox.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Erick Brunzell (lbsolost)]

I added the packaging tag because this was a packaging error in the 20.04 and 20.04.1 images.

[Erick Brunzell (lbsolost)]

I updated the bug description to better explain why I believe this is a bug, but I left the original content below the break.

[Erick Brunzell (lbsolost)]

I filed bug #1912032 concerning the failure of nvidia drivers to install during a fresh installation of Ubuntu 20.04.1 because when download updates is selected it auto-updates to a 5.8 series kernel during installation.

[Brian Murray (brian-murray)]

I did a fresh installation of the 20.04 LTS media from 20200423 and can confirm this. See the attached screenshot.

[Łukasz Zemczak (sil2100)]

It seems this behavior is expected, although probably insufficiently documented. We will make sure to clear things out in various documentation bits, but with 20.04 it has been decided that the Ubuntu Desktop flavor will be shipping a HWE kernel by default from the start. Apologies for this unclear situation, I have no idea how this got missed in the announcements.

Per:
https://wiki.ubuntu.com/FocalFossa/ReleaseNotes#Ubuntu_Desktop

[Erick Brunzell (lbsolost)]

Documentation is horribly insufficient in this regard. To quote the link given, "Ubuntu Desktop flavour now always tracks HWE kernel (hardware enablement). It means that from 20.04.2 release Ubuntu Desktop will gain new major kernel versions every 6 months through to summer of 2022".

The words "from 20.04.2 release Ubuntu Desktop will gain new major kernel versions every 6 months" speak volumes, sadly incorrect volumes. 20.04.2 is not even due for release until next month but we've been getting HWE kernel updates for a couple of weeks already!

All of the following documentation needs serious updating and must include methods to remain on the GA kernel if one wishes:

The first, also from the release notes, states clearly "Ubuntu 20.04 LTS is based on the long-term supported Linux release series 5.4":

https://wiki.ubuntu.com/FocalFossa/ReleaseNotes#Updated_Packages

None of these have been updated to adequately reflect the change to opt-out of HWE rather than opt-in if using original (20.04) or first point release (20.04.1) installation media as had always been the case previously.

https://wiki.ubuntu.com/Kernel/LTSEnablementStack
https://wiki.ubuntu.com/Kernel/RollingLTSEnablementStack
https://ubuntu.com/kernel/lifecycle

[Michel-Ekimia (michel.ekimia)]

Thanks for all the valuable info about this problem.

I'm not sure I understand if the observed behaviour is the one expected ?

On of my client had his Ryzen 2200g workstation broken ( initramfs prompt ) after a 5.8 update , going back to 5.4 worked

- Virtualbox module still does not build on 5.8

- Broadcom Wifi module neither

Then I don't think 5.8 was ready to be deployed ?

Should we force remove linux-hwe package from workstations for now ?

[Dimitri John Ledkov (xnox)]

https://wiki.ubuntu.com/Kernel/LTSEnablementStack was updated with expanded instructions around 20.04 LTS, and make it more obvious that one is most likely tracking HWE kernel on Desktop; unless one is on the OEM kernel. Note that on certified hardware downgrades to GA stack may cause loss of functionality.

References from LTSEnablementStack to RollingLTSEnablementStack, have been clarified that rollingness of the hwe stacks is same as it was since 16.04. It used to be the case that one could install Quantal/Raring/Saucy hwe kernels and keep them until Trusty hwe comes out. However since 16.04, we have been rolling hwe stacks, such that the three intermediate kernel series roll over.

ubuntu.com/kernel/lifecycle update is in progress and is tracked at https://github.com/canonical-web-and-design/ubuntu.com/issues/9101

[Dimitri John Ledkov (xnox)]

@michel.ekimia

https://bugs.launchpad.net/ubuntu/+source/virtualbox/+bug/1901904 has full support for the v5.8 kernel in focal, released on 18th of January 2021.

What package versions are you referring to? for which release? and which archive origin? And which combinations of the hypervisor/guests?

[Dimitri John Ledkov (xnox)]

Also release team posted this update https://discourse.ubuntu.com/t/improvements-for-hardware-support-in-ubuntu-desktop-installation-media/20606 about this issue, which they were forced to do before the weekend instead of during normal working hours this week. They were hoping to collect more evidence before posting, however another statement pushed out without coordination with the tech-board forced the release team to publish incomplete statement.

[Brian Murray (brian-murray)]

@michel.ekimia

The Broadcom wifi driver issue has been fixed in https://bugs.launchpad.net/ubuntu/+source/bcmwl/+bug/1872908

[Brian Murray (brian-murray)]

We believe that all the documentation errors and shortcomings have been addressed, subsequently I'm setting this bug to Fix Released. Additionally, the bugs with packages due to the 5.8 kernel have also been resolved and if they have not new bug reports about the specific package should be opened.

Thanks again for reporting this bug and ensuring that we've communicated this change in behavior.