Recent mainline packages are built with Hirsuite 21.04, not Focal 20.04 LTS

[TuxInvader]

Hi all,

The Mainline wiki states that the mainline kernels are built with the previous LTS toolchain, but the recent 5.12.x and 5.11.x releases are being built with Hirsuite 21.04, and before that Groovy? If this is intentional, then the wiki should be updated to reflect the change in policy.

From https://wiki.ubuntu.com/Kernel/MainlineBuilds

  Mainline kernel build toolchain
  These kernels are built with the toolchain (gcc, g++, etc.) from the previous Ubuntu LTS release.
  (e.g. Ubuntu 14.04 "Trusty Tahr" / 16.04 "Xenial Xerus" / 18.04 "Bionic Beaver", etc.) Therefore,
  out-of-tree kernel modules you already have built and installed for use with your release kernels
  are not likely to work with the mainline builds.

The 5.12 kernel was built with GCC 10.3.0, and 5.11.16 with 10.2.0. On my Focal LTS system I have GCC 9.3.0.

The Mainline kernel build toolchain
These kernels are built with the toolchain (gcc, g++, etc.) from the previous Ubuntu LTS release. (e.g. Ubuntu 14.04 "Trusty Tahr" / 16.04 "Xenial Xerus" / 18.04 "Bionic Beaver", etc.) Therefore, out-of-tree kernel modules you already have built and installed for use with your release kernels are not likely to work with the mainline builds.

The *linux-headers-generic* packages have unmet dependencies on 20.04 LTS.

I could install Groovy built kernels fine, but the Hirsuite ones built with GCC 10.3.0 appear to require libc6 >= 2.33. So the new kernels can't be installed on Focal (libc 2.31).

Thanks,
Mark

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1926938

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[TuxInvader (tuxinvader)]

This is not a bug report, but a question on the build policy of the mainline kernels?

Should mainline kernels work on the previous LTS or not?

[Cornelius Zwalin (zwalin)]

> Should mainline kernels work on the previous LTS or not?

I think the main issue is with focal, the *current* LTS, which can't use mainline kernels.

[Kevin Hester (kevinh)]

Though this problem also just appeared for me on ubuntu 20.10, when I tried to upgrade my working mainline kernel from 5.11.16 to 5.11.17.

[Brian K. White (bkw777)]

A user just claimed that the new kernel packages allow themselves to install on systems that don't have new enough libc6. Shouldn't there be libc6>=2.33 dependency in the .deb so that it won't allow itself to install?
https://github.com/bkw777/mainline/issues/87#issuecomment-831777690

[Cornelius Zwalin (zwalin)]

> Shouldn't there be libc6>=2.33 dependency in the .deb so that it won't allow itself to install?

linux-headers-*.deb has such dependency but not the other debs
This results in a broken installation where the kernel image and modules are installed but the header isn't

[TuxInvader (tuxinvader)]

This bug appears to be either:

1. A documentation bug. The wiki (https://wiki.ubuntu.com/Kernel/MainlineBuilds) needs updating to indicate that mainline kernels are built for the current Ubuntu release only. Remove the section about using the LTS tool-chain.

2. The automated build system is wrong, and it should be building mainline kernels on the current LTS (right now that's focal (20.04)), instead of the current release (21.04) which is what it actually does.

Another option might be to change the automated build system to either build kernels for both the current release *and* the current LTS release... Or generate source packages so users not on the target release can build their own packages.

[danmb (danmbox)]

It would be unfortunate for the "LTS" kernel version to not be installable on LTS Ubuntu.

[Carl Tuxe (c-tuxe)]

It would be a shame for 20.04 LTS users to no longer be able to make use of the Mainline kernel PPA

[danmb (danmbox)]

Still seeing this spurious libc 2.33 dependency (I *assume* it's not the kernel people who decide to change their libc policy in the middle of a release series -- so it's spurious) in 5.10.35.

Are the Ubuntu / Canonical people notified? I'm not sure who takes care of this "Mainline PPA" (which doesn't seem to have a Launchpad presence, but just a website with build products).

Is there anyone to CC on Launchpad, so that we may get some more or less official answer?

[TuxInvader (tuxinvader)]

I built a container for building mainline kernels. It build debs by default, but it can also build signed source packages, but unfortunately the sources are too big to be used with a PPA (they fill up the disk during build).

There is another issue with building mainline on focal, and that's the kernels require pahole >= 1.16 (from dwarves package). The groovy version of dwarves can be installed on focal, or you can use my PPA here: https://launchpad.net/~tuxinvader/+archive/ubuntu/lts-mainline

I was hoping to push my kernels to that PPA, but as I said - they're too big for launchpad.

If you want an easy(ish) way to compile your own mainline packages you can use this: https://hub.docker.com/r/tuxinvader/focal-mainline-builder

[danmb (danmbox)]

Super, I was just looking for a way to build kernels in a Docker container.

Maybe the Docker image could be modified to build dwarves for itself at some point.

I think you can request more space for a PPA from launchpad.

[TuxInvader (tuxinvader)]

I updated my container so you can pick which flavour to build (generic or lowlatency), and exclude some of the packages (udebs, cloud-tools). The PPA now has enough space to build the source package.

So I've published 5.12.2 generic packages here:

  https://launchpad.net/~tuxinvader/+archive/ubuntu/lts-mainline

I'll keep it updated with 5.12.x kernels until this gets resolved.

[Rael Gugelmin Cunha (rael-gc)]

Even 5.10 series is affected. I'm on 20.04 and was installing 5.10 series. Now I'm stuck on 5.10.32 because 5.10.33 requires new version of libc6.

[danmb (danmbox)]

There is/was also some discussion on https://askubuntu.com/questions/1334633/mainline-kernel-now-depends-on-libc6-2-33-non-installable-in-focal -- I'm also interested in 5.10.

[TuxInvader (tuxinvader)]

I'll upload the latest 5.11.x kernel to a lts-mainline-previous PPA, and longterm 5.10.x kernels to lts-mainline-longterm.

5.10.35 will be available here: https://launchpad.net/~tuxinvader/+archive/ubuntu/lts-mainline-longterm

5.11.19 will be available here: https://launchpad.net/~tuxinvader/+archive/ubuntu/lts-mainline-previous

cheers :-)

:wq

[danmb (danmbox)]

Nice work and very timely, @tuxinvader. Perhaps leave an answer on askubuntu for future reference.

[Timofei Kushnir (t-kushnir)]

Yesterday new version (5.12.3-051203) of mainline kernel was issued. The problem is still there:

. . .
Setting up linux-headers-5.12.3-051203 (5.12.3-051203.202105120733) ...
dpkg: dependency problems prevent configuration of linux-headers-5.12.3-051203-generic:
 linux-headers-5.12.3-051203-generic depends on libc6 (>= 2.33); however:
  Version of libc6:amd64 on system is 2.31-0ubuntu9.2.

dpkg: error processing package linux-headers-5.12.3-051203-generic (--install):
 dependency problems - leaving unconfigured
. . .

[fuomag (fuomag)]

Still present on 5.12.4 as well

[danmb (danmbox)]

Running fine on 5.10.36 from tuxinvader's https://launchpad.net/~tuxinvader/+archive/ubuntu/lts-mainline-longterm (mentioned above).

Mainline PPA official packages still broken on focal, 5.10.38 is coming up.

[Martin Pecka (peci1)]

@tuxinvader Could you please also set up a bionic PPA? I manually download the DEBs and install them on Bionic and everything works fine.

[TuxInvader (tuxinvader)]

@peci1 there's nothing to stop you adding my focal ppa to your bionic system. The apt-add tool might refuse to add it, but you can manually create the ppa.list file yourself if it does?

Create: /etc/apt/sources.list.d/mainline-ppa.list with the below content

deb http://ppa.launchpad.net/tuxinvader/lts-mainline/ubuntu focal main
deb-src http://ppa.launchpad.net/tuxinvader/lts-mainline/ubuntu focal main

[Martin Pecka (peci1)]

Ah, you're right. I expected there should be a metapackage linux-5.12.4, but now I see I have to install the respective packages like linux-header-5.12.4* etc. Thanks!

[Jorge Gustavo (jgr)]

I was able to install 5.12.4 using @tuxinvader ppa on 20.04 with current libc6 2.31. Great work @tuxinvader!

I have used:

sudo add-apt-repository ppa:tuxinvader/lts-mainline
sudo apt install linux-image-unsigned-5.12.4-051204-generic linux-modules-5.12.4-051204-generic linux-headers-5.12.4-051204-generic

[chang long (2021yunotcompile)]

@tuxinvader is there a tutorial somewhere on how to manually replicate your build (specifically on building dependencies)? most tuts are wildly out of date and are not working. i'd like to install the ppa but also learn how you did it.

[DanglingPointer (ferncasado)]

All you guys mocking around with ppas could have compiled 5.12.5 by now 20x!

The last working mainline kernel is 5.11.16. Nothing later works when installing into 18.04 LTS.

I've been forced to compile each latest kernel since. But because of that predicament, I've now got it down to a fixed repeatable routine and can do it with my eyes closed or even script it at some point. Best thing about it is it is compiled against my cpu architecture and with -O3 optimisations (just like gentoo).

[danmb (danmbox)]

Just a quick tip: to see all available 5.10.3* packages,

printf '%s\0' linux-{image-unsigned,headers,modules}-5.10.{3,4} |
  xargs -0 -n 1 apt-cache pkgnames | LC_ALL=C sort | less

@DanglingPointer the PPA is based on a Docker image which in turn is based on a GitHub project (see comment 11) of which the whole point is to build kernels. Now, short of a PPA, how do you propose keeping multiple systems up-to-date? Move around raw deb's?

[DanglingPointer (ferncasado)]

PPAs are a mechanism for "moving around raw debs"... ;-)

But you have a point, however if all systems are in the same network, it is far much more easier to copy all debs to all servers and sudo dpkg -i./*.deb in automation.

If systems are geographically not in the same network and crossing the internet (IoT for example), then yes, PPA's are a way to address the use-case, yes; or automation by fetching the debs from an internet source (AWS S3,personal download space, etc). Many ways of accomplishing the same thing without the limitations of launchpad.

Alternatively if the disparate systems can afford it, automate download of mainline from kernel.org, copy config from ubuntu mainline kernels for the same version; and compile using -march=native -mtune=native -O3 -pipe. You'll get more bang for the buck on each hardware theoretically. This is the route I took since 5.11.16. You also get the chance to use GCC-11 with all its optimisations on -O3 (you'll need to install it first)! You'll also need this...

http://archive.ubuntu.com/ubuntu/pool/universe/d/dwarves-dfsg/dwarves_1.17-1_amd64.deb

From a principle point of view, Canonical should sort out the mess with their mainline kernels by following their stated goals by using LTS libraries and only upgrading during LTS releases. That way no unofficial community PPAs end up getting referenced for such a critical part of an operating system (the kernel) like what has been advertised above compiled in some docker container in some basement of someone's house!
The weak and gullible will end up using these random unofficial PPAs in some small business or worse yet a larger org without IT departments knowing!

I would sleep better knowing my kernel source came from kernel.org and that dwarves came from Ubuntu archives then some anonymous person's PPA!

[eitch (eitch)]

@DanglingPointer

Could you write down your way of building the kernel? I've been using mainline debs, but i would like to profit from the optimizations by building from source. Thanks a lot!

[TuxInvader (tuxinvader)]

I think we're getting way off topic here. The fundamental issue in this bug report is that the policy being employed when building the mainline kernels does not follow the policy documented. Either the mainline builders need to start following the policy (and building on LTS) or change the policy (and continue building on Ubuntu "next").

If you use my PPA and have questions about it, then please open an issue on the github project here: https://github.com/TuxInvader/focal-mainline-builder

DanglingPointer (cool handle) is correct in a lot of what he said, there is no inherent trust conveyed through the PPA system. Any one can setup a PPA and upload software packages to launchpad. Downloading software from a PPA is little different to downloading a snap or binary package from a website. You need to decided whether you want to trust the origin. I would like to point out the following though:

1. PPAs aren't compiled in the uploaders basement, a source package is built and uploaded to the PPA and then the compilation takes place on launchpad. I'm not suggesting this is more secure obviously. I could still have messed with the source package ;-)

2. Kernel PPAs are not more of a threat to your system than any other PPA. Your kernel runs your hardware and has full access to the system, but does a malicious kernel module have more access to your data (read the stuff you care about) than your user account? Beware of all PPAs, not just ones which package kernels.

In the interest of transparency, I have reverted my docker container to use the official dwarves package. If you want to use my container to build your kernel you can now see everything it does, it is after all a shell script.

If you chose to continue using my PPA, then I would also point out that those binaries are built using my dwarves package, because that's just how launchpad works.

Cheers :-)

PS. If you're not using 5.12.x, then you may have noticed that mainline hasn't built any packages for a while, 5.11.x and 5.10.x started failing around the time they switched to impish, and coincided with the build machine running out of disk space. I opened a report here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1928535

[Doug Smythies (dsmythies)]

When we try to help people, either here on launchpad or on forums, very often we would like them to try the latest mainline kernel, just as a test to determine if the issue has already been fixed upstream. This bug prevents that useful option for previous Ubuntu versions.

This dependency should be eliminated such that Ubuntu Mainline kernels can be installed on any valid Ubuntu release at least back to the previous LTS, as has been the normal situation for at least a decade.

Earlier someone was asking how to compile the mainline kernel. My notes, without the optimizations someone else was mentioning, on "how to" are here, and they are current, even thou there is a 15.10 tag: https://askubuntu.com/questions/718381/how-to-compile-and-install-custom-mainline-kernel/718662#718662

[DanglingPointer (ferncasado)]

@eitch

Try Dougss steps without the aggressive optimisations first so you get the hang of it. Make sure it works and your hardware all works.

My steps are a little different to Doug's and perhaps a little more complicated. But once you get the hang of it, it is repeatable and can be scripted. It also means you'll have optimised kernel for your hardware.
In a high level outline these are my steps.
Prerequisite is install gcc-11 and the dwarves version from Ubuntu archives.
test...
$ cc --version; gcov --vresion
You shoudl see respectively these:
cc (Ubuntu 11.1.0-1ubuntu1~18.04.1) 11.1.0
gcov (Ubuntu 11.1.0-1ubuntu1~18.04.1) 11.1.0

1) wget the kernel you want from kernel.org
This is more efficient cleaner then linus's git since it is a single file download and there's no git metadata overhead nor hidden folder and files.
Ensure it is downloaded to a directory location where there are no "deb" files.

2) To play safe with hardware compatibility we will leverage Ubuntu curated compiled kernel configuration by copying it to our custom build.
I do this by temporarily installing the Ubuntu compiled kernel using Mainline CLI - https://ubuntuhandbook.org/index.php/2020/08/mainline-install-latest-kernel-ubuntu-linux-mint/
Like I said, the kernel won't work, we just want the configuration they spent a lot of time curating for that exact version.

3) Uncompress the kernel you downloaded and change into the new directory created

4) Copy the config from /boot/ into the new kernel directory from step 3 but name the file ".config"

5) Uninstall the Ubuntu compiled kernel you installed. I use Mainline CLI

6) edit the ".config" file and look for "CONFIG_SYSTEM_TRUSTED_KEYS". Delete the string between the quotes. It should look like CONFIG_SYSTEM_TRUSTED_KEYS=""
We need to do this because we are doing a custom kernel. Otherwise build may fail as this could conflict.

7) I also optionally use xz compression for my kernel instead of the default LZ4. There are other options.
To change look for "CONFIG_KERNEL_LZ4=y" and comment it out then uncomment
Then uncomment "# CONFIG_KERNEL_XZ is not set" and change to "CONFIG_KERNEL_XZ=y"
Like I said this is optional. Only needed really if your /boot/ is a separate small partition with limited space.

8) Remember to save in steps 6 and 7 above

9) edit the MakeFile
I use nano for this
copy and replace every single "O2" with "O3"
This will ensure no gcc conflicts if by chance both are quoted in a compile command as option parameters.
Don't forget to save

10) export KCFLAGS="-march=native -mtune=native -O3 -pipe" KCPPFLAGS="-march=native -mtune=native -O3 -pipe"
To ensure we get to use the aggressive optimisations, we'll quote "-O3" again here just to be doubly sure!
-march=native and -mtune=native are pretty self explanatory. This is where we're basically saying... build this kernel for this CPU specifically and tune it for this CPU.
-pipe basically means '|' the output to the next thing that needs it instead of writing files. This will use more RAM which is faster than disk IO.
(special note on LTO using -flto; this is still not supported in the latest 5.12.x kernels with gcc-11. However I have seen ...

[TuxInvader (tuxinvader)]

This has become a security issue!

So the kernels are now being built with impish, and unless people are using my PPA or building their own, then the last install-able kernel for LTS (focal) was 5.11.16.

The 5.11.x series went EOL with 5.11.22, and all 5.11.x have a level 7.8 CVE: https://nvd.nist.gov/vuln/detail/CVE-2021-32606

As far as I can see this is still vulnerable up to and including 5.12.8. Linus applied the fix to his tree here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/net/can?id=2b17c400aeb44daf041627722581ade527bb3c1d but that hasn't yet been applied to the stable 5.12.x branch.

Ooops: https://www.openwall.com/lists/oss-security/2021/05/28/2

If you're using mainline on focal, then you might want to drop back to a 5.10.x release which doesn't have the vulnerability, or use my PPA, or follow one of the build instructions posted in this thread.

[Kai-Heng Feng (kaihengfeng)]

Mainline kernel is not supported:
https://wiki.ubuntu.com/Kernel/MainlineBuilds#Support_.28BEWARE:_there_is_none.29

[Jouni "rautamiekka" Järvinen (rautamiekka)]

Might as well delete a completely useless page then.

[David R. Bergstein (dbergst)]

Like many of you out there I totally disagree with the decision to mark this bug as invalid. I myself am running newer hardware that requires support from kernel 12.x and can now only use compatible kernels like the ones produced by tux-invader, not the ones on the mainline ppa.

[Doug Smythies (dsmythies)]

Within the context of this bug report it is valid, as it is specifically about the mainline PPA.

[DanglingPointer (ferncasado)]

What a joke! So how is the community going to contribute, help, and support the mainline kernels for "debug purposes" now?

Everyone on LTS that wants mainline will now be forced to compile their own kernels or arbitrarily trust PPAs of unofficial 3rd parties if choosing to stay with Ubuntu.

@David R Bergstein,
Your other alternative to 3rd party unofficial PPA's or compiling your own is ditching Ubuntu altogether and switching to rolling release distros like OpenSUSE Tumbleweed or Fedora.

[Martin Pecka (peci1)]

Ok, this gets emotional, so let's concentrate on what's important.

Is there anyone speaking for the official mainline builds? What problems would it bring to change the policy and build the mainline kernels with latest (or even oldest!) LTS tooling?

It is apparent that the community would benefit from it. It is understandable that Ubuntu doesn't want to "officially support" mainline kernels (just because they're a kind of testing stuff), but apparently there are people (disclosure: including myself) who need to run mainline kernels on older distros (Bionic in my case). It is completely fine to run into problems with a mainline kernel, but why add hurdles to those who want or need to try?

[MikeMecanic (xyz-t)]

This bug and there is no bug started in Mint 20.1 with Kernel 5.12-rc7 alongside the shim 1.46 multiple issues. Since then it works here secure boot enabled or not.

The bug comes with a broken header and it kills update manager.

All we do here to make it work is to delete the broken header in Synaptic, After, update manager runs normally. Same for the Kernel with 3 deb packages only. Give it a try, there is no side effect.

If you don't want me to test mainline Kernel in Mint (noise free), then you have to remove the second header.

We never give up no matter what the problem is, even with 3 legs.

[MikeMecanic (xyz-t)]

Don't underestimated our capabilities, a Debian user is like red wine, the more experience he gets, the better he is.

lsb_release -a && uname -r
No LSB modules are available.
Distributor ID: Linuxmint
Description: Linux Mint 20.1
Release: 20.1
Codename: ulyssa
5.13.0-051300rc4-generic
mokutil --sb
SecureBoot enabled
SecureBoot validation is disabled in shim

I don't have to get rid of the broken header, but update manager refuses to work. Same for Synaptic:

Commit Log for Tue Jun 1 10:32:54 2021

Completely removed the following packages:
linux-headers-5.13.0-051300rc4-generic

[Ryba (sepatel)]

I truly do not understand why this is even a debate. The simple fact is that either 20.04 is or is not an LTS release. If you are telling people that you are going to support it for 5 years, then that means being able to provide security updates to them as well as allowing them to use hardware that was created during the 5 years following April 2020 within reason. To do that, people must be able to update the kernel plain and simple.

You cannot simply tell people that they have to use experimental/unstable operating system for the next 2 years and/or live with unpatched zero-day security holes in their kernel.

And if you want a kernel version that is built on the latest toolchain that is also perfectly fine, but put those separately like how the armhf, ppc64el, etc versions are done. Have another build for the latest toolchain if that is needed for future work to take place but I can see that as being a problem longer term. So really make a proper ppa that has the separation of folders by distributions like how every single other ppa is setup.

This is a pretty serious epic failure on Ubuntu's end that is affecting a lot of distributions and users and will start pushing people away as the core fundementals of a kernel do not even work safely.

Also, huge huge huge thanks to tuxinvader for providing a PPA prevent us from having to look for alternative solutions.

[MikeMecanic (xyz-t)]

sudo apt update && sudo apt upgrade -y
The following packages will be upgraded:
  dnsmasq-base evolution-data-server evolution-data-server-common
  firefox-locale-en google-chrome-unstable libcamel-1.2-62 libdrm-amdgpu1
  libdrm-common libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libdrm2
  libebackend-1.2-10 libebook-1.2-20 libebook-contacts-1.2-3 libecal-2.0-1
  libedata-book-1.2-26 libedata-cal-2.0-1 libedataserver-1.2-24
  libedataserverui-1.2-2 libegl-mesa0 libgbm1 libgl1-mesa-dri libglapi-mesa
  libglx-mesa0 libxatracker2 mesa-va-drivers mesa-vdpau-drivers
  mesa-vulkan-drivers
29 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
exit

A debian tester has the abilities to bypass all sorts of barriers. At large, WE HAVE NO LIMIT AT FINDING.

The idea behind running mainline Kernel with 3 packages is very very simple, the ppa main page shows that ppc64e1 build and s390x build have only 3 packages. In order to complete our test, we downloaded 3 packages (daily) for amd64 build and compile them to see if it still work.

That way we wouldn't have to deal with the broken issue. It is the case without the first package.

Here's the result:

cd //home/Our_user_name/Downloads
sudo dpkg -i *.deb
Selecting previously unselected package linux-headers-5.13.0-051300rc4daily20210601.
(Reading database ... 302842 files and directories currently installed.)
Preparing to unpack linux-headers-5.13.0-051300rc4daily20210601_5.13.0-051300rc4daily20210601.202105312224_all.deb ...
Unpacking linux-headers-5.13.0-051300rc4daily20210601 (5.13.0-051300rc4daily20210601.202105312224) ...
Selecting previously unselected package linux-image-unsigned-5.13.0-051300rc4daily20210601-generic.
Preparing to unpack linux-image-unsigned-5.13.0-051300rc4daily20210601-generic_5.13.0-051300rc4daily20210601.202105312224_amd64.deb ...
Unpacking linux-image-unsigned-5.13.0-051300rc4daily20210601-generic (5.13.0-051300rc4daily20210601.202105312224) ...
Selecting previously unselected package linux-modules-5.13.0-051300rc4daily20210601-generic.
Preparing to unpack linux-modules-5.13.0-051300rc4daily20210601-generic_5.13.0-051300rc4daily20210601.202105312224_amd64.deb ...
Unpacking linux-modules-5.13.0-051300rc4daily20210601-generic (5.13.0-051300rc4daily20210601.202105312224) ...
Setting up linux-headers-5.13.0-051300rc4daily20210601 (5.13.0-051300rc4daily20210601.202105312224) ...
Setting up linux-image-unsigned-5.13.0-051300rc4daily20210601-generic (5.13.0-051300rc4daily20210601.202105312224) ...
I: /boot/vmlinuz.old is now a symlink to vmlinuz-5.13.0-051300rc4-generic
I: /boot/initrd.img.old is now a symlink to initrd.img-5.13.0-051300rc4-generic
I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-051300rc4daily20210601-generic
I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-051300rc4daily20210601-generic
Setting up linux-modules-5.13.0-051300rc4daily20210601-generic (5.13.0-051300rc4daily20210601.202105312224) ...
Processing triggers for linux-image-unsigned-5.13.0-051300rc4daily20210601-generic (5.13.0-051300rc4daily20210601.202105312224) ...
/etc/kernel/postinst.d/dkms:
 * dkms: running auto installation service for kernel 5.13.0-051300rc4daily20210601-gene...

[TuxInvader (tuxinvader)]

Hi all,

Controversial and TL;DR version - If one of the kernel devs can update the wiki page here https://wiki.ubuntu.com/Kernel/MainlineBuilds#Mainline_kernel_build_toolchain, then as far as I'm concerned they can close this bug as resolved.

The longer version.....

Ubuntu is great! Thank you Debian! Thank you Canonical!

In hindsight I probably shouldn't have linked the CVE in comment #33. I wasn't suggesting that Ubuntu was insecure in any way, I just wanted to highlight to those "stuck" on an old (unsupported) mainline kernel that they might have a problem, and need to make alternative arrangements.

Ubuntu 20.04 LTS will be supported for a long time however, and it will get regular kernel updates (approx every 6 months) through the HWE/LTS programme.

   See: https://wiki.ubuntu.com/Kernel/LTSEnablementStack

So, in a few months time focal will get a new kernel (5.11 I think) which will be fully supported. Yay! And 6 months after that it will get another new kernel. Double yay!

All official Ubuntu kernels get backported security fixes as bugs are found. The 5.4, 5.8, and soon 5.11 kernels will all be kept secure through security updates for the full life time of the LTS release. Ubuntu is not insecure!

This bug was opened because the documentation for mainline here https://wiki.ubuntu.com/Kernel/MainlineBuilds#Mainline_kernel_build_toolchain states that the kernels are built with the most recent LTS release.

Obviously that is wrong, the mainline kernels are built with the newest/development version of Ubuntu, and the wiki should be updated to reflect that.

Now, it would be nice if the mainline build systems were updated to build both the newest and the most recent LTS release, but that sometimes requires additional work. In fact - you can't build the mainline kernel on LTS as it stands, because newer kernels built with the ubuntu kconfig require a newer dwarves package than ships with LTS.

If you need to run a newer kernel than is in focal today, then you have several options, one of them is build your own (at least one set of instructions is in this thread somewhere), another is find a PPA (I'm sure one was mentioned in this thread somewhere too), do without the hardware until the next HWE release, or use a newer version of Ubuntu.

2c

[Thomas Anderson (furfix)]

Same issue with the latest stable kernel published today:

linux-headers-5.12.9-051209-generic : Depends: libc6 (>= 2.33) but 2.31-0ubuntu9.2 is installed

Ubuntu Server 20.04.2 LTS machine.

Please fix it. I don't feel comfortable installing 3rd party Kernels and I need to use mainline kernel to be able to use the iGPU on my 11gen Intel CPU for rendering.
Thanks in advance!!

[danmb (danmbox)]

5.10 the official kernel LTS release. If Ubuntu decides to long-term other releases, it's their choice, I'm sticking to 5.10. Making 5.10 even more difficult to install than it already is (going to the mainline "PPA", which isn't really a PPA, is already unfriendly), even if they document the fact on a wiki, is still a bug.

[Shatadru Banerjee (satadru-bag)]

So we will just let it go? Or, somebody from mainline kernel team is looking into the matter?
This is ridiculous, what is the point of having a 5 year LTS which can't support the newest kernels that are necessary for newer hardware? In simple words, one cannot use Ubuntu LTS with new hardware. One needs to use the latest ubuntu OS which might be plagued with bugs and many other issues, and furthermore keep changing it every 6 months? I am dying to hear any sensible argument in favour of this.

[elhoir (jfarroyo82)]

Hello folks,
Im also affected by this issue
In the meantime, you can try with this PPA:

https://launchpad.net/~tuxinvader/+archive/ubuntu/lts-mainline-longterm

[TuxInvader (tuxinvader)]

It's a shame that mainline don't build LTS compatible kernels anymore, but what can we do about it? It would be nice if they would build for ubuntu-next *and* previous LTS, but they don't.

So new kernels for any Ubuntu release relies on the HWE updates that come with the point releases. The next focal update (20.04.3) should back port the kernel from 21.04 (which is 5.11) and it should arrive in August. The previous HWE kernel was 5.8.

What I don't understand with the HWE packages, is that there is both a linux-generic-hwe-20.04 and a linux-generic-hwe-20.04-edge. It feels to me like `edge` should be more like a rolling release between the HWE releases that come with an LTS update, but they're not?? Currently the edge and non-edge versions are exactly the same.

$ apt-cache madison linux-generic-hwe-20.04-edge
linux-generic-hwe-20.04-edge | 5.8.0.55.62~20.04.39 | http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages

$ apt-cache madison linux-generic-hwe-20.04
linux-generic-hwe-20.04 | 5.8.0.55.62~20.04.39 | http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages

Strange.

[DanglingPointer (ferncasado)]

Thanks to this "bug", yes it IS a"bug"; thousands of ubuntu users have learnt to compile mainline kernels!

I never had to compile a linux kernel until this happened to 5.11.17+! I've always just installed whatever the mainline kernel team built.

The silver lining here is that my ivy-bridge servers are all now running way more efficient and faster with aggressive optimisations like what I posted previously on this thread! Thus delaying the need to replace them all for a few more years yet!

Recently (yesterday) I ditched the GNU build utils and gcc/g++ and switched to llvm-12, clang-12, and ld.lld-12 and enabled thin-LTO along with the same aggressive optimisations I posted previously in my KCFLAGS and KCPPFLAGS; and now it is even more snappier!

I would never have learnt how to do all of the above if it weren't for this "bug", and honestly don't see myself ever using the ubuntu mainline kernels ever again except for copying their ".config" file to prime my builds and ensure hardwares are enabled.

Thank you bug; all my servers are way faster now!

Now I itch to compile even more stuff that I use daily that's open source so I can aggressively optimise them and squeeze more juice out of my setup!

[dllud (dllud)]

DanglingPointer: you should take a look at Gentoo Linux, I guess you will like it.

Most others out here, including myself, are using Ubuntu because we cannot afford the time to configure and compile packages for all our machines (it gets crazy when you manage an heterogeneous cluster). Ubuntu Kernel Team, please revert and keep building with the previous LTS toolchain.

[MikeMecanic (xyz-t)]

Neon unstable switched last week (June 15th unstable ISO) to Kernel 5.8.0.55 default (Focal base). For mainline Kernel, this is the second month, that the debian packages boot properly.

Full cycle next Sunday.

All we do is not downloading the first header (2.4MB), the smallest one. No error and it runs normally.

From Synaptic > Status > Installed (local or obsolete)

linux-headers-5.13.0-051300rc7: 75.6 MB
linux-image-unsigned-5.13.0-051300rc7-generic: 10.3 MB
linux-modules-5.13.0-051300rc7-generic: 387 MB

All the best,

[Shatadru Banerjee (satadru-bag)]

The above does not work for me. Any update?
Is it worth compile my own kernel?
Or, time to switch to another distro?

[DanglingPointer (ferncasado)]

If you want to stay with Ubuntu 20.04 LTS with latest stable mainline, you'll need to build it.
Otherwise, try openSuse-Tumbleweed or Fedora.

I'm sticking with ubuntu so just building my own kernels. Runs faster now as well since it is optimised for my rig specifically now.

[TuxInvader (tuxinvader)]

It's becoming increasingly difficult to build your own kernels on focal due to the kernel dependencies on newer tool-chain elements. You now need dwarves 1.21 (or 1.20 with this patch https://bugs.launchpad.net/ubuntu/+source/dwarves-dfsg/+bug/1928244 ) to build a 5.13 kernel.

I have a focal based container for building mainline here: https://github.com/TuxInvader/focal-mainline-builder

The container includes the build dependencies for mainline kernels, and builds the latest dwarves and libbpf source packages from impish.

[k3dar7 (k3dar7)]

@DanglingPointer #55

"If you want to stay with Ubuntu 20.04 LTS with latest stable mainline, you'll need to build it."

not true... if not have external kernel modules (ex. using dkms builded module for VirtualBox, Nvidia, tlp, wifi driver for card not supported by mainline, etc..) then only not install problematic linux-headers-5.13.2-051302-generic package

in other case, is possible install it via "workaround"...

# only unpack without install/configure
dpkg --unpack linux-headers-5.13.2-051302-generic_5.13.2-051302.202107141437_amd64.deb

in /usr/src/linux-headers-5.13.2-051302-generic/ replace this files:
"scripts/basic/fixdep tools/objtool/fixdep tools/bpf/resolve_btfids/fixdep scripts/mod/modpost tools/objtool/objtool"
from: linux-headers-5.11.16-051116-generic_5.11.16-051116.202104211235_amd64.deb
(latest headers package version before switch to glibc 2.33)

# configure it with ignoring libc6 depends:
dpkg --configure --ignore-depends=libc6 linux-headers-5.13.2-051302-generic

# fake libc6 depend version in dpkg status file
sed "/Depends: linux-headers-5.13.2-051302, libc6/s/libc6 (>= 2...)/libc6 (>= 2.31)/" -i /var/lib/dpkg/status

i using this workaround with my own tool for installing "Ubuntu Mainline Kernel" on all my Xubuntu 20.04 machines and without problem from 5.11.17 to latest 5.13.2 ;-)

[TuxInvader (tuxinvader)]

FYI - This bug is for back-porting dwarves and libbpf to focal fixing support for newer kernels: https://bugs.launchpad.net/ubuntu/+source/dwarves-dfsg/+bug/1912811

It wont make mainline releases work on focal, but it will again be possible to compile a kernel using the focal toolchain.

[DanglingPointer (ferncasado)]

Awesome.

I've been building dwarves and libbpf using the c library from 20.04. I believe you are too Tuxinvader.

But yes, there are extra hoops to jump over having to do those! If they get it in then that would be a couple less things to boilerplate.

Would be good if they fixed DKMS as well to ensure it uses the same build tools that the kernel was built with! Currently it is defaulting to gcc and gnu build tools. So if you built your kernel with LLVM and Clang with the LLD linker along with -flto=thin, then many DKMS modules don't work like Virtualbox, Zram, etc because it doesn't know it needs to use LLVM-Clang-LLD and tries to use the same build options with gcc.

Frustrating cause more hoops to jump through if you want to LTO your kernel. So for now stuck with GCC.

[zolstarym (zolstarym)]

Hardware compatibility means that I need 5.11 and up, but this was keeping me from getting newer kernels with up to date security. I am kinda shocked that it is apparently being ignored by devs.

Thanks to tuxinvader, as builds on that ppa seem to work well with mint.

[Matias N. Goldberg (dark-sylinc)]

This bug is related to

https://bugs.launchpad.net/ubuntu/+source/dwarves-dfsg/+bug/1928244
https://bugs.launchpad.net/ubuntu/+source/dwarves-dfsg/+bug/1912811

Kernel 5.13 can be compiled in one of the two ways:

 1. Use dwarves 1.21, which needs Ubuntu 21.04 and breaks installation for older systems
 2. Disable CONFIG_DEBUG_INFO_BTF, thus allowing it to be compiled with older toolchains and thus, for older Ubuntu releases.

I believe the sensible approach would be to temporarily follow the second option until a better solution is found as proposed in 1912811.

[k3dar7 (k3dar7)]

@Matias N. Goldberg (dark-sylinc) #61

not sure ;-) as i write in #57, this way is installable on 20.04...

k3dar@t430s:/$ lsb_release -d
Description: Ubuntu 20.04.2 LTS

k3dar@t430s:/$ uname -a
Linux t430s 5.13.7-051307-generic #202107310732 SMP Sat Jul 31 07:53:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

k3dar@t430s:/$ grep CONFIG_DEBUG_INFO_BTF /boot/config-$(uname -r)
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y

[Shatadru Banerjee (satadru-bag)]

Can anyone please confirm the #57 works without any major problem? If yes, I request someone please write the instructions comprehensively.
Thanks.

[tlk (sarcasticskull)]

can confirm this (simpler) workaround works https://askubuntu.com/a/1335849
tested with 5.13.0

force-install the newer headers (edit out the libc6 2.33 dep in the dpkg status file)

Then just dpkg -x linux-headers-5.11.16-051116-generic_5.11.16 and replace fixdep and modpost

[k3dar7 (k3dar7)]

@tlk this is similar/same workaround, info posted in #57 is in principles:
1. install problematic package without configuring
2. replace binary files in unpacked package path
3. configure package (this also do dkms modules build)
4. fake libc depend version in status file

this (details in #57) method i use from >5.11.16 on several of my laptops, one of them my primary nb running >10 hour/day, with several times a day hibernated/resumed, many months without problem ;-)

[DanglingPointer (ferncasado)]

You guys mocking around trying to hack headers and wotnot, just build the bloody kernel and you'll have a full kernel without wondering if you broke something by hacking stuff! You could have built the kernel a 100x by now!

1) Just build dwarves from hirsute
2) build libbpf from hirsute
3) ensure to install both
4) build linux kernel

You can go through tuxinvader's scripts to see how it is done. It is simple to read through.

[tlk (sarcasticskull)]

@65
tbh I haven't analyzed what you've offered, but a quick glance over your post *and* knowing what *I've* done to get it working tells me that no, it's not the same. What I've done barely breaks into the usual workflow of installing the image/headers from mainline and building the out-of-tree modules against it. Maybe you're offering the same but it's hard to be positive about it from your post.

PS yeah it seems very similar now, but I didn't reconfigure nothing and only things I've replaced were fixdep and modpost

@66
Just build
why? If everything needed is already there. Basically just copy over those pesky fixdep and modpost from a compatable release and you're set.

[Jens Glathe (glathe)]

Since HWE in focal changed today (?) to 5.13.0-14, trying to get it to run I tried #66 DanglingPointer's approach (was on the same path trying to build from the hirsute repo on Mint 20.2 - success with 5.11.0-25 after building dwarves locally :)). libbpf was missing, installed from debian. zstd was missing, too, installed from debian. Aaaand... it works. Next up will be to build the impish repo on this machine.

Interesting build - detail: Running "time sudo dpkg-buildpackage -d -nc --no-sign -j33" is faster on the same machine as using fakeroot. With sudo the machine is maxed out, build times in the 20 minutes range on Ryzen 9-3950X with 170A EDC limit.

[DanglingPointer (ferncasado)]

Good stuff @Jens Glathe!

I recommend everyone stuck to just build it and they'll see it work perhaps even better than before, especially if you start natively optimising the build against your hardware. Your OS kernel will make your system feel and run snappier!

This is my build command post configuration (GCC)...
$ time KCFLAGS="-march=native -msse2avx -pipe -O3" KCPPFLAGS="-march=native -msse2avx -pipe -O3" make -j$(( $(nproc) + 1 )) deb-pkg LOCALVERSION=-danglingpointer-zen3-optimised

This is my build command post CLANG configuration with LTO...
time KCFLAGS="-march=native -pipe -O3" KCPPFLAGS="-march=native -pipe -O3" make -j$(( $(nproc) + 1 )) LLVM=1 LLVM_IAS=1 deb-pkg LOCALVERSION=-danglingpointer-zen3-clang-ltothin

For Clang LTO=thin you will need to configure for it.
1) prep the config using the config from Ubuntu Mainline. Basically just copy it and then $ make olddefconfig
2) scripts/config -e LTO_CLANG_THIN
3) run the clang make command above with KCFLAGS.

Check phoronix for GCC(all versions) vs CLANG-12

Only problem I have with CLANG is that it can bugger up any DKMS programs you may have. Virtualbox is an example which is geared towards GCC. I have CLANG built linux kernels running everywhere except wherever I have Virtualbox. There I use GCC-11.

[tlk (sarcasticskull)]

^^^^^ Recent gentoo convert?

"only problem" it breaks stuff, esp. those DKMS modules - the reason we need those headers to begin with. Noice.

Your OS kernel will make your system feel and run snappier! - fake news)
especially funny for someone on a Ryzen 9 lol.

and clang... meh.

[Jens Glathe (glathe)]

Was an interesing day, I was able to build and run mainline 5.13.12 on Mint 20.2. Was the first time I used the mainline package from kernel.org, and it was a bit fiddly, however copying over debian/ and debian.master/ is a great start, thanks for the tip. Only thing that doesn't work (yet) is nVidia 460.93 driver. This will be resolved eventually. Anyway, to build mainline on 20.04 it is sufficient to:

- build dwarves locally
- install the libbpf
- install zstd infrastructure (you might be able to configure this out of the kernel config)

No libc6 >=2.33 needed.

[DanglingPointer (ferncasado)]

@tlk,- Ignorance doesn't mean knowledge, it is quite the opposite...

DKMS doesn't break because of headers when using clang with lto. The headers are there and complete! Everything complete with well produced binaries from LLVM/Clang.

The problem is because programs (not the headers and not the kernel) dynamically re-compilling off-tree modules, e.g.: the guest additions of Virtualbox; uses GCC with the exact same build configuration as the Kernel. i.e. the LLVM/Clang with LTO build process; thus it uses compile options used with LLVM/CLang e.g. -flto=thin (there are others); that are not compatible with GCC with GCC! Then it fails obviously because GCC has no idea what to do with those options.

The Linux kernel, header, dbg, etc; all of it is perfectly made and approved by the Linux kernel team and forms part of the official kernel build documentation.
see here: https://www.kernel.org/doc/html/latest/kbuild/llvm.html

Not all DKMS enabled programs fail, and there's a way to fix the compile configuration, just that it is a lot of hoops to do and each build or distro environment upgrade could overwrite the config since DKMS is a distro responsibility and not Linus's when approving Linux versions. Distro upgrades don't overwrite the kernel.

As for Clang vs GCC...
https://www.phoronix.com/scan.php?page=article&item=clang12-gcc11-icelake&num=6

Also I have computers, servers, IoT devices all over the place and only 1 of them is zen3; most of them are ivybridge and the oldest is a penryn (core2 duo!)
There is no penryn architecture option in GCC and misses out on SSE4, SSE4.1
This is what GCC offers:
MMX, SSE, SSE2, SSE3 and SSSE3
https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html

Your statement is fake news. show me benchmarks where GCC built programs are faster than CLANG-12 otherwise you are spreading rubbish and telling people to use linux with missing or broken headers. I can only think of one where GCC beat Clang-12 and that's Python 3.9.6.

[DanglingPointer (ferncasado)]

For penryn the -march option is "core2" in that GCC options link.

[DanglingPointer (ferncasado)]

@Jens Glathe - Good stuff!

I wrote down some high-level steps in #32.

the mainline kernels from kernel.org have a lot of stuff disabled which will make things "fiddly". Easiest way to enable them for all the new hardware is to download the same mainline ubuntu kernel version even if it is broken. You just want the config in /boot.

copy that config for your version from the broken ubuntu mainline kernel /boot/ to your build path for the kernel with the same version.

e.g. here's the last one I did...
cp -v /boot/config-5.13.11-051311-generic ./linux-5.13.11/.config

where ./linux-5.13.11/ in the above example is the extracted folder from the tarball from kernel.org.

Then once you've got the config, you can delete the broken ubuntu mainline kernel.

With the ./linux-5.13.11/.config preconfigured. You may need to remove the debian certificates and configure for zstd. I don't use zstd and always disable it and change to xz since I have a lot of IoT devices with tiny storages and I get more breathing space with XZ. For an idea on how to do all these see #32 above

after all the config run
$ make olddefconfig

then you can then build.

For GCC-11 on my ivybridges it takes ~50mins
on my zen3 it takes 15mins.
on a skylake pentium it takes about 4 hours!

[tlk (sarcasticskull)]

@72
opening salvo of " Ignorance doesn't mean knowledge, it is quite the opposite..."
...aaaand then completely misreading a simple line of text.
k.

where have I said it's something to do w/ the headerz??? I've only meant that the whole frigging exercise is to get the modules to work on my site while you offer a roundabout way (complete with the obligatory 0.05% increase in gowd knows what) and then admitting it'll break the darn modules...

the rest of the stuff is... I'd best leave it be))

[nvrmndr (nvrmndr)]

So this is still a problem with kernel 5.14 without a working straightforward solution. This is becoming frustrating since this issue is now 4 months old.

@TuxInvader: I tried your repository (ppa:tuxinvader/lts-mainline) with 5.14, but something went very wrong - after booting with this kernel, my PC was basically unresponsive, it took like 2 minutes for the terminal to open.

[k3dar7 (k3dar7)]

@nvrmndr #76

steps in #57 still work for 5.14 ;-)

anyway i did make some changes in my private tool, so can be published and is here:
https://github.com/k3dar/kumk

you can simply install 5.14 via: sudo kumk install 5.14

[TuxInvader (tuxinvader)]

@nvrmndr - Works for me :-)
Please open a bug on my github project if you want help debugging it.

[Hash Ratez (hashratez)]

Dear TuxInvader...

I have no idea why you do what you do but THANK YOU. I HAVE BEEN GOING NUTS trying to get past the kernel bug for Ubuntu upgrades. WTF, they call 20.04 an LTS but you cannot keep up with kernel security patches. ARE YOU KIDDING?

YOU SAVED ME.

Hours and hours of install reboot, uninstall. Nothing worked (please note I am a total Linux admin hack, thus really can't bake my own kernels). I found this link, installed it and BAM, now running 5.10.61 etc. Just wanted to keep the LTS 5.10 patched and you can forget Ubuntu for not keeping up.

REF: https://launchpad.net/~tuxinvader/+archive/ubuntu/lts-mainline-longterm

ONE COMMAND FIXES EVERTYHING (after you add TuxInvader PPA): sudo apt-get install linux-generic-5.10

That will upgrade you to the latest 5.10.x build. It is that easy.

THANK YOU. Really...

[Ozgur Arslan (blueorange589)]

I suppose I found a super simple solution by chance. Just ran

sudo apt update && sudo apt upgrade -y

There were 3 repositories returning errors in my case. Such as,
E: The repository 'cdrom://Ubuntu 20.04 LTS _Focal Fossa_ - Release amd64 (20200423) focal Release' does not have a Release file.

Just launched software updater, removed PPAs with errors, now I'm able to install new software without getting

Depends: libc6 (>= 2.33) but 2.31-0ubuntu9.2 is to be installed

error.

[k3dar7 (k3dar7)]

@hashratez
right/official command (without adding any 3rd repository) is:
sudo apt install --install-recommends linux-generic-hwe-20.04
on officialy supported HW is done automatically, on other you must manualy run for upgrade to latest HWE line, more info: https://wiki.ubuntu.com/Kernel/LTSEnablementStack

@blueorange589
yes, about it is whole this bug report, newer that 5.11.15 "Ubuntu Mainline Kernel Builds" need libc >=2.33 for installing all packages... if you not need adding 3rd modules/drivers (nvidia, virtualbox, some unsupported wifi/lan card,...) then simple not install problematic one package: "linux-headers-....._amd64.deb" and error is gone...

[Jags Desai (jagsdesai)]

@tuxinvader

While trying to install Kernel 5.14.3 (Sep 12, 2021) on Ubuntu MATE 21.04, I ran into similar issue but for libc6 2.34.

Error:

dpkg: dependency problems prevent configuration of linux-headers-5.14.3-051403-generic: linux-headers-5.14.3-051403-generic depends on libc6 (>= 2.34); however: version of libc6:amd64 on system is 2.33-0ubuntu5.

[TuxInvader (tuxinvader)]

Hi @jagsdesai

I think you're using the package from mainline. I publish a PPA for 20.04 LTS, and version 5.14.3 has just finished building on launchpad this morning, so any package you installed yesterday was nothing to do with me ;-)

Here's my dependencies....

$ dpkg -I linux-headers-5.14.3-051403-generic_5.14.3-051403.202109130749_amd64.deb
 ...
 Depends: linux-headers-5.14.3-051403, libc6 (>= 2.22), libelf1 (>= 0.142), libssl1.1 (>= 1.1.0), zlib1g (>= 1:1.2.3.3)

[k3dar7 (k3dar7)]

@jagsdesai
both #81 #57 still work with 5.14.3, installed yesterday on Xubuntu 20.04 ;-)

[Jags Desai (jagsdesai)]

@tuxinvader @k3dar7 many thanks for the replies.

Yes, I installed packages from mainline. Sorry, I thought this bug report is for mainline packages.

Now I did see your PPA through AskUbuntu thread but is it just for "Focal" or from your PPA, I can install kernels in Hirsute too?

[chang long (2021yunotcompile)]

@k3dar7 i tried using your tool however it keeps saying command not found.

steps:

1. dl zip
2. extract
3. cd to folder (kumk-main)
4. sudo kumk install 5.14.4

no dice. am I doing something really dumb?

[k3dar7 (k3dar7)]

@2021yunotcompile

you missed:
3b. sudo cp -a kumk /usr/local/bin/

or without "installing"
4. sudo ./kumk install 5.14.4

or completely different way for "installing"
1. sudo wget https://raw.githubusercontent.com/k3dar/kumk/main/kumk -O /usr/local/bin/kumk
2. sudo chmod a+x /usr/local/bin/kumk
3. sudo kumk install 5.14.4

[chang long (2021yunotcompile)]

@k3dar7

i did this option -> 4. sudo ./kumk install 5.14.4

this is popping up now:

$ dpkg: dependency problems prevent configuration of linux-headers-5.14.4-051404-generic:
 linux-headers-5.14.4-051404-generic depends on libc6 (>= 2.34); however:
  Version of libc6:amd64 on system is 2.33-0ubuntu5.

@tuxinvader im running 21.04 so i guess they're using 21.10 to build now?

[k3dar7 (k3dar7)]

@2021yunotcompile
kumk has been used workaround way only on system with libc6 <2.33...
but i changed this in just updated kumk version,
now is used if kernel package "need" higher libc6 than is installed...

[Mark Rijckenberg (markrijckenberg)]

@k3dar7: Thanks a lot for your valuable help!

Following script works for me on Ubuntu 21.04 (64-bit):

sudo wget https://raw.githubusercontent.com/k3dar/kumk/main/kumk -O /usr/local/bin/kumk
sudo chmod a+x /usr/local/bin/kumk
sudo kumk install latest

[chang long (2021yunotcompile)]

@DanglingPointer (ferncasado)

gonna try #74. i really do want to understand how all this works. this might be the most reliable option since even ppa's break.

[Jens Glathe (glathe)]

Hmm. Got 5.14.9 running directly from the https://kernel.ubuntu.com/~kernel-ppa/mainline/ repository. Works with nvidia driver 4.70 on Mint 20.2 :)

build sequence:

- checkout cod/mainline/v5.14.9 from git://git.launchpad.net/~ubuntu-kernel-test/ubuntu/+source/linux/+git/mainline-crack
- install libfuse-dev (shown in the selftest fail log, so I guessed it can't hurt)
- time sudo debian/rules clean
- time sudo dpkg-buildpackage -d -nc --no-sign -j33
- sudo chown -R <user>:<user> ..

Prerequisites is having built dwarves, libbpf, and probably zstd like mentioned in #71. Package runs successfully on several machines.

One oddity I've seen: running the build again will fail. You need to git clean -df multiple times, and do a git reset --hard . But pretty straight forward anyway.

[Peter J. Mello (roguescholar)]

I've updated the Wiki page with my best approximation of the consensus which developed here. If it meets with the satisfaction of all concerned parties, I believe we can close this bug.

https://wiki.ubuntu.com/Kernel/MainlineBuilds#Mainline_kernel_build_toolchain

[Francisco Pombal (pombal.francisco)]

> I believe we can close this bug.

The change to the wiki is welcome, since it now reflects the current situation.

However, in my opinion, the real problem uncovered here has not been addressed:

Ubuntu should provide mainline kernels built with the latest LTS toolchain alongside the current one that uses bleeding edge toolchains, so that using/testing more recent kernels becomes more accessible to a wider audience (i.e., the vast majority who does not compile their kernels from source).

Historically, Ubuntu has had a major barrier to adoption: new users who buy their hardware in between LTS cycles will usually have a very sub-par experience with Ubuntu because of the outdated kernel. Their choice is to either:

- use the latest intermediate release, which may still not provide a recent enough kernel and is a bad choice for a first time user anyway
- use the latest LTS which may not be usable at all on their systems with the very outdated kernel (this tends to happen a lot with laptops in particular).

With all due respect to the user who put up https://launchpad.net/~tuxinvader/+archive/ubuntu/kernel-build-tools and their efforts in doing so, there should be an official PPA/infrastructure in place for this. We all know users installing lots of PPAs without thinking is a huge risk and a practice that is advised against.
But then you can't just leave the user with only one remaining, untenable choice, which is for their system to be unusable, in the name of security concerns that they don't quite understand.
This is a common criticism of Ubuntu: it effectively _forces_ users into installing a whole bunch of random PPAs willy-nilly to get certain basic features.

Until something is done about this, "LTS" releases are actually unusable (or usable with many jarring bugs) for many users. The HWE releases are better than nothing, but not nearly enough.

A post above put it very nicely:

> If you are telling people that you are going to support it for 5 years, then that means being able to provide security updates to them as well as allowing them to use hardware that was created during the 5 years following April 2020 within reason. To do that, people must be able to update the kernel plain and simple.

[k3dar7 (k3dar7)]

is there really any reason to add in 5.15.7 dependence on libssl3 package, when it is only in unreleased 22.04??

[k3dar7 (k3dar7)]

@roguescholar
is "nice" change info on wiki, but really, why is not supported last LTS release? need as mainly because st*pid blocking hibernation with enabled secure boot in stock kernel...

[Jens Glathe (glathe)]

@k3dar7 interesting, had no error yesterday on building 5.15.7 Need to investigate.

[Jens Glathe (glathe)]

@k3dar7 now... I did some search on what installs libssl3.

$ ldconfig -p | grep ssl3
 libssl3.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libssl3.so

$ apt-file search libssl3.so
firefox: /usr/lib/firefox/libssl3.so
libnss3: /usr/lib/x86_64-linux-gnu/libssl3.so
thunderbird: /usr/lib/thunderbird/libssl3.so
thunderbird-dbg: /usr/lib/debug/usr/lib/thunderbird/libssl3.so

So in my case, it appears to be libnss3.

I could build and deploy 5.15.7 without issues, though.

[k3dar7 (k3dar7)]

@Jens Glathe
problem is official Ubuntu Mainline Kernel from 5.15.7 change depends package from libssl1 to libssl3...

Package: linux-headers-5.15.6-051506-generic
Depends: linux-headers-5.15.6-051506, libc6 (>= 2.34), libelf1 (>= 0.142), libssl1.1 (>= 1.1.0), zlib1g (>= 1:1.2.3.3)

Package: linux-headers-5.15.7-051507-generic
Depends: linux-headers-5.15.7-051507, libc6 (>= 2.34), libelf1 (>= 0.142), libssl3 (>= 3.0.0~~alpha1), zlib1g (>= 1:1.2.3.3)

i don't know if this step is really need, or "only" f*cking all user of all *buntu releases except non-released 22.04 where libssl3 packages is... ;-)

sure not understand why official "Ubuntu Mainline Kernel" stop supporting last LTS release...

anyway my before here presented tool "kumk" updated to workarounding also this... ;-)