Launchpad builders kernel 4.15 too old to compile software that uses uring

[Otto Kekäläinen]

While working on Mariadb 10.6 for Debian and Ubuntu upload (I maintain MariaDB in Debian and Ubuntu) I noticed the Launchpad builders were failing due to mariadbd emitting:

    io_uring_queue_init() failed with ENOSYS: try uprading the kernel

The root cause is that the Launchpad buildd seems to be running on Ubuntu 18.04 with a 4.15 kernel:

    Kernel version: Linux bos02-s390x-002 4.15.0-154-generic #161-Ubuntu SMP Fri Jul 30 12:59:37

To be able to develop for newer Ubuntu releases (upcoming Impish mainly) the builders should be running at least Ubuntu 20.04 as it has a 5.4 Linux kernel.

What are your plans to upgrade the builder fleet?

Is there some way to opt-in to have Ubuntu 20.04 builder hosts earlier for https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.6/+builds?build_text=&build_state=all ?

[Otto Kekäläinen (otto)]

I will soon upload MariaDB 10.6 with liburing support to Debian unstable. Eventually that will lead to MariaDB tests in Ubuntu failing if Launchpad keep using only old kernels..

[Colin Watson (cjwatson)]

It should be possible to switch builder VMs to focal at this point (riscv64 builders run on focal, so the basic machinery works), but it will take some time and I'm not sure we're going to get to it before the end of the year.

[Seth Arnold (seth-arnold)]

Just some notes from the sidelines: back in the day, it was common for build systems to tailor an executable for the host that built it, rather than the host that would run it. This caused no end of trouble.

Most software these days feels better prepared to compile on a kernel or processor of one vintage and execute on a kernel or processor of another vintage and still pick the best available capabilities of the target system. I'd hate to see this regress.

It may feel like more work but I believe in the long run it's less work overall to decouple the build environment from the execution environment.

Thanks

[Otto Kekäläinen (otto)]

Related upstream issue: https://jira.mariadb.org/browse/MDEV-26569

To my knowledge it is not possible to build against io_uring and then run it against a kernel that is too old. So for the moment the decision is to either ship with io_uring (which Ubuntu and Debian support everywhere apart from Launchpad) or to not ship it (which would be a degraded experience for users).

[Colin Watson (cjwatson)]

This is now also tracked in https://warthogs.atlassian.net/browse/LP-559 (Canonical only).

[Colin Watson (cjwatson)]

One obstacle I found is that we need to port from bzr-builder to brz-builder in order to be able to build VM images for focal, since that's used by source package recipe builds. I've prepared suitable packaging in https://code.launchpad.net/~launchpad/+recipe/brz-builder-daily (it all gets a bit meta), though launchpad-buildd will also need a small change to make use of this.

[Otto Kekäläinen (otto)]

Tried to build MariaDB 10.6 today in Feb 2022 at https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.6/+builds?build_text=&build_state=all and seems the Launchpad builders still run kernel 4.15 and this bug/feature request is still open.

[Colin Watson (cjwatson)]

We needed to decommission our powerpc builders first, because the logistics of running builders on both xenial and focal were going to get a bit much (we had a hack to make it work for riscv64, but it was a pretty awkward and confusing one). That was completed last week.

The next obstacle is getting launchpad-buildd's test suite working on focal, which it seems we'd neglected to do up to now and there are a few small problems. Once that's fixed I expect we should be able to upgrade the whole build farm to focal fairly quickly; I ran some small-scale tests of this a month or two ago and things seemed mostly fine.

[Colin Watson (cjwatson)]

I've got the test suite working on focal now and upgraded the dogfood build farm, but discovered a regression affecting build types that use LXD. I'll have another go once https://code.launchpad.net/~cjwatson/launchpad-buildd/+git/launchpad-buildd/+merge/415846 lands.

[Colin Watson (cjwatson)]

I've filed https://portal.admin.canonical.com/C136114 (internal-only) to deploy a new launchpad-buildd version that fixes various issues on focal, mainly getting tests to work but also fixing a missing dependency that broke things like snap builds. Once that's done, we can build focal-based VM images, then switch resets over to dispatching them.

[Colin Watson (cjwatson)]

Launchpad builders are now running a focal base image with Linux 5.4. mariadb-10.6 still seems to fail to build on jammy with:

CMake Error at cmake/ssl.cmake:157 (MESSAGE):
  Cannot find appropriate system libraries for SSL. Use WITH_SSL=bundled to
  enable SSL support

However, that seems like a separate issue - I gather it's https://bugs.debian.org/1005950.

I retried impish builds in https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.6/+packages, and they seem to be succeeding, so I think that's all we need to close this bug.

[Otto Kekäläinen (otto)]

Thanks Colin! The Launchpad builds that were failing on this uring issue masked the fact that builds started to fail for another reason when OpenSSL 3.0 entered Ubuntu.

I do however already have a fix for Debian#1005950 in https://salsa.debian.org/mariadb-team/mariadb-server/-/commits/dev-otto and new test builds will appear at https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.6/+builds?build_text=&build_state=all in a couple of hours.

[Otto Kekäläinen (otto)]

Confirming that builds at https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.6/+builds?build_text=&build_state=all work again and this issue is fixed.

[Otto Kekäläinen (otto)]

Now repo ran out of disk space.. If you Colin have some magic powers to expedite https://answers.launchpad.net/launchpad/+question/700724 I would appreciate it.