| 2022-04-27 15:23:13 |
Andreas Hasenack |
bug |
|
|
added bug |
| 2022-04-27 15:23:20 |
Andreas Hasenack |
tags |
|
server-todo |
|
| 2022-04-27 15:23:28 |
Andreas Hasenack |
bug |
|
|
added subscriber Ubuntu Server |
| 2022-04-27 15:23:34 |
Andreas Hasenack |
bug |
|
|
added subscriber Canonical Server Team |
| 2022-04-28 18:35:34 |
Andreas Hasenack |
bug task added |
|
systemd (Ubuntu) |
|
| 2022-04-28 18:45:15 |
Andreas Hasenack |
summary |
FTBFS: test failure due to low memlock limit |
FTBFS: mariadb fails to start due to low MEMLOCK limit |
|
| 2022-04-28 22:25:00 |
Daniel Black |
bug watch added |
|
https://github.com/axboe/liburing/issues/246 |
|
| 2022-04-29 14:19:11 |
Launchpad Janitor |
systemd (Ubuntu): status |
New |
Confirmed |
|
| 2022-04-29 14:44:58 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~ahasenack/ubuntu/+source/mariadb-10.6/+git/mariadb-10.6/+merge/420845 |
|
| 2022-04-29 20:51:55 |
Dan Streetman |
bug |
|
|
added subscriber Dan Streetman |
| 2022-04-30 12:45:25 |
Launchpad Janitor |
mariadb-10.6 (Ubuntu): status |
In Progress |
Fix Released |
|
| 2022-05-18 15:13:34 |
Christian Ehrhardt |
removed subscriber Ubuntu Server |
|
|
|
| 2022-05-22 20:51:27 |
Dan Streetman |
removed subscriber Dan Streetman |
|
|
|
| 2022-06-01 14:52:34 |
Dr. Jens Harbott |
bug |
|
|
added subscriber Dr. Jens Harbott |
| 2022-06-15 15:15:16 |
Andreas Hasenack |
nominated for series |
|
Ubuntu Jammy |
|
| 2022-06-15 15:15:16 |
Andreas Hasenack |
bug task added |
|
systemd (Ubuntu Jammy) |
|
| 2022-06-15 15:15:16 |
Andreas Hasenack |
bug task added |
|
mariadb-10.6 (Ubuntu Jammy) |
|
| 2022-06-15 15:15:27 |
Andreas Hasenack |
mariadb-10.6 (Ubuntu Jammy): assignee |
|
Andreas Hasenack (ahasenack) |
|
| 2022-06-15 15:15:30 |
Andreas Hasenack |
mariadb-10.6 (Ubuntu Jammy): status |
New |
Triaged |
|
| 2022-06-15 15:15:35 |
Andreas Hasenack |
mariadb-10.6 (Ubuntu Jammy): importance |
Undecided |
High |
|
| 2022-06-17 13:04:22 |
Andreas Hasenack |
systemd (Ubuntu Jammy): assignee |
|
Andreas Hasenack (ahasenack) |
|
| 2022-06-17 13:04:24 |
Andreas Hasenack |
systemd (Ubuntu Jammy): status |
New |
In Progress |
|
| 2022-06-17 13:04:27 |
Andreas Hasenack |
systemd (Ubuntu Jammy): importance |
Undecided |
High |
|
| 2022-06-17 18:35:22 |
Andreas Hasenack |
description |
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
[Impact]
* An explanation of the effects of the bug on users and
* justification for backporting the fix to the stable release.
* In addition, it is helpful, but not required, to include an
explanation of how the upload fixes this bug.
[Test Plan]
* detailed instructions how to reproduce the bug
* these should allow someone who is not familiar with the affected
package to reproduce the bug and verify that the updated package fixes
the problem.
* if other testing is appropriate to perform before landing this update,
this should also be described here.
[Where problems could occur]
* Think about what the upload changes in the software. Imagine the change is
wrong or breaks something else: how would this show up?
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This must '''never''' be "None" or "Low", or entirely an argument as to why
your upload is low risk.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
|
| 2022-06-17 18:42:40 |
Andreas Hasenack |
description |
[Impact]
* An explanation of the effects of the bug on users and
* justification for backporting the fix to the stable release.
* In addition, it is helpful, but not required, to include an
explanation of how the upload fixes this bug.
[Test Plan]
* detailed instructions how to reproduce the bug
* these should allow someone who is not familiar with the affected
package to reproduce the bug and verify that the updated package fixes
the problem.
* if other testing is appropriate to perform before landing this update,
this should also be described here.
[Where problems could occur]
* Think about what the upload changes in the software. Imagine the change is
wrong or breaks something else: how would this show up?
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This must '''never''' be "None" or "Low", or entirely an argument as to why
your upload is low risk.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
[Impact]
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there.
Common scenarios where this combination exists is a focal host, running a jammy container (lxd or docker), or even a chroot (the launchpad builders).
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container, or a root-less chroot, this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
* detailed instructions how to reproduce the bug
* these should allow someone who is not familiar with the affected
package to reproduce the bug and verify that the updated package fixes
the problem.
* if other testing is appropriate to perform before landing this update,
this should also be described here.
[Where problems could occur]
* Think about what the upload changes in the software. Imagine the change is
wrong or breaks something else: how would this show up?
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This must '''never''' be "None" or "Low", or entirely an argument as to why
your upload is low risk.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
|
| 2022-06-17 18:45:50 |
Andreas Hasenack |
description |
[Impact]
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there.
Common scenarios where this combination exists is a focal host, running a jammy container (lxd or docker), or even a chroot (the launchpad builders).
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container, or a root-less chroot, this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
* detailed instructions how to reproduce the bug
* these should allow someone who is not familiar with the affected
package to reproduce the bug and verify that the updated package fixes
the problem.
* if other testing is appropriate to perform before landing this update,
this should also be described here.
[Where problems could occur]
* Think about what the upload changes in the software. Imagine the change is
wrong or breaks something else: how would this show up?
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This must '''never''' be "None" or "Low", or entirely an argument as to why
your upload is low risk.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
* detailed instructions how to reproduce the bug
* these should allow someone who is not familiar with the affected
package to reproduce the bug and verify that the updated package fixes
the problem.
* if other testing is appropriate to perform before landing this update,
this should also be described here.
[Where problems could occur]
* Think about what the upload changes in the software. Imagine the change is
wrong or breaks something else: how would this show up?
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This must '''never''' be "None" or "Low", or entirely an argument as to why
your upload is low risk.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
|
| 2022-06-17 18:48:20 |
Andreas Hasenack |
description |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
* detailed instructions how to reproduce the bug
* these should allow someone who is not familiar with the affected
package to reproduce the bug and verify that the updated package fixes
the problem.
* if other testing is appropriate to perform before landing this update,
this should also be described here.
[Where problems could occur]
* Think about what the upload changes in the software. Imagine the change is
wrong or breaks something else: how would this show up?
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This must '''never''' be "None" or "Low", or entirely an argument as to why
your upload is low risk.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
The test plan is to launch mariadb in a jammy lxd container running on a focal host.
lxc launch ubuntu:focal f --vm
lxc shell
lxd init # just hit enter for all questions
lxc launch ubuntu:jammy j
lxc shell j
ulimit -l # confirm it's less than 256
apt update && apt install mariadb-server -y
After the installation, mariadb will not be running, and this can be checked with:
systemctl status mariadb-server
or
journal -u mariadb.server
You will see something like this:
Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ;
And a crash dump.
With the fixed version, the service will be running normally after installation.
[Where problems could occur]
* Think about what the upload changes in the software. Imagine the change is
wrong or breaks something else: how would this show up?
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This must '''never''' be "None" or "Low", or entirely an argument as to why
your upload is low risk.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
|
| 2022-06-17 18:49:40 |
Andreas Hasenack |
description |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
The test plan is to launch mariadb in a jammy lxd container running on a focal host.
lxc launch ubuntu:focal f --vm
lxc shell
lxd init # just hit enter for all questions
lxc launch ubuntu:jammy j
lxc shell j
ulimit -l # confirm it's less than 256
apt update && apt install mariadb-server -y
After the installation, mariadb will not be running, and this can be checked with:
systemctl status mariadb-server
or
journal -u mariadb.server
You will see something like this:
Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ;
And a crash dump.
With the fixed version, the service will be running normally after installation.
[Where problems could occur]
* Think about what the upload changes in the software. Imagine the change is
wrong or breaks something else: how would this show up?
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This must '''never''' be "None" or "Low", or entirely an argument as to why
your upload is low risk.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
The test plan is to launch mariadb in a jammy lxd container running on a focal host.
lxc launch ubuntu:focal f --vm
lxc shell
lxd init # just hit enter for all questions
lxc launch ubuntu:jammy j
lxc shell j
ulimit -l # confirm it's less than 256
apt update && apt install mariadb-server -y
After the installation, mariadb will not be running, and this can be checked with:
systemctl status mariadb-server
or
journalctl -u mariadb.server
You will see something like this:
Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ;
And a crash dump.
With the fixed version, the service will be running normally after installation.
[Where problems could occur]
* Think about what the upload changes in the software. Imagine the change is
wrong or breaks something else: how would this show up?
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This must '''never''' be "None" or "Low", or entirely an argument as to why
your upload is low risk.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
|
| 2022-06-17 19:08:56 |
Andreas Hasenack |
description |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
The test plan is to launch mariadb in a jammy lxd container running on a focal host.
lxc launch ubuntu:focal f --vm
lxc shell
lxd init # just hit enter for all questions
lxc launch ubuntu:jammy j
lxc shell j
ulimit -l # confirm it's less than 256
apt update && apt install mariadb-server -y
After the installation, mariadb will not be running, and this can be checked with:
systemctl status mariadb-server
or
journalctl -u mariadb.server
You will see something like this:
Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ;
And a crash dump.
With the fixed version, the service will be running normally after installation.
[Where problems could occur]
* Think about what the upload changes in the software. Imagine the change is
wrong or breaks something else: how would this show up?
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This must '''never''' be "None" or "Low", or entirely an argument as to why
your upload is low risk.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
The test plan is to launch mariadb in a jammy lxd container running on a focal host.
lxc launch ubuntu:focal f --vm
lxc shell
lxd init # just hit enter for all questions
lxc launch ubuntu:jammy j
lxc shell j
ulimit -l # confirm it's less than 256
apt update && apt install mariadb-server -y
After the installation, mariadb will not be running, and this can be checked with:
systemctl status mariadb-server
or
journalctl -u mariadb.server
You will see something like this:
Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ;
And a crash dump.
With the fixed version, the service will be running normally after installation.
[Where problems could occur]
The proposed fix is not a surgical strike. It's unfortunate that we didn't get to the bottom of why LTO is causing this behavior. Reverting it is still the quickest and less risky change at the moment, though. This gets us on par with upstream binary builds, and debian builds, and these also have wide test coverage and ample user base.
The other regression possibility is that this is a rebuild of mariadb in the current jammy environment, and the package that is currently in jammy was built on March 10th, 2022. Most likely the reverse dependencies were updated in jammy since then.
It's unclear how 10.6.7-2ubuntu1 migrated in jammy. I checked build logs and dep8 logs, and can't tell why the tests passed. At least the build log in jammy shows the host kernel was 5.4.x, so it should have been affected. My only explanation is that at that time, the MEMLOCK limit was higher in that environment for some reason, and didn't trigger this bug. Then at some point later, launchpad builders changed, and MEMLOCK was reduced to 64Mb. https://irclogs.ubuntu.com/2022/04/28/%23ubuntu-devel.html#t15:00 has some discussion about it.
[Other Info]
Not at this time.
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
|
| 2022-06-17 19:15:33 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~ahasenack/ubuntu/+source/mariadb-10.6/+git/mariadb-10.6/+merge/424986 |
|
| 2022-06-17 20:54:01 |
Andreas Hasenack |
systemd (Ubuntu Jammy): status |
In Progress |
New |
|
| 2022-06-17 20:54:04 |
Andreas Hasenack |
systemd (Ubuntu Jammy): assignee |
Andreas Hasenack (ahasenack) |
|
|
| 2022-06-17 20:54:07 |
Andreas Hasenack |
mariadb-10.6 (Ubuntu Jammy): status |
Triaged |
In Progress |
|
| 2022-06-17 20:54:31 |
Andreas Hasenack |
description |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
The test plan is to launch mariadb in a jammy lxd container running on a focal host.
lxc launch ubuntu:focal f --vm
lxc shell
lxd init # just hit enter for all questions
lxc launch ubuntu:jammy j
lxc shell j
ulimit -l # confirm it's less than 256
apt update && apt install mariadb-server -y
After the installation, mariadb will not be running, and this can be checked with:
systemctl status mariadb-server
or
journalctl -u mariadb.server
You will see something like this:
Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ;
And a crash dump.
With the fixed version, the service will be running normally after installation.
[Where problems could occur]
The proposed fix is not a surgical strike. It's unfortunate that we didn't get to the bottom of why LTO is causing this behavior. Reverting it is still the quickest and less risky change at the moment, though. This gets us on par with upstream binary builds, and debian builds, and these also have wide test coverage and ample user base.
The other regression possibility is that this is a rebuild of mariadb in the current jammy environment, and the package that is currently in jammy was built on March 10th, 2022. Most likely the reverse dependencies were updated in jammy since then.
It's unclear how 10.6.7-2ubuntu1 migrated in jammy. I checked build logs and dep8 logs, and can't tell why the tests passed. At least the build log in jammy shows the host kernel was 5.4.x, so it should have been affected. My only explanation is that at that time, the MEMLOCK limit was higher in that environment for some reason, and didn't trigger this bug. Then at some point later, launchpad builders changed, and MEMLOCK was reduced to 64Mb. https://irclogs.ubuntu.com/2022/04/28/%23ubuntu-devel.html#t15:00 has some discussion about it.
[Other Info]
Not at this time.
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
The test plan is to launch mariadb in a jammy lxd container running on a focal host.
lxc launch ubuntu:focal f --vm
lxc shell
lxd init # just hit enter for all questions
lxc launch ubuntu:jammy j
lxc shell j
ulimit -l # confirm it's less than 256
apt update && apt install mariadb-server -y
After the installation, mariadb will not be running, and this can be checked with:
systemctl status mariadb.service
or
journalctl -u mariadb.server
You will see something like this:
Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ;
And a crash dump.
With the fixed version, the service will be running normally after installation.
[Where problems could occur]
The proposed fix is not a surgical strike. It's unfortunate that we didn't get to the bottom of why LTO is causing this behavior. Reverting it is still the quickest and less risky change at the moment, though. This gets us on par with upstream binary builds, and debian builds, and these also have wide test coverage and ample user base.
The other regression possibility is that this is a rebuild of mariadb in the current jammy environment, and the package that is currently in jammy was built on March 10th, 2022. Most likely the reverse dependencies were updated in jammy since then.
It's unclear how 10.6.7-2ubuntu1 migrated in jammy. I checked build logs and dep8 logs, and can't tell why the tests passed. At least the build log in jammy shows the host kernel was 5.4.x, so it should have been affected. My only explanation is that at that time, the MEMLOCK limit was higher in that environment for some reason, and didn't trigger this bug. Then at some point later, launchpad builders changed, and MEMLOCK was reduced to 64Mb. https://irclogs.ubuntu.com/2022/04/28/%23ubuntu-devel.html#t15:00 has some discussion about it.
[Other Info]
Not at this time.
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
|
| 2022-07-01 23:02:42 |
Steve Langasek |
mariadb-10.6 (Ubuntu Jammy): status |
In Progress |
Fix Committed |
|
| 2022-07-01 23:02:43 |
Steve Langasek |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2022-07-01 23:02:47 |
Steve Langasek |
bug |
|
|
added subscriber SRU Verification |
| 2022-07-01 23:02:51 |
Steve Langasek |
tags |
server-todo |
server-todo verification-needed verification-needed-jammy |
|
| 2022-07-06 12:50:43 |
Andreas Hasenack |
description |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
The test plan is to launch mariadb in a jammy lxd container running on a focal host.
lxc launch ubuntu:focal f --vm
lxc shell
lxd init # just hit enter for all questions
lxc launch ubuntu:jammy j
lxc shell j
ulimit -l # confirm it's less than 256
apt update && apt install mariadb-server -y
After the installation, mariadb will not be running, and this can be checked with:
systemctl status mariadb.service
or
journalctl -u mariadb.server
You will see something like this:
Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ;
And a crash dump.
With the fixed version, the service will be running normally after installation.
[Where problems could occur]
The proposed fix is not a surgical strike. It's unfortunate that we didn't get to the bottom of why LTO is causing this behavior. Reverting it is still the quickest and less risky change at the moment, though. This gets us on par with upstream binary builds, and debian builds, and these also have wide test coverage and ample user base.
The other regression possibility is that this is a rebuild of mariadb in the current jammy environment, and the package that is currently in jammy was built on March 10th, 2022. Most likely the reverse dependencies were updated in jammy since then.
It's unclear how 10.6.7-2ubuntu1 migrated in jammy. I checked build logs and dep8 logs, and can't tell why the tests passed. At least the build log in jammy shows the host kernel was 5.4.x, so it should have been affected. My only explanation is that at that time, the MEMLOCK limit was higher in that environment for some reason, and didn't trigger this bug. Then at some point later, launchpad builders changed, and MEMLOCK was reduced to 64Mb. https://irclogs.ubuntu.com/2022/04/28/%23ubuntu-devel.html#t15:00 has some discussion about it.
[Other Info]
Not at this time.
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
[Impact]
Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders.
Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders).
Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case.
If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required).
The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work.
MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later.
The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional).
[Test Plan]
The test plan is to launch mariadb in a jammy lxd container running on a focal host.
lxc launch ubuntu:focal f --vm
lxc shell f
lxd init # just hit enter for all questions
lxc launch ubuntu:jammy j
lxc shell j
ulimit -l # confirm it's less than 256
apt update && apt install mariadb-server -y
After the installation, mariadb will not be running, and this can be checked with:
systemctl status mariadb.service
or
journalctl -u mariadb.server
You will see something like this:
Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ...
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ;
And a crash dump.
With the fixed version, the service will be running normally after installation.
[Where problems could occur]
The proposed fix is not a surgical strike. It's unfortunate that we didn't get to the bottom of why LTO is causing this behavior. Reverting it is still the quickest and less risky change at the moment, though. This gets us on par with upstream binary builds, and debian builds, and these also have wide test coverage and ample user base.
The other regression possibility is that this is a rebuild of mariadb in the current jammy environment, and the package that is currently in jammy was built on March 10th, 2022. Most likely the reverse dependencies were updated in jammy since then.
It's unclear how 10.6.7-2ubuntu1 migrated in jammy. I checked build logs and dep8 logs, and can't tell why the tests passed. At least the build log in jammy shows the host kernel was 5.4.x, so it should have been affected. My only explanation is that at that time, the MEMLOCK limit was higher in that environment for some reason, and didn't trigger this bug. Then at some point later, launchpad builders changed, and MEMLOCK was reduced to 64Mb. https://irclogs.ubuntu.com/2022/04/28/%23ubuntu-devel.html#t15:00 has some discussion about it.
[Other Info]
Not at this time.
[Original Description]
<rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime.
<rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it.
<rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6).
<ahasenack> I think the lp builders are using the focal hwe kernel
<ahasenack> 5.4.0-something
<ahasenack> let me check that build log
<rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is.
<ahasenack> hm, both are 10.6.7
<ahasenack> release and proposed
<rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before?
<rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong.
<ahasenack> this is the current failure
<ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
<ahasenack> and ulimit -l confirms that the limit is lower
<ahasenack> Max locked memory 65536 65536 bytes
<ahasenack> just 64kbytes
<rbasak> Yeah but then how did the release pocket build work?
<ahasenack> either the limit was different back then
<ahasenack> or ... stuff |
|
| 2022-07-06 13:10:08 |
Andreas Hasenack |
tags |
server-todo verification-needed verification-needed-jammy |
server-todo verification-done-jammy verification-needed |
|
| 2022-07-10 14:06:49 |
Launchpad Janitor |
systemd (Ubuntu Jammy): status |
New |
Confirmed |
|
| 2022-07-12 23:24:32 |
Launchpad Janitor |
mariadb-10.6 (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2022-07-12 23:24:38 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2023-03-01 17:44:00 |
Bryce Harrington |
bug |
|
|
added subscriber Ubuntu Server |
| 2023-03-01 17:46:16 |
Bryce Harrington |
removed subscriber Canonical Server |
|
|
|
| 2023-03-01 19:32:15 |
Andreas Hasenack |
systemd (Ubuntu Jammy): status |
Confirmed |
Won't Fix |
|
| 2023-03-01 19:34:08 |
Andreas Hasenack |
tags |
server-todo verification-done-jammy verification-needed |
verification-done-jammy verification-needed |
|
| 2023-06-21 17:42:15 |
Nick Rosbrook |
systemd (Ubuntu): status |
Confirmed |
Fix Released |
|
