| 2024-06-16 15:48:16 |
Shantur Rathore |
bug |
|
|
added bug |
| 2024-06-17 08:24:08 |
Shantur Rathore |
bug |
|
|
added subscriber Steve Beattie |
| 2024-06-17 23:16:16 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Noble |
|
| 2024-06-17 23:16:16 |
Matthew Ruffell |
bug task added |
|
linux (Ubuntu Noble) |
|
| 2024-06-17 23:16:22 |
Matthew Ruffell |
linux (Ubuntu): status |
New |
Fix Released |
|
| 2024-06-18 05:36:09 |
Matthew Ruffell |
bug |
|
|
added subscriber Matthew Ruffell |
| 2024-07-09 07:05:29 |
Kai-Heng Feng |
bug |
|
|
added subscriber Kai-Heng Feng |
| 2024-07-23 17:00:06 |
Launchpad Janitor |
linux (Ubuntu Noble): status |
New |
Confirmed |
|
| 2024-07-23 17:10:49 |
Chris MacNaughton |
bug |
|
|
added subscriber Chris MacNaughton |
| 2024-07-23 17:34:21 |
Kamal Mostafa |
bug |
|
|
added subscriber Kamal Mostafa |
| 2024-07-23 17:34:48 |
Adam Bell |
bug |
|
|
added subscriber Adam Bell |
| 2024-07-23 19:47:54 |
Shantur Rathore |
merge proposal linked |
|
https://code.launchpad.net/~rathore4u/ubuntu/+source/linux/+git/version-seeds/+merge/469973 |
|
| 2024-07-24 03:19:47 |
Jordan |
bug |
|
|
added subscriber Jordan |
| 2024-07-29 23:02:52 |
Matthew Ruffell |
summary |
linux 6.8 fails to boot on arm64 if any param is more than 140 chars |
Linux 6.8 fails to boot on ARM64 if any param is more than 146 chars |
|
| 2024-07-29 23:03:50 |
Matthew Ruffell |
description |
Hi,
Linux 6.8 kernel fails to boot on ARM64 when any Linux command line param is more than 140 characters.
Test Machine
=============
Rockchip RK3399 based RockPro64 with latest u-boot 2024.07-rc3 in EFI mode booting grubaa64.efi
Reproduced on
==============
Ubuntu 22.04.4 with Linux HWE Proposed 6.8
Ubuntu 24.04 with Linux 6.8
Works on
=========
Ubuntu 22.04.4 with Linux 5.15, 6.6, 6.7, 6.9
Steps to reproduce
====================
1. Install Ubuntu 24.04 which comes with Linux 6.8 by default or Ubuntu 22.04.4 install Linux HWE 6.8 from https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/proposed?field.series_filter=jammy
2. Edit /boot/grub/grub.cfg and add the following param to any boot entry with Linux 6.8
testparam=f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b5
3. Reboot the machine and select the boot entry in grub with the testparam as above.
4. Observe kernel never boots.
Cause
========
After bisecting the kernel, I found that the bug was introduced in Linux 6.8-rc1 ( and released in Linux 6.8 ) with commit
commit dc3f5aae06381b43bc9d0d416bd15ee1682940e9
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Nov 29 12:16:12 2023 +0100
arm64: idreg-override: Avoid parameq() and parameqn()
This got fixed with Linux 6.9-rc1 and released in Linux 6.9 as part of rework
commit e223a449125571daa62debd8249fa4fc2da0a961
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Feb 14 13:28:50 2024 +0100
arm64: idreg-override: Move to early mini C runtime
That got merged to Linux 6.9-rc1 with commit
commit 6d75c6f40a03c97e1ecd683ae54e249abb9d922b
Merge: fe46a7dd189e 1ef21fcd6a50
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu Mar 14 15:35:42 2024 -0700
Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Fix Options
=============
Option 1.
For Linux 6.8 the safest fix would be to revert
commit dc3f5aae06381b43bc9d0d416bd15ee1682940e9
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Nov 29 12:16:12 2023 +0100
arm64: idreg-override: Avoid parameq() and parameqn()
Option 2.
Unless Ubuntu Team is interested in bringing the whole Mini C Runtime rework to Linux 6.8 with
commit 6d75c6f40a03c97e1ecd683ae54e249abb9d922b
Merge: fe46a7dd189e 1ef21fcd6a50
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu Mar 14 15:35:42 2024 -0700
Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Please revert the change in Linux 6.8
Thanks |
BugLink: https://bugs.launchpad.net/bugs/2068738
[Impact]
Linux 6.8 kernel fails to boot on ARM64 when any Linux command line param is more than 146 characters.
This most notably affects MAAS deployments, as MAAS generates very long command line parameters for ARM64, e.g.:
nomodeset root=squash:http://10.254.131.130:5248/images/3b08252fa962c37a47d890fb5fe182b631a0c0478d758bf4573efa859cc2c548/ubuntu/arm64/ga-24.04/noble/stable/squashfs ip=::::sjc01-2b16-u07-mgx01b:BOOTIF ip6=off cc:\{'datasource_list': ['MAAS']\}end_cc cloud-config-url=http://10-254-131-128--25.maas-internal:5248/MAAS/metadata/latest/by-id/de6dn3/?op=get_preseed ro overlayroot=tmpfs overlayroot_cfgdisk=disabled log_host=10.254.131.130 log_port=5247 --- BOOTIF=01-${net_default_mac}
This was introduced in 6.8-rc1 by:
commit dc3f5aae06381b43bc9d0d416bd15ee1682940e9
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Nov 29 12:16:12 2023 +0100
Subject: arm64: idreg-override: Avoid parameq() and parameqn()
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dc3f5aae06381b43bc9d0d416bd15ee1682940e9
There is no workaround, other than using command line parameters less than 146 characters. This is not tenable for MAAS users.
[Fix]
The fix arrived in a major refactor of early ARM64 init, where they moved from assembly to the pi mini c library. The specific commit that fixed the issue is:
commit e223a449125571daa62debd8249fa4fc2da0a961
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Feb 14 13:28:50 2024 +0100
Subject: arm64: idreg-override: Move to early mini C runtime
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e223a449125571daa62debd8249fa4fc2da0a961
However, this needs a lot of dependencies, mostly all the "mini c runtime" commits in the below merge commit:
commit 6d75c6f40a03c97e1ecd683ae54e249abb9d922b
Merge: fe46a7dd189e 1ef21fcd6a50
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu Mar 14 15:35:42 2024 -0700
Subject: Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6d75c6f40a03c97e1ecd683ae54e249abb9d922b
The amount of code is generally unacceptable for an SRU due to regression risk. I also don't think that reverting "arm64: idreg-override: Avoid parameq() and parameqn()" is the right solution either.
Thankfully, T.J. did some debugging of the root cause in comment #20 [1], and found the issue occurs because of memcmp() in include/linux/fortify-string.h detecting an attempted out-of-bounds read when comparing buf and aliases[i].alias.
That triggers the fortified memcmp()'s:
if (p_size < size || q_size < size)
fortify_panic(__func__);
where q_size == 146, size == 147, and it crashes the kernel.
[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2069534/comments/20
I know SAUCE patches are to be avoided if possible, but T.J's solution is minimal and fixes the root cause without the regression risk of backporting the entire mini C runtime, so I suggest we go with T.J's patch.
commit a4c616d2156c9c4cf7c91e6983c8bf0d51985df1
Author: Tj <tj.iam.tj@proton.me>
Date: Fri Jul 26 13:48:44 2024 +0000
Subject: UBUNTU: SAUCE: arm64: v6.8: cmdline param >= 146 chars kills kernel
Link: https://lore.kernel.org/stable/JsQ4W_o2R1NfPFTCCJjjksPED-8TuWGr796GMNeUMAdCh-2NSB_16x6TXcEecXwIfgzVxHzeB_-PMQnvQuDo0gmYE_lye0rC5KkbkDgkUqM=@proton.me/T/#u
[Testcase]
1) Deploy an ARM64 VM or use a bare metal ARM64 board with Noble, running 6.8.
2) Edit /boot/grub/grub.cfg and add the following param to any boot entry with
Linux 6.8
testparam=f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b5732f126a62b4232
3) Reboot the machine and select the boot entry in grub with the testparam as
above.
4) Observe kernel never boots.
[Where problems could occur]
We are changing command line parsing on ARM64 systems, such that we only do a memcmp() with aliased entries if the parameter we are parsing has the same length as an aliased entry. This really shouldn't have any change in functionality at all.
If a regression were to occur, then command line parsing on ARM64 systems could be broken, and it could lead to early boot failures, likely caught on automated kernel tests.
[Other Info]
This fix is 6.8 specific. It is already fixed upstream by the mini C runtime in 6.9 and later. This patch is for noble only. |
|
| 2024-07-29 23:03:58 |
Matthew Ruffell |
linux (Ubuntu Noble): status |
Confirmed |
In Progress |
|
| 2024-07-29 23:04:00 |
Matthew Ruffell |
linux (Ubuntu Noble): importance |
Undecided |
High |
|
| 2024-07-29 23:04:03 |
Matthew Ruffell |
linux (Ubuntu Noble): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2024-07-29 23:04:15 |
Matthew Ruffell |
tags |
|
noble seg |
|
| 2024-07-29 23:05:44 |
Matthew Ruffell |
description |
BugLink: https://bugs.launchpad.net/bugs/2068738
[Impact]
Linux 6.8 kernel fails to boot on ARM64 when any Linux command line param is more than 146 characters.
This most notably affects MAAS deployments, as MAAS generates very long command line parameters for ARM64, e.g.:
nomodeset root=squash:http://10.254.131.130:5248/images/3b08252fa962c37a47d890fb5fe182b631a0c0478d758bf4573efa859cc2c548/ubuntu/arm64/ga-24.04/noble/stable/squashfs ip=::::sjc01-2b16-u07-mgx01b:BOOTIF ip6=off cc:\{'datasource_list': ['MAAS']\}end_cc cloud-config-url=http://10-254-131-128--25.maas-internal:5248/MAAS/metadata/latest/by-id/de6dn3/?op=get_preseed ro overlayroot=tmpfs overlayroot_cfgdisk=disabled log_host=10.254.131.130 log_port=5247 --- BOOTIF=01-${net_default_mac}
This was introduced in 6.8-rc1 by:
commit dc3f5aae06381b43bc9d0d416bd15ee1682940e9
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Nov 29 12:16:12 2023 +0100
Subject: arm64: idreg-override: Avoid parameq() and parameqn()
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dc3f5aae06381b43bc9d0d416bd15ee1682940e9
There is no workaround, other than using command line parameters less than 146 characters. This is not tenable for MAAS users.
[Fix]
The fix arrived in a major refactor of early ARM64 init, where they moved from assembly to the pi mini c library. The specific commit that fixed the issue is:
commit e223a449125571daa62debd8249fa4fc2da0a961
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Feb 14 13:28:50 2024 +0100
Subject: arm64: idreg-override: Move to early mini C runtime
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e223a449125571daa62debd8249fa4fc2da0a961
However, this needs a lot of dependencies, mostly all the "mini c runtime" commits in the below merge commit:
commit 6d75c6f40a03c97e1ecd683ae54e249abb9d922b
Merge: fe46a7dd189e 1ef21fcd6a50
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu Mar 14 15:35:42 2024 -0700
Subject: Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6d75c6f40a03c97e1ecd683ae54e249abb9d922b
The amount of code is generally unacceptable for an SRU due to regression risk. I also don't think that reverting "arm64: idreg-override: Avoid parameq() and parameqn()" is the right solution either.
Thankfully, T.J. did some debugging of the root cause in comment #20 [1], and found the issue occurs because of memcmp() in include/linux/fortify-string.h detecting an attempted out-of-bounds read when comparing buf and aliases[i].alias.
That triggers the fortified memcmp()'s:
if (p_size < size || q_size < size)
fortify_panic(__func__);
where q_size == 146, size == 147, and it crashes the kernel.
[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2069534/comments/20
I know SAUCE patches are to be avoided if possible, but T.J's solution is minimal and fixes the root cause without the regression risk of backporting the entire mini C runtime, so I suggest we go with T.J's patch.
commit a4c616d2156c9c4cf7c91e6983c8bf0d51985df1
Author: Tj <tj.iam.tj@proton.me>
Date: Fri Jul 26 13:48:44 2024 +0000
Subject: UBUNTU: SAUCE: arm64: v6.8: cmdline param >= 146 chars kills kernel
Link: https://lore.kernel.org/stable/JsQ4W_o2R1NfPFTCCJjjksPED-8TuWGr796GMNeUMAdCh-2NSB_16x6TXcEecXwIfgzVxHzeB_-PMQnvQuDo0gmYE_lye0rC5KkbkDgkUqM=@proton.me/T/#u
[Testcase]
1) Deploy an ARM64 VM or use a bare metal ARM64 board with Noble, running 6.8.
2) Edit /boot/grub/grub.cfg and add the following param to any boot entry with
Linux 6.8
testparam=f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b5732f126a62b4232
3) Reboot the machine and select the boot entry in grub with the testparam as
above.
4) Observe kernel never boots.
[Where problems could occur]
We are changing command line parsing on ARM64 systems, such that we only do a memcmp() with aliased entries if the parameter we are parsing has the same length as an aliased entry. This really shouldn't have any change in functionality at all.
If a regression were to occur, then command line parsing on ARM64 systems could be broken, and it could lead to early boot failures, likely caught on automated kernel tests.
[Other Info]
This fix is 6.8 specific. It is already fixed upstream by the mini C runtime in 6.9 and later. This patch is for noble only. |
BugLink: https://bugs.launchpad.net/bugs/2069534
[Impact]
Linux 6.8 kernel fails to boot on ARM64 when any Linux command line param is more than 146 characters.
This most notably affects MAAS deployments, as MAAS generates very long command line parameters for ARM64, e.g.:
nomodeset root=squash:http://10.254.131.130:5248/images/3b08252fa962c37a47d890fb5fe182b631a0c0478d758bf4573efa859cc2c548/ubuntu/arm64/ga-24.04/noble/stable/squashfs ip=::::sjc01-2b16-u07-mgx01b:BOOTIF ip6=off cc:\{'datasource_list': ['MAAS']\}end_cc cloud-config-url=http://10-254-131-128--25.maas-internal:5248/MAAS/metadata/latest/by-id/de6dn3/?op=get_preseed ro overlayroot=tmpfs overlayroot_cfgdisk=disabled log_host=10.254.131.130 log_port=5247 --- BOOTIF=01-${net_default_mac}
This was introduced in 6.8-rc1 by:
commit dc3f5aae06381b43bc9d0d416bd15ee1682940e9
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Nov 29 12:16:12 2023 +0100
Subject: arm64: idreg-override: Avoid parameq() and parameqn()
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dc3f5aae06381b43bc9d0d416bd15ee1682940e9
There is no workaround, other than using command line parameters less than 146 characters. This is not tenable for MAAS users.
[Fix]
The fix arrived in a major refactor of early ARM64 init, where they moved from assembly to the pi mini c library. The specific commit that fixed the issue is:
commit e223a449125571daa62debd8249fa4fc2da0a961
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Feb 14 13:28:50 2024 +0100
Subject: arm64: idreg-override: Move to early mini C runtime
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e223a449125571daa62debd8249fa4fc2da0a961
However, this needs a lot of dependencies, mostly all the "mini c runtime" commits in the below merge commit:
commit 6d75c6f40a03c97e1ecd683ae54e249abb9d922b
Merge: fe46a7dd189e 1ef21fcd6a50
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu Mar 14 15:35:42 2024 -0700
Subject: Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6d75c6f40a03c97e1ecd683ae54e249abb9d922b
The amount of code is generally unacceptable for an SRU due to regression risk. I also don't think that reverting "arm64: idreg-override: Avoid parameq() and parameqn()" is the right solution either.
Thankfully, T.J. did some debugging of the root cause in comment #20 [1], and found the issue occurs because of memcmp() in include/linux/fortify-string.h detecting an attempted out-of-bounds read when comparing buf and aliases[i].alias.
That triggers the fortified memcmp()'s:
if (p_size < size || q_size < size)
fortify_panic(__func__);
where q_size == 146, size == 147, and it crashes the kernel.
[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2069534/comments/20
I know SAUCE patches are to be avoided if possible, but T.J's solution is minimal and fixes the root cause without the regression risk of backporting the entire mini C runtime, so I suggest we go with T.J's patch.
commit a4c616d2156c9c4cf7c91e6983c8bf0d51985df1
Author: Tj <tj.iam.tj@proton.me>
Date: Fri Jul 26 13:48:44 2024 +0000
Subject: UBUNTU: SAUCE: arm64: v6.8: cmdline param >= 146 chars kills kernel
Link: https://lore.kernel.org/stable/JsQ4W_o2R1NfPFTCCJjjksPED-8TuWGr796GMNeUMAdCh-2NSB_16x6TXcEecXwIfgzVxHzeB_-PMQnvQuDo0gmYE_lye0rC5KkbkDgkUqM=@proton.me/T/#u
[Testcase]
1) Deploy an ARM64 VM or use a bare metal ARM64 board with Noble, running 6.8.
2) Edit /boot/grub/grub.cfg and add the following param to any boot entry with
Linux 6.8
testparam=f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b5732f126a62b4232
3) Reboot the machine and select the boot entry in grub with the testparam as
above.
4) Observe kernel never boots.
[Where problems could occur]
We are changing command line parsing on ARM64 systems, such that we only do a memcmp() with aliased entries if the parameter we are parsing has the same length as an aliased entry. This really shouldn't have any change in functionality at all.
If a regression were to occur, then command line parsing on ARM64 systems could be broken, and it could lead to early boot failures, likely caught on automated kernel tests.
[Other Info]
This fix is 6.8 specific. It is already fixed upstream by the mini C runtime in 6.9 and later. This patch is for noble only. |
|
| 2024-07-30 08:10:14 |
TJ |
description |
BugLink: https://bugs.launchpad.net/bugs/2069534
[Impact]
Linux 6.8 kernel fails to boot on ARM64 when any Linux command line param is more than 146 characters.
This most notably affects MAAS deployments, as MAAS generates very long command line parameters for ARM64, e.g.:
nomodeset root=squash:http://10.254.131.130:5248/images/3b08252fa962c37a47d890fb5fe182b631a0c0478d758bf4573efa859cc2c548/ubuntu/arm64/ga-24.04/noble/stable/squashfs ip=::::sjc01-2b16-u07-mgx01b:BOOTIF ip6=off cc:\{'datasource_list': ['MAAS']\}end_cc cloud-config-url=http://10-254-131-128--25.maas-internal:5248/MAAS/metadata/latest/by-id/de6dn3/?op=get_preseed ro overlayroot=tmpfs overlayroot_cfgdisk=disabled log_host=10.254.131.130 log_port=5247 --- BOOTIF=01-${net_default_mac}
This was introduced in 6.8-rc1 by:
commit dc3f5aae06381b43bc9d0d416bd15ee1682940e9
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Nov 29 12:16:12 2023 +0100
Subject: arm64: idreg-override: Avoid parameq() and parameqn()
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dc3f5aae06381b43bc9d0d416bd15ee1682940e9
There is no workaround, other than using command line parameters less than 146 characters. This is not tenable for MAAS users.
[Fix]
The fix arrived in a major refactor of early ARM64 init, where they moved from assembly to the pi mini c library. The specific commit that fixed the issue is:
commit e223a449125571daa62debd8249fa4fc2da0a961
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Feb 14 13:28:50 2024 +0100
Subject: arm64: idreg-override: Move to early mini C runtime
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e223a449125571daa62debd8249fa4fc2da0a961
However, this needs a lot of dependencies, mostly all the "mini c runtime" commits in the below merge commit:
commit 6d75c6f40a03c97e1ecd683ae54e249abb9d922b
Merge: fe46a7dd189e 1ef21fcd6a50
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu Mar 14 15:35:42 2024 -0700
Subject: Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6d75c6f40a03c97e1ecd683ae54e249abb9d922b
The amount of code is generally unacceptable for an SRU due to regression risk. I also don't think that reverting "arm64: idreg-override: Avoid parameq() and parameqn()" is the right solution either.
Thankfully, T.J. did some debugging of the root cause in comment #20 [1], and found the issue occurs because of memcmp() in include/linux/fortify-string.h detecting an attempted out-of-bounds read when comparing buf and aliases[i].alias.
That triggers the fortified memcmp()'s:
if (p_size < size || q_size < size)
fortify_panic(__func__);
where q_size == 146, size == 147, and it crashes the kernel.
[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2069534/comments/20
I know SAUCE patches are to be avoided if possible, but T.J's solution is minimal and fixes the root cause without the regression risk of backporting the entire mini C runtime, so I suggest we go with T.J's patch.
commit a4c616d2156c9c4cf7c91e6983c8bf0d51985df1
Author: Tj <tj.iam.tj@proton.me>
Date: Fri Jul 26 13:48:44 2024 +0000
Subject: UBUNTU: SAUCE: arm64: v6.8: cmdline param >= 146 chars kills kernel
Link: https://lore.kernel.org/stable/JsQ4W_o2R1NfPFTCCJjjksPED-8TuWGr796GMNeUMAdCh-2NSB_16x6TXcEecXwIfgzVxHzeB_-PMQnvQuDo0gmYE_lye0rC5KkbkDgkUqM=@proton.me/T/#u
[Testcase]
1) Deploy an ARM64 VM or use a bare metal ARM64 board with Noble, running 6.8.
2) Edit /boot/grub/grub.cfg and add the following param to any boot entry with
Linux 6.8
testparam=f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b5732f126a62b4232
3) Reboot the machine and select the boot entry in grub with the testparam as
above.
4) Observe kernel never boots.
[Where problems could occur]
We are changing command line parsing on ARM64 systems, such that we only do a memcmp() with aliased entries if the parameter we are parsing has the same length as an aliased entry. This really shouldn't have any change in functionality at all.
If a regression were to occur, then command line parsing on ARM64 systems could be broken, and it could lead to early boot failures, likely caught on automated kernel tests.
[Other Info]
This fix is 6.8 specific. It is already fixed upstream by the mini C runtime in 6.9 and later. This patch is for noble only. |
BugLink: https://bugs.launchpad.net/bugs/2069534
[Impact]
Linux 6.8 kernel fails to boot on ARM64 when any Linux command line param is more than 146 characters.
This most notably affects MAAS deployments, as MAAS generates very long command line parameters for ARM64, e.g.:
nomodeset root=squash:http://10.254.131.130:5248/images/3b08252fa962c37a47d890fb5fe182b631a0c0478d758bf4573efa859cc2c548/ubuntu/arm64/ga-24.04/noble/stable/squashfs ip=::::sjc01-2b16-u07-mgx01b:BOOTIF ip6=off cc:\{'datasource_list': ['MAAS']\}end_cc cloud-config-url=http://10-254-131-128--25.maas-internal:5248/MAAS/metadata/latest/by-id/de6dn3/?op=get_preseed ro overlayroot=tmpfs overlayroot_cfgdisk=disabled log_host=10.254.131.130 log_port=5247 --- BOOTIF=01-${net_default_mac}
This was introduced in 6.8-rc1 by:
commit dc3f5aae06381b43bc9d0d416bd15ee1682940e9
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Nov 29 12:16:12 2023 +0100
Subject: arm64: idreg-override: Avoid parameq() and parameqn()
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dc3f5aae06381b43bc9d0d416bd15ee1682940e9
There is no workaround, other than using command line parameters less than 146 characters. This is not tenable for MAAS users.
[Fix]
The fix arrived in a major refactor of early ARM64 init, where they moved from assembly to the pi mini c library. The specific commit that fixed the issue is:
commit e223a449125571daa62debd8249fa4fc2da0a961
Author: Ard Biesheuvel <ardb@kernel.org>
Date: Wed Feb 14 13:28:50 2024 +0100
Subject: arm64: idreg-override: Move to early mini C runtime
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e223a449125571daa62debd8249fa4fc2da0a961
However, this needs a lot of dependencies, mostly all the "mini c runtime" commits in the below merge commit:
commit 6d75c6f40a03c97e1ecd683ae54e249abb9d922b
Merge: fe46a7dd189e 1ef21fcd6a50
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu Mar 14 15:35:42 2024 -0700
Subject: Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6d75c6f40a03c97e1ecd683ae54e249abb9d922b
The amount of code is generally unacceptable for an SRU due to regression risk. I also don't think that reverting "arm64: idreg-override: Avoid parameq() and parameqn()" is the right solution either.
Thankfully, Tj did some debugging of the root cause in comment #20 [1], and found the issue occurs because of memcmp() in include/linux/fortify-string.h detecting an attempted out-of-bounds read when comparing buf and aliases[i].alias.
That triggers the fortified memcmp()'s:
if (p_size < size || q_size < size)
fortify_panic(__func__);
where q_size == 146, size == 147, and it crashes the kernel.
[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2069534/comments/20
I know SAUCE patches are to be avoided if possible, but Tj's solution is minimal and fixes the root cause without the regression risk of backporting the entire mini C runtime, so I suggest we go with Tj's patch.
commit a4c616d2156c9c4cf7c91e6983c8bf0d51985df1
Author: Tj <tj.iam.tj@proton.me>
Date: Fri Jul 26 13:48:44 2024 +0000
Subject: UBUNTU: SAUCE: arm64: v6.8: cmdline param >= 146 chars kills kernel
Link: https://lore.kernel.org/stable/JsQ4W_o2R1NfPFTCCJjjksPED-8TuWGr796GMNeUMAdCh-2NSB_16x6TXcEecXwIfgzVxHzeB_-PMQnvQuDo0gmYE_lye0rC5KkbkDgkUqM=@proton.me/T/#u
[Testcase]
1) Deploy an ARM64 VM or use a bare metal ARM64 board with Noble, running 6.8.
2) Edit /boot/grub/grub.cfg and add the following param to any boot entry with
Linux 6.8
testparam=f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b5732f126a62b4232
3) Reboot the machine and select the boot entry in grub with the testparam as
above.
4) Observe kernel never boots.
[Where problems could occur]
We are changing command line parsing on ARM64 systems, such that we only do a memcmp() with aliased entries if the parameter we are parsing has the same length as an aliased entry. This really shouldn't have any change in functionality at all.
If a regression were to occur, then command line parsing on ARM64 systems could be broken, and it could lead to early boot failures, likely caught on automated kernel tests.
[Other Info]
This fix is 6.8 specific. It is already fixed upstream by the mini C runtime in 6.9 and later. This patch is for noble only. |
|
| 2024-07-30 14:25:13 |
Stefan Bader |
bug task added |
|
linux-hwe-6.8 (Ubuntu) |
|
| 2024-07-30 14:25:22 |
Stefan Bader |
linux-hwe-6.8 (Ubuntu): status |
New |
Invalid |
|
| 2024-07-30 14:25:36 |
Stefan Bader |
nominated for series |
|
Ubuntu Jammy |
|
| 2024-07-30 14:25:36 |
Stefan Bader |
bug task added |
|
linux (Ubuntu Jammy) |
|
| 2024-07-30 14:25:36 |
Stefan Bader |
bug task added |
|
linux-hwe-6.8 (Ubuntu Jammy) |
|
| 2024-07-30 14:25:48 |
Stefan Bader |
linux (Ubuntu Jammy): status |
New |
Invalid |
|
| 2024-07-30 14:25:58 |
Stefan Bader |
linux-hwe-6.8 (Ubuntu Noble): status |
New |
Invalid |
|
| 2024-07-30 14:26:05 |
Stefan Bader |
linux-hwe-6.8 (Ubuntu Jammy): status |
New |
In Progress |
|
| 2024-07-30 14:27:09 |
Stefan Bader |
linux-hwe-6.8 (Ubuntu Jammy): importance |
Undecided |
High |
|
| 2024-07-30 14:27:09 |
Stefan Bader |
linux-hwe-6.8 (Ubuntu Jammy): assignee |
|
Stefan Bader (smb) |
|
| 2024-07-30 14:35:01 |
Stefan Bader |
linux-hwe-6.8 (Ubuntu Jammy): status |
In Progress |
Fix Committed |
|
| 2024-07-31 09:33:34 |
Stefan Bader |
linux (Ubuntu Noble): status |
In Progress |
Fix Committed |
|
| 2024-08-01 16:15:57 |
Ubuntu Kernel Bot |
tags |
noble seg |
kernel-spammed-jammy-linux-hwe-6.8-v2 noble seg verification-needed-jammy-linux-hwe-6.8 |
|
| 2024-08-02 02:46:00 |
Matthew Ruffell |
tags |
kernel-spammed-jammy-linux-hwe-6.8-v2 noble seg verification-needed-jammy-linux-hwe-6.8 |
kernel-spammed-jammy-linux-hwe-6.8-v2 noble seg verification-done-jammy-linux-hwe-6.8 |
|
| 2024-08-07 20:19:58 |
Ubuntu Kernel Bot |
tags |
kernel-spammed-jammy-linux-hwe-6.8-v2 noble seg verification-done-jammy-linux-hwe-6.8 |
kernel-spammed-jammy-linux-hwe-6.8-v2 kernel-spammed-noble-linux-v2 noble seg verification-done-jammy-linux-hwe-6.8 verification-needed-noble-linux |
|
| 2024-08-08 02:50:18 |
Matthew Ruffell |
tags |
kernel-spammed-jammy-linux-hwe-6.8-v2 kernel-spammed-noble-linux-v2 noble seg verification-done-jammy-linux-hwe-6.8 verification-needed-noble-linux |
kernel-spammed-jammy-linux-hwe-6.8-v2 kernel-spammed-noble-linux-v2 noble seg verification-done-jammy-linux-hwe-6.8 verification-done-noble-linux |
|
| 2024-08-13 14:03:05 |
Launchpad Janitor |
linux-hwe-6.8 (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2024-08-13 14:03:05 |
Launchpad Janitor |
cve linked |
|
2024-25742 |
|
| 2024-08-13 14:03:05 |
Launchpad Janitor |
cve linked |
|
2024-35984 |
|
| 2024-08-13 14:03:05 |
Launchpad Janitor |
cve linked |
|
2024-35990 |
|
| 2024-08-13 14:03:05 |
Launchpad Janitor |
cve linked |
|
2024-35992 |
|
| 2024-08-13 14:03:05 |
Launchpad Janitor |
cve linked |
|
2024-35997 |
|
| 2024-08-13 14:03:05 |
Launchpad Janitor |
cve linked |
|
2024-36008 |
|
| 2024-08-13 14:03:05 |
Launchpad Janitor |
cve linked |
|
2024-36016 |
|
| 2024-08-20 21:45:02 |
Launchpad Janitor |
linux (Ubuntu Noble): status |
Fix Committed |
Fix Released |
|
| 2024-08-20 21:45:02 |
Launchpad Janitor |
cve linked |
|
2024-39292 |
|
| 2024-08-20 21:45:02 |
Launchpad Janitor |
cve linked |
|
2024-39484 |
|
| 2024-08-20 22:12:29 |
Ubuntu Kernel Bot |
tags |
kernel-spammed-jammy-linux-hwe-6.8-v2 kernel-spammed-noble-linux-v2 noble seg verification-done-jammy-linux-hwe-6.8 verification-done-noble-linux |
kernel-spammed-jammy-linux-hwe-6.8-v2 kernel-spammed-jammy-linux-lowlatency-hwe-6.8-v2 kernel-spammed-noble-linux-v2 noble seg verification-done-jammy-linux-hwe-6.8 verification-done-noble-linux verification-needed-jammy-linux-lowlatency-hwe-6.8 |
|
| 2024-08-26 13:37:19 |
Ubuntu Kernel Bot |
tags |
kernel-spammed-jammy-linux-hwe-6.8-v2 kernel-spammed-jammy-linux-lowlatency-hwe-6.8-v2 kernel-spammed-noble-linux-v2 noble seg verification-done-jammy-linux-hwe-6.8 verification-done-noble-linux verification-needed-jammy-linux-lowlatency-hwe-6.8 |
kernel-spammed-jammy-linux-gcp-6.8-v2 kernel-spammed-jammy-linux-hwe-6.8-v2 kernel-spammed-jammy-linux-lowlatency-hwe-6.8-v2 kernel-spammed-noble-linux-v2 noble seg verification-done-jammy-linux-hwe-6.8 verification-done-noble-linux verification-needed-jammy-linux-gcp-6.8 verification-needed-jammy-linux-lowlatency-hwe-6.8 |
|
