Ubuntu 24.04 doesn't deploy on any ARM64 machine

[Shantur Rathore]

Hi,

I am using MAAS 3.5 and unable to deploy any Ubuntu 24.04 images for arm64.

When tried to deploy the machine loads grub and grub loads linux but the kernel doesn't boot. I have figured out the cause, it is due to root param in the linux command line be more than 140 chars.

I have reported the bug here - https://bugs.launchpad.net/linux/+bug/2068791 which seems like an upstream issue that got fixed in Linux 6.9-rc1. I am not sure if that is the correct project to assign bugs.

Can this please be fixed?

Thanks,
Shantur

[Jacopo Rota (r00ta)]

Did you add extra kernel args? Or can you reproduce this without any customisation?

[Shantur Rathore (rathore4u)]

No extra args.

It can be reproduced without any customisation on my known working setup.

I have a RockPro64 (but could be any ARM64 SBC) running latest u-boot with boot_targets = "dhcp"

- Start deployment with Ubuntu 24.04
- Power on SBC
- Observe the Serial / UART console

[Jacopo Rota (r00ta)]

Hey @shantur,

Thank you very much for reporting this (and for playing with 3.5!).

Could you please extract the cmdline and forward it to us by calling

http://<rack_ip>:5248/grub/grub.cfg-[boot_interface_mac_address]

(for example http://192.0.1.254:5248/grub/grub.cfg-00:00:00:00:00:00)

[Shantur Rathore (rathore4u)]

Hey @jacopo,

Thanks for checking this.
Below is the grub.cfg

➜ ~ curl http://192.168.20.2:5248/grub/grub.cfg-82:af:7b:1f:2d:89
set default="0"
set timeout=0

menuentry 'Ephemeral' {
    echo 'Booting under MAAS direction...'
    linux /images/61685f790e999ae0de15da36146fe6a7877b324696a82068583059e75b84ad86/ubuntu/arm64/ga-24.04/noble/stable/boot-kernel nomodeset root=squash:http://192.168.20.2:5248/images/eeb78c78a0777303d46c6a2a6bdb315be315531e67d2829579d72df0d274e9fc/ubuntu/arm64/ga-24.04/noble/stable/squashfs ip=::::node2:BOOTIF ip6=off cc:\{'datasource_list': ['MAAS']\}end_cc cloud-config-url=http://192-168-20-0--24.maas-internal:5248/MAAS/metadata/latest/by-id/t86ntp/?op=get_preseed ro overlayroot=tmpfs overlayroot_cfgdisk=disabled log_host=192.168.20.2 log_port=5247 --- usb-storage.quirks=1de1:e101:u BOOTIF=01-${net_default_mac}
    initrd /images/fcdd3941c01dca42cde1fadec5892e80fa431dea9b2b4b34e34539f36b214e26/ubuntu/arm64/ga-24.04/noble/stable/boot-initrd
}%

[Shantur Rathore (rathore4u)]

I would like to add, this seems like a generic Linux 6.8 issue.

I am able to reproduce this on Ubuntu 22.04.4 as well.

1. Install Ubuntu 22.04.4 ( with the custom_preseed workaround here - https://bugs.launchpad.net/maas/+bug/2069079
2. Install linux-hwe-6.8 kernel for 22.04.4 from kernel proposed ppa.
3. Verify things are booting fine after upgrade with Linux 6.8 kernel
4. Edit grub.cfg and add a long unused param to linux command line like - testparam=f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b54edcba27e5f790d47911a4cc3e726d8d256878d3df9175c020e0f081c381e7b5
5. Try booting again
6. Kernel fails to boot with the param.
7. Remove param, kernel boots fine.

[Jacopo Rota (r00ta)]

Just to give you more context, we are of course aware of these hard limits and that we don't really have control on them.

However, what we MUST do on MAAS side is to try to keep the parameters short. In MAAS 3.5 we have increased a bit the length of a parameter and this is part of the issue you are facing.

We discussed this and we plan to optimize the length of these parameters in 3.5.1 (as you can see in the milestones of this bug).

Thank you again for raising this

[Shantur Rathore (rathore4u)]

@Jacopo - I disagree that this is MAAS bug.

There is no such limitations on kernel command line.
The bug is present only in kernel v6.8
I have tested kernel versions v6.6 .... v6.7 .... v6.8-rc1. to v6.8-rc8.... v6.8.... v6.9-rc1 ... v6.9

The kernel bug was introduced in v6.8-rc1 and was undetected but fixed in v6.9-rc1. All older kernels v5.15 till v6.7 work fine and v6.9-rc1 to v6.9 work fine too.

This should be fixed in Linux 6.8 kernel for both linux-hwe-6.8 for Ubuntu 22.04.4 and linux-6.8 for Ubuntu 24.04.

I am bisecting right now to find the exact commit that introduced this bug.

[Anton Troyanov (troyanov)]

I think there is a limit.

> The number of kernel parameters is not limited, but the length of the
complete command line (parameters including spaces etc.) is limited to a
fixed number of characters.
> This limit depends on the architecture and is between 256 and 4096
characters. It is defined in the file ./include/asm/setup.h as
COMMAND_LINE_SIZE. [1]

Each platform has its own COMMAND_LINE_SIZE. For arm64 it will be 2048 [2]

[1]:
https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html
[2]:
https://github.com/torvalds/linux/blob/2ccbdf43d5e758f8493a95252073cf9078a5fea5/arch/arm64/include/uapi/asm/setup.h#L25C9-L25C26

[Jacopo Rota (r00ta)]

As Anton pointed out, there are these limits and MAAS should keep the parameters short in order to avoid hitting these limits (also because the user might provide extra parameters and MAAS might need to add stuff in the future, so we should ensure we are not in the danger zone with the length of the cmdline in general).

I'm keeping this bug open since in 3.5 we have increased the length of some parameters and this could be considered a regression. We will keep track of the optimisation work we are going to perform for 3.5.1 in this bug

[Shantur Rathore (rathore4u)]

Agreed its a limit to whole command line not for one param.

I welcome the optimisation but can this bug still be raised on the kernel team who can fix this regression in both Ubuntu 24.04 and Linux 6.8 HWE for 22.04

Thanks

[Shantur Rathore (rathore4u)]

For reference - bug raise with Linux kernel - https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2069534

[Jerzy Husakowski (jhusakowski)]

Even though this is not a MAAS issue but an issue introduced in a version of ARM64 kernel, we want to reduce the risk of exceeding kernel parameter length limit and work on reducing the size of parameters (URLs, filenames, etc.) that are under MAAS control. Since these parameters have become longer in 3.5 and later, we will aim to resolve this on these releases.

[Shantur Rathore (rathore4u)]

I have figured out a way to mitigate this issue by reducing the size of checksum when creating and referencing the kernel, initrd and squashfs file on file system.

All it does is just uses the first 16 characters of checksum for file names.
With these changes the file names are well under 140 limit that I was seeing and 24.04 happily deploys.

Can this be included in 3.5 please?

The changes are simple

1. Replace

bfile.sha256,

with

bfile.sha256[0:16],

at

https://github.com/canonical/maas/blob/de2288e04a9ad6fee7dd8954f6502d754f2ef280/src/maasserver/rpc/boot.py#L106

2. Replace

Replace

self._base_path = get_bootresource_store_path() / self.sha256

with

self._base_path = get_bootresource_store_path() / self.sha256[0:16]

https://github.com/canonical/maas/blob/de2288e04a9ad6fee7dd8954f6502d754f2ef280/src/maasserver/utils/bootresource.py#L95

I can send a merge request if needed.

[Shantur Rathore (rathore4u)]

Any thoughts ?