Subiquity with autoinstall cannot pick a storage and install with VROC or Intel RST firmware RAID (detected as IMSM)

[Yao Wei]

We encountered a situation the when IMSM is enabled, with autoinstall to automatically pick a disk to install to, it would fail to pick the correct storage.

Using Ubuntu Server it could pick and install into the virtual RAID device manually

To reduce possible error scope, the test was under ubuntu server noble daily build (20240125). Log attached with the following:
/var/log/installer
journalctl log
autoinstall.yaml

[Yao Wei (medicalwei)]

Also attach the same issue but over ubuntu-desktop-provision

[Dan Bungert (dbungert)]

Please provide information about what we can do to reproduce this problem. A qemu simulation would be ideal, or message privately with hardware access information.

[Yao Wei (medicalwei)]

Details about how to access the test unit is messaged via email privately to dbungert and ogayot.

[Yao Wei (medicalwei)]

Debug log when putting the installer onto a RAID disk and booting from it. It seems like subiquity picked one of the disks that formed the RAID to install to.

[Yao Wei (medicalwei)]

Turned out that, in filesystem controllers, it only picks all disks (without compound devices like RAID), so the layout mode only applies to simple disk devices, not compound devices like RAID or LVM. This I guess is to prevent some issues like installing into non-bootable RAID, setting up LVM over LVM for encryption setup, etc.

I've made the changes in the attachment in order to let auto layout detect the RAID disks, without considering about the other compound cases, therefore this patch is only a reference of my discovery and is not for production.

For reference, this is the result I got from Intel VROC from the patched subiquity to list all devices before match:
2024-03-06 01:18:43,971 INFO subiquity.server.controllers.filesystem:1418 all devices.....
2024-03-06 01:18:43,971 INFO subiquity.server.controllers.filesystem:1419 [
 Raid(name='md125', raidlevel='raid0', devices={}, spare_devices={}, preserve=True, container=raid-md127, id='raid-md125'),
 Raid(name='md127', raidlevel='container', devices=[disk-nvme2n1, disk-nvme3n1], spare_devices=[], preserve=True, metadata='imsm', id='raid-md127'),
 Raid(name='md126', raidlevel='container', devices=[disk-nvme4n1, disk-nvme6n1], spare_devices=[], preserve=True, metadata='imsm', id='raid-md126'),
 Disk(serial='KXG50ZNV256G_NVMe_TOSHIBA_256GB_77HS107ITNTT_1', wwn='eui.000000000000001000080d02002ac166', nvme_controller=nvme-controller-nvme6, path='/dev/nvme6n1', preserve=True, id='disk-nvme6n1', type='disk'),
 Disk(ptable='gpt', serial='Micron_3400_NVMe_512GB_21122E618437_1', wwn='eui.000000000000000100a075212e618437', nvme_controller=nvme-controller-nvme5, path='/dev/nvme5n1', preserve=True, id='disk-nvme5n1', type='disk'),
 Disk(serial='PC_SN730_NVMe_WDC_256GB_20333A801561_1', wwn='eui.e8238fa6bf530001001b448b49132209', nvme_controller=nvme-controller-nvme2, path='/dev/nvme2n1', preserve=True, id='disk-nvme2n1', type='disk'),
 Disk(serial='PC_SN730_NVMe_WDC_256GB_20333A802075_1', wwn='eui.e8238fa6bf530001001b448b49132a14', nvme_controller=nvme-controller-nvme3, path='/dev/nvme3n1', preserve=True, id='disk-nvme3n1', type='disk'),
 Disk(ptable='gpt', serial='PC_SN810_NVMe_WDC_512GB_23092V800479', path='/dev/sda', preserve=True, id='disk-sda', type='disk'),
 Disk(ptable='gpt', serial='PM9C1a_Samsung_256GB_______S75JNE0W700023_1', wwn='eui.002538a7310004c9', nvme_controller=nvme-controller-nvme0, path='/dev/nvme0n1', preserve=True, id='disk-nvme0n1', type='disk'),
 Disk(ptable='gpt', serial='PM9C1a_Samsung_256GB_______S75JNE0W700056_1', wwn='eui.002538a7310004ea', nvme_controller=nvme-controller-nvme1, path='/dev/nvme1n1', preserve=True, id='disk-nvme1n1', type='disk'),
 Disk(serial='THNSN5256GPUK_NVMe_TOSHIBA_256GB_X65S11JQT18T_1', wwn='eui.00080d0200113fc5', nvme_controller=nvme-controller-nvme4, path='/dev/nvme4n1', preserve=True, id='disk-nvme4n1', type='disk')
 ]

[Dan Bungert (dbungert)]

We only offer devices in guided storage if there is a plan on how to make them bootable. Are these devices bootable or are they reliant on a secondary device being bootable and using this for other storage?

[Dan Bungert (dbungert)]

apparent duplicate of LP: #1961079

[Dan Bungert (dbungert)]

Fixing the devices to match against was the right sort of idea, but we already have some checks that disks or raids are bootable, so the proposal I'm pursuing is taking advantage of that.

Attached is an updated autoinstall file that may be successful. Please assist in testing.

[Yao Wei (medicalwei)]

I tested #9 on 22.04.4 server image, which it also update the subiquity snap automatically.

The devpath on that file "/devices/virtual/block/md126" does not work, but if that match directive is removed, or "/devices/virtual/block/md*" is used instead, it will install onto "/devices/virtual/block/md125", which is a volume backed by container "/devices/virtual/block/md127".

This seems to be a good result to me.

[Dan Bungert (dbungert)]

That's great news.

So not a duplicate of LP: #1961079, and I'll go about turning this into a real PR on the main branch of subiquity for release as part of Subiquity 24.04.1 and later.

[Dan Bungert (dbungert)]

https://github.com/canonical/subiquity/pull/1934

[Dan Bungert (dbungert)]

We believe this issue listed in this bug has been resolved in Subiquity 24.04.1, which can be obtained on the Ubuntu 24.04 LTS ISO or as a snap refresh.

I'm aware that additional testing has found issues, which will be pursued on that bug.