Partitions on dmraid managed array are not mounted, system cannot boot

[Evgeny Grin]

Starting from jammy (22.04LTS) system failed to add any partitions from dmraid-managed array (Intel Software RAID). The array itself is mounted correctly (both as /dev/dm-0 and /dev/mapper/isw_xxxx), but no partitions are visible (no device like /dev/mapper/isw_xxxx1, /dev/mapper/isw_xxxx2).

It was reported many times on various sites, for example:
https://askubuntu.com/questions/1443274/boot-fails-after-upgrading-from-20-04-to-22-04-on-dmraid-device-no-partitons
https://askubuntu.com/questions/1405817/system-boots-to-initramfs-after-upgrade-from-21-10-to-22-04
https://askubuntu.com/questions/1448141/installation-problems-ubuntu-22-10-on-acer-aspire-3410-1tb-4gb-ram
https://askubuntu.com/questions/1420457/not-booting-from-raid-after-upgrade-from-21-10-to-22-04

If kpartx or partprobe are used on mounted raid device then partitions are added, however the next reboot requires manual actions again.

I was able to trace the problem down to the root cause.

The dmraid package provides udev 97-dmraid.rules file.
This file has lines

ENV{DM_STATE}=="ACTIVE", ENV{DM_UUID}=="DMRAID-*", \
 RUN+="/sbin/kpartx -a /dev/$kernel"

The used environment variable DM_STATE was set by /usr/lib/udev/dmsetup_env (from multipath-tools package) and DM_UUID is set by both dmsetup_env and 55-dm.rules (from dmsetup package).

However, Debian removed dmsetup_env starting from version 0.8.8-1 (dated Jan 2022) of multipath-tools with wrong comment that additional variables are not used anywhere.
See https://metadata.ftp-master.debian.org/changelogs/main/m/multipath-tools/multipath-tools_0.9.4-7_changelog and https://salsa.debian.org/linux-blocks-team/multipath-tools/-/commit/4ab9ce259ffa75ed3e2d145b3f2effc22af7b4c6

As the result DM_STATE is not set anywhere and kpartx from 97-dmraid.rules is not executed anymore.

The possible fixes are:
* restore dmsetup_env (the file could be reduced to set only DM_STATE variable)
* replace second rule in 97-dmraid.rules with
ENV{DM_ACTIVATION}=="1", ENV{DM_UUID}=="DMRAID-*", \
 RUN+="/sbin/kpartx -a /dev/$kernel"

The fix should be used for jammy (22.04LTS), lunar (23.04), mantic (23.10) and noble.

Sidenote: mdadm cannot correctly detect the Intel Software RAID used on my computer.

[Evgeny Grin (k2k-p)]

The problem can be easily reproduced in VM.
Just add two disk devices and create ISW raid on them with dmraid.

I used the next command to create test array:

dmraid -f isw -C HDD_SWRAID --type 0 --disk /dev/sda,/dev/sdb

Then create any partition on the array and reboot. The block devices will be "visible" for the system, but partitions are not.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Suggested patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[pasaporte (josedky73)]

Thank you for the fix, I couldn't understand why the raid would be detected on first install, and then if I assigned a partition to be mounted on load the system would just time out, I really appreciate it. Kudos to you.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mitchell Dzurick (mitchdz)]

This bug seems to be what's causing an issue with [0].

I'm planning to make a PPA with the updated patch to dmraid. If I could get some help testing the PPA, that would be great and will help with validating the patch.

[0] - https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2075442

[Mitchell Dzurick (mitchdz)]

my PPA for testing is here[0].

I made a debian bug report at[1].

[0] - https://launchpad.net/~mitchdz/+archive/ubuntu/lp2047303-dmraid-kpartx-boot
[1] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1078223

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ben (ben-durrans)]

Good news, I can confirm that the fixed packages in the PPA worked on 22.04.4, 24.04 and 24.10 (after I also installed kpartx manually, as it is no longer a dependency of dmraid in 24.10, so some guides may need updating).

[Evgeny Grin (k2k-p)]

I can confirm that my patch works for 22.04 (or a few other outdated versions).
My patch is identical to what is used in the PPA.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dmraid - 1.0.0.rc16-12ubuntu4

---------------
dmraid (1.0.0.rc16-12ubuntu4) oracular; urgency=medium

  * d/dmraid.udev: check for DM_ACTIVATION, not DM_STATE (LP: #2047303).
    multipath-tools used to set DM_STATE, but no longer does. This rule was a
    trigger to run kpartx, and thus no longer works. What this means is that
    if you have partitions in a RAID device, they will not be seen on boot.
    Forwarded to Debian via debbug#1078223.

 -- Mitchell Dzurick <email address hidden> Wed, 07 Aug 2024 15:00:56 -0700

[Mitchell Dzurick (mitchdz)]

If someone wants to do the SRU process for Noble/Jammy, feel free to go ahead. I have a few other tasks on my plate before I can start on it.