Wily LVM-RAID1 – md: personality for level 1 is not loaded

[MegaBrutal]

[Impact]
system does not boot after converting lvm volume to raid1 w/o having mdadm installed.

[Test case]

1. Install server to LVM
2. Add second disk to it
3. Run pvcreate /dev/vdb
4. Run vgextend ubuntu-vg /dev/vdb
5. Run lvconvert -m1 --type raid1 ubuntu-vg/ubuntu-lv

Reboot and check that it still boots.

6. Remove mdadm
7. Upgrade to lvm2 from proposed

Reboot and check that it still boots.

8. Downgrade lvm2 to release

Reboot and check that it fails to boot

[Regression potential]
Not really anything, we just add the raid1 module to initramfs, so it might be loaded during boot, and raid1 logical volumes might appear earlier.

[Original bug report]
After upgrading to Wily, raid1 LVs don't activate during the initrd phase. Since the root LV is also RAID1-mirrored, the system doesn't boot.

I get the following message each time LVM tries to activate a raid1 LV:
md: personality for level 1 is not loaded!

Everything was fine with Vivid. I had to downgrade to Vivid kernel (3.19.0-30) to get my system to a usable state. I pretty much hope it to be a temporary workaround and I'll get the new 4.2.0 kernel work with Wily in days.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1509717

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[MegaBrutal (qbu6to)]

I can't collect logs on non-booting system.

[MegaBrutal (qbu6to)]

Reproducible: upgraded another Ubuntu installation in VM and got the same result.
Since the bug prevents booting, I suggest to increase priority to High.

[Andy Whitcroft (apw)]

It seems that that module is built and installed into /lib/modules, but is not in the initramfs. Sounds like an initramfs-tools bug.

[Thomas Johnson (ntmatter)]

As a workaround, you can add the relevant modules to the initramfs. A quick walkthrough is as follows:

- Boot from the install media, and choose "Rescue a broken system."
- Run through all of the basic configuration steps, configuring location, keyboard, networking, timezone, etc.
- When prompted for root filesystem, select your usual root volume (eg, /dev/my-vg/root)
- Also mount a separate /boot partition
- Execute a shell in /dev/my-vg/root
- Type "mount" and ensure that the correct / and /boot volumes are actually mounted.
- Add the raid1 and mirror modules to /etc/initramfs-tools/modules, and rebuild the initramfs
# echo raid1 >> /etc/initramfs-tools/modules
# echo dm_mirror >> /etc/initramfs-tools/modules
# update-initramfs -u
- Exit out of the shell, and reboot the system. Don't forget to remove the install media!

[MegaBrutal (qbu6to)]

Thanks for the workaround! It seems adding raid1 is enough.

[Chaskiel Grundman (cg2v)]

Also affects 18.04

The "easiest" fix I can see is to add the raid1 and raid10 modules to the manually added modules in /usr/share/initramfs-tools/hooks/lvm2 (dm_raid.ko depends on raid456.ko, but not raid1.ko or raid10.ko)

[Launchpad Janitor (janitor)]

This bug was fixed in the package lvm2 - 2.03.02-2ubuntu7

---------------
lvm2 (2.03.02-2ubuntu7) focal; urgency=medium

  * Include raid1 in the list of modules installed by the initramfs hook,
    as this is not a kernel module dependency of dm-raid but if the user's
    root disk is configured as RAID1 it is definitely required.
    Closes: #841423, LP: #1509717.

 -- Steve Langasek <email address hidden> Fri, 22 Nov 2019 13:59:47 -0800

[Łukasz Zemczak (sil2100)]

Hello MegaBrutal, or anyone else affected,

Accepted lvm2 into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lvm2/2.03.02-2ubuntu6.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello MegaBrutal, or anyone else affected,

Accepted lvm2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lvm2/2.02.176-4.1ubuntu3.18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted lvm2 (2.03.02-2ubuntu6.1) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

ganeti/2.16.0-5ubuntu1 (ppc64el)
systemd/242-7ubuntu3.2 (amd64)
resource-agents/1:4.2.0-1ubuntu2 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#lvm2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Brian Murray (brian-murray)]

The fix for this bug has been awaiting testing feedback in the -proposed repository for bionic for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

[Julian Andres Klode (juliank)]

sorry, working on verifying it, got distracted by other stuff.

[Julian Andres Klode (juliank)]

verified in bionic

Did all the steps (install vm, create pv, extend pv, remove dmadm, upgrade).

It booted after the upgrade to 2.02.176-4.1ubuntu3.18.04.3, and it failed again following a downgrade to the 2.02.176-4.1ubuntu3.18.04.2 from updates pocket.

[Julian Andres Klode (juliank)]

eoan verified.

So, eoan is tougher. I installed it with subiquity and tried to follow the steps, but failed to convert to raid1, apparently, subiquity created the LV 1 PE larger than before or something, so it needed 1 free PE in the original PV to be able to add the raid1 metadata, which it did not have.

Worked around this by extending the PV size after the install (after tryin to shrink the existing LV first, messing up the install, and reinstalling ...).

With proposed lvm2 ubuntu6.1 it boots as wanted, downgrading to ubuntu6 in release breaks it again.

[Julian Andres Klode (juliank)]

The regressions in bionic seem unrelated to me adding the raid1 module to the list of modules installed to the initramfs.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lvm2 - 2.02.176-4.1ubuntu3.18.04.3

---------------
lvm2 (2.02.176-4.1ubuntu3.18.04.3) bionic; urgency=medium

  [ Steve Langasek ]
  * Include raid1 in the list of modules installed by the initramfs hook,
    as this is not a kernel module dependency of dm-raid but if the user's
    root disk is configured as RAID1 it is definitely required.
    Closes: #841423, LP: #1509717.

 -- Julian Andres Klode <email address hidden> Thu, 23 Jan 2020 16:45:10 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for lvm2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lvm2 - 2.03.02-2ubuntu6.1

---------------
lvm2 (2.03.02-2ubuntu6.1) eoan; urgency=medium

  [ Steve Langasek ]
  * Include raid1 in the list of modules installed by the initramfs hook,
    as this is not a kernel module dependency of dm-raid but if the user's
    root disk is configured as RAID1 it is definitely required.
    Closes: #841423, LP: #1509717.

 -- Julian Andres Klode <email address hidden> Thu, 23 Jan 2020 16:41:24 +0100

[MegaBrutal (qbu6to)]

Hi Steve, Julian, Łukasz, everyone,

Sorry that I didn't test the package on time – my life circumstances have significantly changed since I reported this bug, nowadays I don't have enough free time to do as much testing as I did back in the day. I've been using the workaround ever since then, returning to this now after more than 4 years really feels nostalgic.

I still had my test VM from 2016, upgraded it up to Eoan and confirmed the problem was present there as well. Then I installed the new lvm2 package which just got into -updates and indeed it fixed the issue. I know this finding is not so useful after Julian has already tested, but I wanted to see.

Since this problem was present from Wily and been around in all Ubuntu releases onwards, are you going to port this fix to all currently supported releases? While it's fixed for Bionic and Eoan; Xenial, Focal, and the actual development version (Groovy) are still affected. Now that I caught up on this again, do you want me to verify the problem in all the remaining releases?

Anyway, thanks everyone who contributed to fixing this! :)

[Julian Andres Klode (juliank)]

That's certainly incorrect, lvm2 was fixed in focal (and hence groovy). That only leaves xenial, but updating xenial close to its EOL for something as unusual as this seems unneccessary. The supported configuration for raid1 is mdadm after all, not removing mdadm and the meta packages that depend on it and using lvm's raid support.

[MegaBrutal (qbu6to)]

Sorry, I didn't actually test Focal and Groovy.

Xenial will be supported for about a year, so I think it worth to fix it there as well; however it's relative what worth to fix and what not, as most users probably left Xenial behind already or applied a workaround if they were ever affected by this issue. On the other hand it can be a real pain if you don't know about this and just happen to convert your root LV to raid1 and then your system doesn't boot.

"The supported configuration for raid1 is mdadm after all, not removing mdadm and the meta packages that depend on it and using lvm's raid support."

I don't know where LVM over DM-RAID is defined as the "supported configuration" as opposed to RAID1 over LVM. I always thought about them as equal alternatives for slightly different use cases. My rule of thumb is when you can afford to get identical sized disks and mirror everything, go with LVM over DM-RAID. If you can't guarantee to have identical sized disks and need a more dynamic solution, then RAID1 over LVM (i.e. using LVM's raid1 support) is more suitable. I recommend the former in productive server environments and the latter in smaller SOHO, home servers and personal computers. If one of my disks goes bye-bye, I wouldn't want to reinstall my system or lose my documents, so I have my / and /home LVs in raid1. However I couldn't care less about my Steam library because I can just re-download the games anytime, so I don't have it mirrored, as I'd rather not want it to use twice the precious disk space.

[Julian Andres Klode (juliank)]

There are two things:

- On servers, ubuntu-server depends on mdadm, so removing mdadm also removes ubuntu-server package, which means it's not really supported

- In general, the installers don't offer installing like that.

I don't know why we did not have a task for xenial.