Dual-boot install using mdadm root fails to boot

[Christian Gunning]

I just finished installing a freshly downloaded jaunty alternate install cd using mdadm partitions in linux as a dual-boot with windows, leaving windows in it's factory-fresh "fakeraid" partition(s). The only difficult part was getting rid of dmraid.

I first shrank the windows partitions. Using the alternate cd, i set up root on a mdadm partition. Everything as expected. Windows booted fine after it fsck'd itself. But linux dropped to the initramfs busybox with the now-familiar error about "no root found, try waiting, etc." upon first boot.

Using a livecd, i was able to boot, install mdadm, activate my root, mount it and then /boot on top of it, chroot into it, mount sys, proc, and dev, and then apt-get remove dmraid. Finally, i had to uninstall and re-install mdadm to purge dmraid from initramfs ( see discussion here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=534274 ).

Windows "fakeraid" and linux mdadm are now peacefully coexisting. It's not clear to me if the core pathology is in dmraid, or in what the installer puts into initramfs. Alternately, being able to pass "nodmraid" as a boot argument would help.

[Danny Wood (danwood76)]

To get rid of dmraid you just don't install it.
The live cd doesn't have the dmraid package preloaded and I'm sure getting rid of dmraid is as simple as un-installing it through apt and rebuilding the initramfs. (2 commands from the terminal)

This is no bug in dmraid.

[Danny Wood (danwood76)]

Also you could purge the fakeraid from the drives by clearing the metadata at the end of the disks and then dmraid wouldn't even load.

[Christian Gunning (helmingstay)]

Just to reiterate, I achieved the desired results described in the
original bug after extensive research, trial and error before i filed
the bug.

"To get rid of dmraid you just don't install it. The live cd doesn't
have the dmraid package preloaded"

Ok. But that's not at issue. The alternate install cd was used,
which *does* install dmraid by default. See original bug.

" I'm sure getting rid of dmraid is as simple as un-installing it
through apt and rebuilding the initramfs."

Not true! Perhaps this is an upstream issue, but it's very unexpected
behavior that "apt-get remove dmraid" does ***not*** purge dmraid from
initramfs. Uninstall and reinstall of mdadm is required. See original
bug for discussion, esp. link to debian bug.

> Also you could purge the fakeraid from the drives by clearing the
> metadata at the end of the disks and then dmraid wouldn't even load.

Ok, but that defeats the goal stated in the original bug, since
windows could not use its default configuration of fakeraid.

I understand that the interaction between dmraid, mdadm, the 2
different install disks, initramfs, bios and windows is non-trivial,
but none of the responses to date address the original bug, and some
contain false assertions.

[Luke Yelavich (themuso)]

You cannot have mdadm and dmraid co-exist, unless the mdadm partitions and the dmraid array are on separate drives. YOu can only have one or the other.

So if you have WIndows sitting on fakeraid/dmraid array, you install Ubuntu using the laternate CD, and when it asks for you to activate the SATA RAID array, you choose yes, and create partitions for filesystems as you would normally.

[Danny Wood (danwood76)]

After you apt-get remove you must rebuild the initramfs (sudo update-initramfs -u -k all), as described in my post. From your post it sounds like the uninstall from apt will remove the initramfs hooks just not rebuild the array.

As luke said you have one or the other, there is no advantage of using both types on one drive.

[Tormod Volden (tormodvolden)]

Luke and danwood76, it seems like you haven't read the original bug report correctly. He does not want to use dmraid in Linux, only his Windows partition uses dmraid. His problem is that dmraid is unconditionally installed when a fakeraid is detected on his disks.

And clearly it must be a bug that the initrd is not rebuilt after purging dmraid.

I can inform you that the live CD boots up with dmraid loaded. If there are no fakeraids involved, the dmraid package will be deleted from the installation target system by ubiquity.

[Danny Wood (danwood76)]

After reading the bug more deeply Giuseppe states that it does invoke the update-initramfs but only on the latest kernel.

The thing I dont understand is why you would want to use mdadm and dmraid together, each provides the same functionality although dmraid allows you to boot off of RAIDs other than RAID1. Mdadm and Dmraid can only cause issues when used together as they are both fighting to do the same thing.

I'm fairly sure on my last install from the livecd I had to install dmraid seperately to starting ubiquity, mind you it may just be that ubiquity installs it anyway and I was being inpatient.

[Giuseppe Iuculano (giuseppe-iuculano)]

Hi,

Tormod Volden ha scritto:
> Luke and danwood76, it seems like you haven't read the original bug
> report correctly. He does not want to use dmraid in Linux, only his
> Windows partition uses dmraid. His problem is that dmraid is

The dmraid-activate script uses the newly introduced -Z flag to instruct the
kernel to remove partition device nodes from array member disk.
This was necessary to avoid race conditions with udev and creating UUID/label
symbolic links either for member disk nodes, or dmraid device-mapper nodes.

When you create a raidset in your bios, a metadata is written to the disk,
For example, if you create a dmraid raid1 array, you creat a raidset of 2
*disks*, not partitions but disks!

BIOS raid is something which works at the disk level, just like hardware raid
usually does.
Then if you create mdraid out of the 2 raw partitions, you create mdraid meta
data to the *partitions*. This is dead wrong, the raw disk partitions should
never be used when the disks are part of a raid set.
As with BIOS raid partitions are at the raidset level, not the partition level.

I'm aware that before the introduction of the -Z flag, this wrong setup worked
(some people are using dmraid only to handle the windows partition), so the
question is if we should support users who have the bad setup or not.

IMHO we have these options:

1) leaving things as they are, and doesn't support people using this setup
2) support this setup (create a conf file possibly handled by a debconf question?)

Luke, what do you think about it?

Cheers,
Giuseppe.

[Danny Wood (danwood76)]

A conf file might be a nice workaround and should keep everyone happy.
So I would vote option 2 although support would still be fairly limited.

[Christian Gunning (helmingstay)]

I want to reiterate that the original bug was reported against the
_alternate_ install cd. If i understand correctly, the alternate
install cd is required to install using any raid functionality. The
initramfs installed by the alternate cd included mdadm, which was
unexpectedly difficult to remove.

In my case, i had to use the livecd to rescue the system, since the
livecd did not (in my case) automatically load dmraid, allowing me to
manually mount my mdadm-generated root partition and modify the
freshly installed system.

A brief note as to *why* - I do not plan on reinstalling or altering
windows (other than to shrink partitions), and i need to use mdadm
under linux. I'm curious to hear evidence of configuration as
unstable.

thanks for all the feedback,
christian

[Danny Wood (danwood76)]

What you have to understand is that DMRAID is just a driver for the bios raid setups.

When you create a RAID from the BIOS it does a RAID1 or whatever on the entire array using all the drives you select. You cant do a partial bios raid setup as this is illogical.

When you use mdadm to create a RAID on a BIOS RAID without dmraid you are effectivly corrupting the bios' RAID array as mdadm will alter the drives that the BIOS has assigned to that array.

It is much simpler to just use dmraid instead of mdadm, it will gather the raid setup from the bios without you having to do anything.

[Christian Gunning (helmingstay)]

Yes, i understand that it's a driver for BIOS magic that i'd rather not use.

> When you create a RAID from the BIOS it does a RAID1 or whatever on the
> entire array using all the drives you select. You cant do a partial bios
> raid setup as this is illogical.

Is there a usage scenario where this causes problems? It's running,
i'm using it, and it appears to work just fine. I agree, it's an ugly
hack, but so is BIOS raid... If it's a dangerous hack, how is this
likely to manifest?

> It is much simpler to just use dmraid instead of mdadm, it will gather
> the raid setup from the bios without you having to do anything.

The "official" ubuntu documentation begs to differ with respect to
both ease and safety:
from https://help.ubuntu.com/community/FakeRaidHowto:
"FakeRAID is not supported by Ubuntu."
"So it secures the system from data loss, but the system can
nonetheless crash. "

Not entirely confidence-inspiring...

[Danny Wood (danwood76)]

That wiki page is a little old: https://wiki.ubuntu.com/DmraidSupport
Ubuntu must support it if its in the installer CD and in the main repos so I'm unsure if that is just outdated like most of the online documentation.

I have been using dmraid (and bios raid) for years and have had no issues so I consider it very stable and usable.

I'm not sure how dangerous your setup is. It may work for you but someone with a different style of BIOS RAID controller may get issues if it stores metadata in special places which will mess with mdadm. Or worse if mdadm accidentally overwrites metedata for the bios raid and you loose the entire array.

[Christian Gunning (helmingstay)]

Interesting, thanks for the info. Sorry to flog this to death.

Online ubuntu docs with respect to dmraid are definitely somewhat
scattered and of varying levels of authority. From a naive "newer is
better" perspective, the page you mention was last edited 2008-08-06,
long before jaunty went live, whereas the fakeraid howto was last
edited 2009-07-03.

Thanks again. I'll post details if anything terrible happens to my
current config.

On Wed, Jul 8, 2009 at 7:38 AM, danwood76<email address hidden> wrote:
> That wiki page is a little old: https://wiki.ubuntu.com/DmraidSupport

--
Far better an approximate answer to the right question, which is often
vague, than the exact answer to the wrong question, which can always
be made precise -- j.w. tukey

[Launchpad Janitor (janitor)]

This bug was fixed in the package dmraid - 1.0.0.rc15-10build1

---------------
dmraid (1.0.0.rc15-10build1) karmic; urgency=low

  * Fake sync with Debian.

dmraid (1.0.0.rc15-10) unstable; urgency=low

  * [1a1b752] debian/dmraid-activate: Use the -Z flag only if root
    partition is mountd in a dmraid array.
    (Closes: #533848) (LP: #392510)

 -- Luke Yelavich <email address hidden> Wed, 22 Jul 2009 10:31:41 +1000

[Tormod Volden (tormodvolden)]

I am not happy with this solution. What if I have / on one disk /dev/sdc and /home on dmraid on two other disks /dev/sda and /dev/sdb? I think my motherboard and BIOS would allow this. I would sure want /dev/sdaX and /dev/sdbX to be hidden by the -Z flag.

What I think should be done instead:
1) make sure update-initramfs is triggered on dmraid package removal
2) add for instance a boot option "nodmraid" which can be probed for in dmraid-activate

What would you think about this?

[Giuseppe Iuculano (giuseppe-iuculano)]

Tormod Volden ha scritto:
> I am not happy with this solution. What if I have / on one disk /dev/sdc
> and /home on dmraid on two other disks /dev/sda and /dev/sdb? I think my
> motherboard and BIOS would allow this. I would sure want /dev/sdaX and
> /dev/sdbX to be hidden by the -Z flag.

if /dev/sdaX and /dev/sdbX aren't part of a dmraid array kernel will not remove
them.

>
> What I think should be done instead:
> 1) make sure update-initramfs is triggered on dmraid package removal

It is already triggered:
http://git.debian.org/?p=users/derevko-guest/dmraid.git;a=blob;f=debian/dmraid.postrm

> 2) add for instance a boot option "nodmraid" which can be probed for in dmraid-activate

And how this should fix this issue?

Cheers,
Giuseppe

[Tormod Volden (tormodvolden)]

> if /dev/sdaX and /dev/sdbX aren't part of a dmraid array kernel will not remove them.

With "and /home on dmraid on two other disks /dev/sda and /dev/sdb" I meant that /dev/sdaX and /dev/sdbX will be part of a dmraid array. But /dev/sdc which contains my root is not part of an array.

> It is already triggered:

Good. For some reason this did not work when Christian tried, since he "had to uninstall and re-install mdadm to purge dmraid from initramfs".

> And how this should fix this issue?

Then Christian and other people with special set-ups can use this option to make sure dmraid is not run. Actually my two suggestions are pretty much the same as what Christian said in his original bug report.

[Giuseppe Iuculano (giuseppe-iuculano)]

Tormod Volden ha scritto:
> With "and /home on dmraid on two other disks /dev/sda and /dev/sdb" I
> meant that /dev/sdaX and /dev/sdbX will be part of a dmraid array. But
> /dev/sdc which contains my root is not part of an array.

Ok, then /dev/sdc will not be removed.

> Then Christian and other people with special set-ups can use this option
> to make sure dmraid is not run. Actually my two suggestions are pretty
> much the same as what Christian said in his original bug report.

I don't consider that as a "special set-ups". As I wrote some time ago, if you
create mdraid out of the 2 raw partitions, you create mdraid meta data to the
*partitions*. This is dead wrong, the raw disk partitions should never be used
when the disks are part of a raid set.

Cheers,
Giuseppe.

[Tormod Volden (tormodvolden)]

> Ok, then /dev/sdc will not be removed.

If I understand "Use the -Z flag only if root partition is mountd in a dmraid array." correctly, the "partitions" on the two (dmraid member) disks /dev/sda and /dev/sdb will be exposed, in my example above. This is why I think this patch is wrong.

> I don't consider that as a "special set-ups".

I agree that Christian's set-up is a broken one that we don't need to support. Anyway it could be handy sometimes to overrule any dmraid meta data left over on a disk and tell the system to not use dmraid. This is why I like "nodmraid", which also would have solved Christian's problem.

So maybe you could mark this bug "invalid" or "wontfix", but I would not say it is "Resolved" by the above patch.

[Giuseppe Iuculano (giuseppe-iuculano)]

Tormod Volden ha scritto:
>> Ok, then /dev/sdc will not be removed.
>
> If I understand "Use the -Z flag only if root partition is mountd in a
> dmraid array." correctly, the "partitions" on the two (dmraid member)
> disks /dev/sda and /dev/sdb will be exposed, in my example above. This
> is why I think this patch is wrong.

In your example above, if your /dev/sdc is not part of an array, kernel will not
remove it. If it happens this is a bug.
If /dev/sda and /dev/sdb are part of an array, they will be removed, and this is
*not* a bug, why it should be..?

>
>> I don't consider that as a "special set-ups".
>
> I agree that Christian's set-up is a broken one that we don't need to
> support. Anyway it could be handy sometimes to overrule any dmraid meta
> data left over on a disk and tell the system to not use dmraid. This is
> why I like "nodmraid", which also would have solved Christian's problem.

Yes I agree, this could be an interesting feature, but imho is not relevant with
this issue.

> So maybe you could mark this bug "invalid" or "wontfix", but I would not
> say it is "Resolved" by the above patch.

Workaround (patch) that I added in the latest revision isn't elegant, but imho
it fixed that issue. If it doesn't, please reopen this bug and write a testcase.

Cheers,
Giuseppe.

[Tormod Volden (tormodvolden)]

> In your example above, if your /dev/sdc is not part of an array, kernel will not
> remove it. If it happens this is a bug.
> If /dev/sda and /dev/sdb are part of an array, they will be removed, and this is
> *not* a bug, why it should be..?

This is what I do not understand. The way I read the patch and its description, if this array is not used for root, -Z will not be used on it, so they will not be removed.

> Yes I agree, this could be an interesting feature, but imho is not relevant with
> this issue.

If "this issue" is this bug report and Christian's problem, I would say it is.

> Workaround (patch) that I added in the latest revision isn't elegant, but imho
> it fixed that issue. If it doesn't, please reopen this bug and write a testcase.

I will leave to Christian to judge if this workaround fixes his issue.

I am starting to wondering if you are talking about the issue in the linked Debian report, which might be somewhat different. I notice that you quoted one of my comments here in that report, and when I mentioned "original bug report" there it was of course about this Ubuntu report, but it might have been confusing.

Other question: Why the special-casing of the root partition, shouldn't the same logic apply to all partitions?

[Giuseppe Iuculano (giuseppe-iuculano)]

Tormod Volden ha scritto:
>> In your example above, if your /dev/sdc is not part of an array, kernel will not
>> remove it. If it happens this is a bug.
>> If /dev/sda and /dev/sdb are part of an array, they will be removed, and this is
>> *not* a bug, why it should be..?
>
> This is what I do not understand. The way I read the patch and its
> description, if this array is not used for root, -Z will not be used on
> it, so they will not be removed.

Yes, apologies for the confusion, if this array is not used for root, -Z will
not be used on it, otherwise my above sentence is true.

>> Workaround (patch) that I added in the latest revision isn't elegant, but imho
>> it fixed that issue. If it doesn't, please reopen this bug and write a testcase.
>
> I will leave to Christian to judge if this workaround fixes his issue.
>
> I am starting to wondering if you are talking about the issue in the
> linked Debian report, which might be somewhat different. I notice that

Why are they different? They are the same issue, mdraid out of dmraid raw
partitions or no-raid installation out of dmraid raw partitions.

> Other question: Why the special-casing of the root partition, shouldn't
> the same logic apply to all partitions?
>

Yes, the logic is the same, but the special-casing prevents the boot failure
when there is a wrong setup.

Cheers,
Giuseppe.

[Tormod Volden (tormodvolden)]

I haven't looked deeply into the mdraid/dmraid conflict, but if I understand correctly, the patch
- was added to accommodate broken setups which we do not want to support
- breaks (with possible data-loss) valid setups (as in my example above)

[Tormod Volden (tormodvolden)]

We have to fix this. It also kicks in when booting a Desktop CD on a machine which has dmraid disks. The raw partitions on the dmraid disk will not be hidden as they should, and show up in the file browser. If the user now double-clicks on one it will be mounted as a normal partition and the RAID will be out of sync.

Special-casing of the root partition is wrong, we'll have to revert http://git.debian.org/?p=users/derevko-guest/dmraid.git;a=commitdiff;h=1a1b752e13238dcb6ab570b5c2567d3aef4090de and make another workaround to satisfy broken setups, like introducing a "nodmraid" boot parameter. By default, any disk with valid dmraid signatures should be treated as dmraid disks and only be accessed through the device mapper.

[Tormod Volden (tormodvolden)]

I pushed a fix here http://git.debian.org/?p=users/tormod-guest/dmraid.git;a=summary

[Tormod Volden (tormodvolden)]

I reopen this bug so don't lose it off the radar. Giuseppe, do you have any comments here or on the Debian bug?

[Giuseppe Iuculano (giuseppe-iuculano)]

Tormod Volden ha scritto:
> I reopen this bug so don't lose it off the radar. Giuseppe, do you have
> any comments here or on the Debian bug?

I already cherry-picked your [f333bc0] (nodmraid boot option), but I have some
doubts about [54b8d6f]. Reverting that change will break all broken
configurations (and a lot of people are using dmraid only to handle the windows
partition with this broken configuration)

Luke, what do you think?

Cheers,
Giuseppe.

[Tormod Volden (tormodvolden)]

In the hope of getting this fixed for Karmic...

[Tormod Volden (tormodvolden)]

Forgot to change maintainer.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dmraid - 1.0.0.rc15-11ubuntu1

---------------
dmraid (1.0.0.rc15-11ubuntu1) karmic; urgency=low

  * debian/dmraid-activate: Remove the special-casing of the root
    device which breaks in many situations and leaves the raw devices
    exposed. This was introduced in Debian to accommodate some broken
    configurations which wanted to access "partitions" on the raid
    raw devices. In Ubuntu, broken configurations has not been supported.
  * debian/dmraid-activate: Add support for "nodmraid" boot option, which
    prevents automatic detection and activation of dmraid arrays. Useful
    for broken configurations... Picked from Debian git. (LP: #392510)

 -- Tormod Volden <email address hidden> Fri, 25 Sep 2009 21:29:36 +0200