[->UUIDudev] mdadm fails to automatically assemble raid during boot (without mdadm.conf)

[Thomas]

Binary package hint: mdadm

Creating an mdadm array works fine:
mdadm --create /dev/md0 -n 3 -l5 /dev/sdb1 /dev/sdc1 /dev/sdd1
Now when booting the array is assembled correctly.

Delete the array and zero-superblock, create new partition and mark as "fd", create the same array again with command above.
Now the system tries to assemble a single device into /dev/md_d0 - when other devices are missing.

I checked each device with -examine and each reports the same uuid, the same host and the same name.
In order to get it working one has to stop the md_d0 and assemble manually:
mdadm --stop /dev/md_d0
mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1

To overcome this problem one has to remove the homehost from mdadm.conf and assemble the array manually during boot.
What helps ist to reinstall the system ...

This bug is pretty often found when googling around. Possible hints are udev and dmraid, whereas I already deinstalled dmraid but didn't help.

Ubuntu 9.10 Linux 2.6.31-14-generic-pae
mdadm 2.6.7.1

* The array gets configured when mdadm.conf reflects the array definitions. But it does not get assembled automatically

[Thomas (teclab-at)]

Forgot: I tried the same with 0.9 and 1.2 superblocks - same effect.
thx

[Sami Haahtinen (ressu)]

Confirming the bug. This is caused by udev trying to assemble the md device with --incremental. The /var/run/mdadm directory is missing from the initramfs image and the array map is not created and mdadm gets confused.

The solution is to add the following line to /usr/share/initramfs-tools/hooks/mdadm

mkdir -p ${DESTDIR}/var/run/mdadm

After this, the array is assembled correctly on boot

[ceg (ceg)]

(In this context I understand your comment, Sami.)

With UUID-based raid assembly not depending on mdadm.conf ARRAY definition maintanance this is no issue. Bug #158918

[ceg (ceg)]

Does that workaround help?
Bug #469574 [->UUIDudev] new array not assembled correctly after reboot

[ceg (ceg)]

I meant the workaround in Bug #469574. Does it help without applying Sami's?

[Jacob D'Agostino (jdagost1)]

I have been experiencing the same problem with a (non-root) RAID-1 array.

Ceg, I tried the workaround (building a proper mdadm.conf file and putting it in initramfs). If I put the array information into mdadm.conf, the array is built successfully during boot, HOWEVER, the member partitions for the array are not hidden from /proc/partitions (which they are if the array is built after bootup), so both the array and the member partitions show up in userland tools (e.g. the Gnome "Places" menu), so the places menu shows three entries rather than one, one for each disk and one for the array, and if I was to mis-click on one of the member disks, my RAID-1 array would be corrupted and need to be rebuilt.

It really looks like there is some kind of race condition going on here.

[Jacob D'Agostino (jdagost1)]

On looking at this further, I seem to only have that problem with a drive-level RAID. A partition-level RAID will still inform the system that the member partitions are not available for use.

Of course, automatic assembly still fails if the array isn't specified in initrd's mdadm.conf. Sami's workaround didn't help that, either.

[Surbhi Palande (csurbhi)]

@tombert, I suggest you to do the following:
1) /usr/share/mdadm/mkconf
2) compare the output of step1 with the output of /etc/mdadm/mdadm.conf. If your array definition is not found in
/etc/mdadm/mdadm.conf, then you will need to add this array definition manually (using its uuids).
(* The initramfs/hook for mdadm shall soon be updated. It should create a mdadm.conf for you or warn you when an active array definition is not found in mdadm.conf)
3) update-initramfs -c -k <kernel-version>. (Verify that the initramfs has your array definition in it)
4) update-grub
5) reboot

This should ensure that your raid is configured while booting. * If /var/run/mdadm is not present then mdadm makes a map in /var/run/ which is already present in initramfs. So this should not be a show stopper for raid bring up. (please refer to man mdadm for more information)
Marking this bug as invalid as of now.

[Thomas (teclab-at)]

Thats a bit fast ... and misunderstood.

The raid array I'am using is not a bootable array - thus the initramfs etc. makes no sense.
Second: raid is automatically reconstructed from the superblocks as soon as the service starts - thus I do not need an mdadm.conf

regards

[Thomas (teclab-at)]

Please put back this item from invalid to confirmed. thx

[Jacob D'Agostino (jdagost1)]

@Surbhi

mdadm is supposed to be able to assemble arrays that are not explicitly
specified in the configuration file by scanning superblocks during bootup.
That's what this bug is about. Explicitly specifying arrays in mdadm.conf is
a workaround, not the solution.

On Wed, Sep 15, 2010 at 8:00 AM, Surbhi Palande
<email address hidden>wrote:

> @tombert, I suggest you to do the following:
> 1) /usr/share/mdadm/mkconf
> 2) compare the output of step1 with the output of /etc/mdadm/mdadm.conf. If
> your array definition is not found in
> /etc/mdadm/mdadm.conf, then you will need to add this array definition
> manually (using its uuids).
> (* The initramfs/hook for mdadm shall soon be updated. It should create a
> mdadm.conf for you or warn you when an active array definition is not found
> in mdadm.conf)
> 3) update-initramfs -c -k <kernel-version>. (Verify that the initramfs has
> your array definition in it)
> 4) update-grub
> 5) reboot
>
> This should ensure that your raid is configured while booting. * If
> /var/run/mdadm is not present then mdadm makes a map in /var/run/ which is
> already present in initramfs. So this should not be a show stopper for raid
> bring up. (please refer to man mdadm for more information)
> Marking this bug as invalid as of now.
>
>
> ** Changed in: mdadm (Ubuntu)
> Status: Confirmed => Invalid
>
> --
> [->UUIDudev] mdadm fails to assemble raid during boot
> https://bugs.launchpad.net/bugs/541058
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[ceg (ceg)]

@Surbhi
when you go around changing many bug statuses you chould probably subscribe to mdadm bug mail in general and be careful with changes.

[Surbhi Palande (csurbhi)]

@tombert, have you tried booting using the Array definition in the configuration file (/etc/mdadm/mdadm.conf?) Does that work for you? If so, then can you add this fact to your summary? Thanks for that!

Also, can you please attach the output of the following (without your ARRAY definition in mdadm.conf)

1) cat /proc/mdsat?
2) madm -E: for: /dev/sdb1 /dev/sdc1 /dev/sdd1
3) set the level for udev as "debug" and then the udev log
4) contents of udevdb
5) output of dmesg

Thanks!

[Thomas (teclab-at)]

... have you tried booting using the Array definition in the configuration file (/etc/mdadm/mdadm.conf?) ...
yes this I tried as I remember correctly - its already half a year ago - please don't be offended if I don't try this again cause I already have critical data on my system.

I hope the log files are containing what you are looking for.

thx for help
thomas

[Surbhi Palande (csurbhi)]

@tombert, thats fine! thanks for the files! I have updated the summary and bug title with the information you provided. Shall provide further updates soon!

[Surbhi Palande (csurbhi)]

@tombert, and any one else who is seeing a similar bug, I hope that the following helps you in resolving this bug. Please do let us know if you do happen to try it out.
-----------------------------------------------
Call for testing mdadm 2.7.1 autoassembly.

JFYI, for hitherto Ubuntu releases the mdadm package shall stay at 2.7.1 However Natty would have mdadm at 3.4.1. This document is intended to test the mdadm fixes for 2.7.1. Here is the rough procedure that needs to be followed:

Testing auto-assembly of your md array when your rootfs lies on it:
1)Install the mdadm package and initramfs package kept at: https://edge.launchpad.net/~csurbhi/+archive/mdadm-autoassembly
2)Run /usr/share/mdadm/mkconf and ensure that your /etc/mdadm/mdadm.conf has the array definition.
a) Save your original initramfs in /boot itself by say /boot/initrd-old.img.
b) Then run update-initramfs -c -k <your-kernel-version>. Store this iniramfs as /boot/initrd-new.img. We shall use this initramfs as a safety net. If you cannot boot with the auto-assembly fixes, then you should not land in a foot in your mouth situation. Through grub's edit menu, you can then resort to this safety net by editing the initrd=initrd-new.img (or if this does not work for some random reason then resort back to your older initrd=initrd-old.img) This way you will be sure that you can still boot your precious system.
c) Now comment or remove the ARRAY definitions from your /etc/mdadm/mdadm.conf and once again run the same “update-initramfs -c -k <your-kernel-version>” to generate a brand new initramfs.
3)Run mdadm –detail –scan and note the UUIDs in the array. Note the hostname stored in your array. Does it not match with your real hostname? Then we can fix that at the initramfs prompt that you inevitably will land at if you try auto-assembly. Also note the device components that form the root md-device. Keep this paper for cross checking when you reboot
4)Reboot.
5)If you are at the initramfs prompt here are the things that you should first ensure:
a) ls /bin/hostname /etc/hostname - are these files present?
b) run “hostname”. Does this show you the hostname that your system is intended to have? Is it the same as the contents of /etc/hostname.
c) ls /var/run – Is this dir there?
If you answer yes to the above three questions, then things are so far so good. Now run the following command:
mdadm –assemble -U uuid /dev/<md-name> <dev-components-listed here>
Your mdadm –detail –scan that you ran previously should have given you the component names if you dont know it right now. Hopefully you have them listed on your paper.
Eg in my case I ran:
mdadm –assemble -U uuid /dev/md0 /dev/sda1 /dev/sdb1
Again run:
mdadm –detail –scan <md-device> and verify that the uuids are indeed updated and the hostname reflects the hostname that is stored /etc/hostname. You can now press Ctr+D and you should come back to the root prompt. However you still need to test auto-assembly of your root md device. To do that simple reboot and you should not see the face of initramfs this time. You should land gently on your root prompt as you expected. If you do not see the light of the rootfs...

[rooijan (rrossouw)]

For me no amount of settings in mdadm.conf worked. --examine looks fine on both my partitions participating in a raid 1 volume.

I played with DEVICE, HOMEHOST, ARRAY and tried all kind of suggestions. The only way I could get my raid1 array to assemble on bootup was doing this:

root@jamaica:~# cat /lib/udev/rules.d/85-mdadm.rules | grep mdadm
# This file causes block devices with Linux RAID (mdadm) signatures to
# automatically cause mdadm to be run.
 RUN+="/sbin/mdadm /dev/md0 --assemble --add /dev/sdb2 /dev/sdc2"
 #RUN+="/sbin/mdadm --incremental $env{DEVNAME}"

Plus:
root@jamaica:~# update-initramfs -u

[rooijan (rrossouw)]

Sorry I forgot it might be helpful to knowthe following also....

root@jamaica:~# cat /etc/issue
Ubuntu 11.10 \n \l

root@jamaica:~# uname -a
Linux jamaica 3.0.0-16-generic #28-Ubuntu SMP Fri Jan 27 17:44:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

root@jamaica:~# mdadm -V
mdadm - v3.1.4 - 31st August 2010

[rooijan (rrossouw)]

Just an update for everyone interested. I solved my issue by one(or multiple) of the following steps. Failing my sdc disk, pasting new superblock, writing new boot sector, partitioning again and adding the disk back. So I am still not sure what caused my problem except I am fairly sure that my 2nd disk had a previous superblock that confused things. Disk was used in another system as part of a md array. What showed me the issue was "mdadm -Es". It showed me two arrays that can be assembled.

Sorry I can not be more helpful except you should run mdadm -Es early on and avoid a lot of tail chasing. If you are seeing an array you did not expect that will cause the problems with "--incremental" when booting up.

Before I fixed the issue I was seeing this:
root@jamaica:~# mdadm --examine --scan
ARRAY /dev/md0 UUID=5551e210:6dbc9e1b:2aa365fc:c3f97a4c
ARRAY /dev/md0 UUID=4795c86a:79111507:dbd3f904:98c17f24

After:
root@jamaica:~# mdadm --examine --scan
ARRAY /dev/md0 UUID=4795c86a:79111507:dbd3f904:98c17f24

Also worth noting is my superblock is 0.90.