mdadm update is scaring me: "YOUR SYSTEM IS NOW UNBOOTABLE!"

[stop]

Binary package hint: mdadm

I just updated mdadm, during update it gave me a scary warning:
W: mdadm: the array /dev/md0 with UUID 21fd4b98:0c4a255f:d4c030d5:a9b8b27a
W: mdadm: is currently active, but it is not listed in mdadm.conf. if
W: mdadm: it is needed for boot, then YOUR SYSTEM IS NOW UNBOOTABLE!
W: mdadm: please inspect the output of /usr/share/mdadm/mkconf, compare
W: mdadm: it to /etc/mdadm/mdadm.conf, and make the necessary changes.

I rebooted my system without making changes; nothing happened....
raid array md0 is is running:
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[0] sdc1[1]
      471041728 blocks [2/2] [UU]

unused devices: <none>

This is my mdadm.conf:
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
#DEVICE partitions
DEVICE /dev/sdb1 /dev/sdc1

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 devices=/dev/sdb1,/dev/sdc1

# This file was auto-generated on Wed, 03 Jun 2009 03:42:25 +0200
# by mkconf $Id$

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: mdadm 2.6.7.1-1ubuntu16
ProcVersionSignature: Ubuntu 2.6.35-22.33-generic 2.6.35.4
Uname: Linux 2.6.35-22-generic x86_64
NonfreeKernelModules: nvidia
Architecture: amd64
Date: Thu Sep 30 20:50:59 2010
EcryptfsInUse: Yes
MDadmExamine.dev.sda: Error: command ['/sbin/mdadm', '-E', '/dev/sda'] failed with exit code 1: mdadm: cannot open /dev/sda: Permission denied
MDadmExamine.dev.sda1: Error: command ['/sbin/mdadm', '-E', '/dev/sda1'] failed with exit code 1: mdadm: cannot open /dev/sda1: Permission denied
MDadmExamine.dev.sda2: Error: command ['/sbin/mdadm', '-E', '/dev/sda2'] failed with exit code 1: mdadm: cannot open /dev/sda2: Permission denied
MDadmExamine.dev.sda5: Error: command ['/sbin/mdadm', '-E', '/dev/sda5'] failed with exit code 1: mdadm: cannot open /dev/sda5: Permission denied
MDadmExamine.dev.sda6: Error: command ['/sbin/mdadm', '-E', '/dev/sda6'] failed with exit code 1: mdadm: cannot open /dev/sda6: Permission denied
MDadmExamine.dev.sdb: Error: command ['/sbin/mdadm', '-E', '/dev/sdb'] failed with exit code 1: mdadm: cannot open /dev/sdb: Permission denied
MDadmExamine.dev.sdb1: Error: command ['/sbin/mdadm', '-E', '/dev/sdb1'] failed with exit code 1: mdadm: cannot open /dev/sdb1: Permission denied
MDadmExamine.dev.sdc: Error: command ['/sbin/mdadm', '-E', '/dev/sdc'] failed with exit code 1: mdadm: cannot open /dev/sdc: Permission denied
MDadmExamine.dev.sdc1: Error: command ['/sbin/mdadm', '-E', '/dev/sdc1'] failed with exit code 1: mdadm: cannot open /dev/sdc1: Permission denied
MachineType: System manufacturer System Product Name
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.35-22-generic root=UUID=190948a5-a920-409e-aea4-7a0e94974738 ro quiet splash
ProcEnviron:
 LANG=en_US.utf8
 SHELL=/bin/bash
ProcMDstat:
 Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
 md0 : active raid1 sdb1[0] sdc1[1]
       471041728 blocks [2/2] [UU]

 unused devices: <none>
SourcePackage: mdadm
dmi.bios.date: 03/09/2009
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 2104
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: P5B
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2104:bd03/09/2009:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP5B:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer
etc.blkid.tab: Error: [Errno 2] No such file or directory: '/etc/blkid.tab'

[Thomas (teclab-at)]

Does affect me as well - here are the steps to reproduce:
Fresh install of Ubuntu Desktop 10.10, performing all online updates - reboot works fine. Then install mdadm the system does not boot any more:

mount: mounting /dev/disk/by-uuid/.... on /root
failed: Device or resource busy
...
...
No init found. Try passing init=bootarg.

[Thomas (teclab-at)]

Here the steps to rescue the system:

Boot with the live CD.

Mount the hard-disc:
mount /dev/sda1 /mnt

Mount system folders into sda1:
mount –t proc none /mnt/proc
mount –o bind /dev /mnt/dev
mount –o bind /sys /mnt/sys

Change the apparent disk root directory:
chroot /mnt

Now delete all occurences of mdadm:
find / -name mdadm | xargs rm -r

Now call:
update-initramfs -u

With this a new ramfs is created without mdadm - the system boots again.

[Thomas (teclab-at)]

This bug is a real killer - please put this to your highest prio or please give some workarounds!
thx

[stop (whoopwhoop)]

Tombert: I think you are suffering from another problem than I am trying to describe in this bug report.... I assume this as your problem is an unbootable system, me problem is that it boots just fine...

[Thomas (teclab-at)]

Found another possibility to make the system start:

from the modules file:
/lib/modules/<kernel>/modules.*

remove all occurances which load raid libraries - they include the path .../md/...

then run:
depmod

then run:
update-initramfs -u

the system boots again.

I'am trying to manually assemble the raid now ... hope this works ...

[Surbhi Palande (csurbhi)]

@tombert, your bug is different than this bug. Please do open another bug for the same and run apport-collect for it. Thanks!

[Surbhi Palande (csurbhi)]

@whoop, will you please do the following:
1) copy your mdadm.conf somewhere safe
2) run /usr/share/mdadm/mkconf and see if the definition of md0 is in place. If so please use "generate" with mkconf to generate a new mdadm.conf.

While updating initramfs, the mdadm's initramfs hook expects that the mdadm.conf has *uuid's* instead of /dev/<node> kind of names. As you can see this is a warning which asks you to verify the array definition, just in case its not present. This is just to prevent the case that your array cannot be auto assembled (which at present it cant be with the current mdadm-2.6.7).

Meanwhile, I shall also look at what can be done about devices instead of uuids. Thanks a lot for filing this bug!

[stop (whoopwhoop)]

/usr/share/mdadm/mkconf gives me:
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE /dev/sdb1 /dev/sdc1

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=21fd4b98:0c4a255f:d4c030d5:a9b8b27a

This also works (maybe the DEVICE stuff is supposed to be UUID's ?)... Anyway both configurations work

[Craig Hansen (craig-hansen)]

On many of my systems, the assignment of drives to letters depends on a number of unpredictable factors, so it's much safer to use the UUID to define arrays than /dev/sd[a-z]1.

Running "sudo mdadm --detail --scan" usually gets you pretty good ARRAY lines to put into your mdadm.conf file.