Ubuntu 10.10 2.6.35-28 kernel does not assemble md devices correctly.

[Sheldon Lee-Wen]

Binary package hint: mdadm

I'm on Ubuntu 10.10, when I tried to boot with the 2.6.35-28 kernel, for some reason on boot md tries to assemble my devices incorrectly, using the whole devices rather than the partitions. The software raid device starts, but then LVM cannot find the disk group.

e.g. Here is the output from dmesg on the 2.6.35-27 kernel, which does work properly:

md/raid:md0: device sdf1 operational as raid disk 1
md/raid:md0: device sde1 operational as raid disk 0
md/raid:md0: device sdd1 operational as raid disk 2
md/raid:md0: allocated 3179kB
md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2
RAID conf printout:
--- level:5 rd:3 wd:3
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdf1
disk 2, o:1, dev:sdd1
md0: detected capacity change from 0 to 2000407494656
md0: unknown partition table

Here is the output form 2.6.35-28 which does not work properly:

md/raid:md0: device sdd operational as raid disk 2
md/raid:md0: device sdf operational as raid disk 1
md/raid:md0: device sde operational as raid disk 0
md/raid:md0: allocated 3179kB
md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2
RAID conf printout:
--- level:5 rd:3 wd:3
disk 0, o:1, dev:sde
disk 1, o:1, dev:sdf
disk 2, o:1, dev:sdd
md0: detected capacity change from 0 to 2000407494656
md0: p1

My md0 device should not have a p1 partition on it!

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: mdadm 2.6.7.1-1ubuntu16
ProcVersionSignature: Ubuntu 2.6.35-27.48-generic-pae 2.6.35.11
Uname: Linux 2.6.35-27-generic-pae i686
NonfreeKernelModules: nvidia
Architecture: i386
Date: Tue Jun 14 09:07:35 2011
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release i386 (20101007)
MDadmExamine.dev.sdb:
 Error: command ['/sbin/mdadm', '-E', '/dev/sdb'] failed with exit code 1: mdadm: metadata format 00.90 unknown, ignored.
 mdadm: metadata format 00.90 unknown, ignored.
 mdadm: No md superblock detected on /dev/sdb.
MDadmExamine.dev.sdb1:
 Error: command ['/sbin/mdadm', '-E', '/dev/sdb1'] failed with exit code 1: mdadm: metadata format 00.90 unknown, ignored.
 mdadm: metadata format 00.90 unknown, ignored.
 mdadm: No md superblock detected on /dev/sdb1.
MDadmExamine.dev.sdb2:
 Error: command ['/sbin/mdadm', '-E', '/dev/sdb2'] failed with exit code 1: mdadm: metadata format 00.90 unknown, ignored.
 mdadm: metadata format 00.90 unknown, ignored.
 mdadm: No md superblock detected on /dev/sdb2.
MDadmExamine.dev.sdb5:
 Error: command ['/sbin/mdadm', '-E', '/dev/sdb5'] failed with exit code 1: mdadm: metadata format 00.90 unknown, ignored.
 mdadm: metadata format 00.90 unknown, ignored.
 mdadm: No md superblock detected on /dev/sdb5.
MDadmExamine.dev.sdb6:
 Error: command ['/sbin/mdadm', '-E', '/dev/sdb6'] failed with exit code 1: mdadm: metadata format 00.90 unknown, ignored.
 mdadm: metadata format 00.90 unknown, ignored.
 mdadm: No md superblock detected on /dev/sdb6.
MDadmExamine.dev.sdb7:
 Error: command ['/sbin/mdadm', '-E', '/dev/sdb7'] failed with exit code 1: mdadm: metadata format 00.90 unknown, ignored.
 mdadm: metadata format 00.90 unknown, ignored.
 mdadm: No md superblock detected on /dev/sdb7.
MachineType: System manufacturer P5K-E
ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.35-27-generic-pae root=UUID=ded4ad54-93bf-4849-99b3-096269bba553 ro crashkernel=384M-2G:64M,2G-:128M quiet splash nomodeset video=uvesafb:mode_option=1920x1200-24,mtrr=3,scroll=ywrap vmalloc=192M
ProcEnviron:
 PATH=(custom, user)
 LANG=en_CA.utf8
 SHELL=/bin/bash
SourcePackage: mdadm
dmi.bios.date: 10/31/2007
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 0806
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: P5K-E
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr0806:bd10/31/2007:svnSystemmanufacturer:pnP5K-E:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP5K-E:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: P5K-E
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer
etc.blkid.tab: Error: [Errno 2] No such file or directory: '/etc/blkid.tab'

[Sheldon Lee-Wen (sheldonl)]

I apologize in advance for filing this against mdadm; perhaps the kernel would have been a more appropriate package to file this against.

[Sheldon Lee-Wen (sheldonl)]

I managed to fix the issue by doing the following with each disk: (I use X here to represent the disk.)
mdadm --manage /dev/md0 --fail /dev/sdX1
mdadm --manage /dev/md0 --remove /dev/sdX1
mdadm --zero-superblock /dev/sdX
mdadm --zero-superblock /dev/sdX1
dd if=/dev/zero of=/dev/sdX bs=1 count=2048
fdisk /dev/sdX (recreated the first partition and mbr on each disk and ensure type set to fd)
mdadm --manage /dev/md0 -a /dev/sdX
(Allow the array to resync to 100% before doing the next disk. Each resync took a few hours.)

Once I had done this for all of the disks in the array I rebooted to the latest kernel 2.6.35-28-generic-pae. On the first boot I still had the same problem so I stopped the array with mdadm --manage --stop /dev/md0 and made the following changes to /etc/mdadm/mdadm.conf

DEVICE /dev/sda /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg /dev/sdc1
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=726e3b03:242ebdf0:85c68096:6992ccf3

I then restarted the array with:
mdadm --assemble -c /etc/mdadm/mdadm.conf /dev/md0

Then I updated all of the initramfs modules:
update-initramfs -k all -u

When I rebooted my array came online automatically without errors.