initramfs does not boot with mdadm (3.2.5-1ubuntu0.1)

Bug #1030292 reported by Eike Thaden
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
mdadm (Ubuntu)
Fix Released
Critical
Unassigned
Precise
Fix Released
High
Steve Langasek
Quantal
Fix Released
Critical
Unassigned

Bug Description

[Impact]
This is an SRU regression confined to precise-proposed. Reverting part of the upstream changes is sufficient to resolve this for precise. The issue remains unfixed in quantal; this is not a blocker for the SRU.

[Test case]
Verify that the submitter's machine boots successfully with the latest mdadm installed.

[Regression potential]
Since the existing mdadm in precise does not do auto-assembly of imsm or ddf arrays using udev rules, the risk of regression by continuing to not assemble them is minimal. Only risk of regression is a mis-edit causing some other array, previously handled correctly in precise, to no longer be assembled; the risk of this is very small.

After upgrading to the mdadm in precise-proposed, all newly created initrds do not boot. I'm using a pseudo raid with to harddisks on a Sony Vaio Z. Raid level is 0. I have one unencrypted boot partition and an encrypted root.

During startup the system asks for the password to decrypt the root partition as usual but fails to mount the partition afterwards. I always get the message that there are one or more degraded partitions. However, with an older version of mdadm everything works perfectly.

My current solution: Downgrade to latest stable mdadm (precise).

Please let me know what additional information is required. I can also perform some tests as long as my (prodcutive) system is not in danger.
---
ApportVersion: 2.0.1-0ubuntu12
Architecture: amd64
DistroRelease: Ubuntu 12.04
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111011)
MDadmExamine.dev.sda: Error: command ['/sbin/mdadm', '-E', '/dev/sda'] failed with exit code 1: mdadm: cannot open /dev/sda: Permission denied
MDadmExamine.dev.sdb: Error: command ['/sbin/mdadm', '-E', '/dev/sdb'] failed with exit code 1: mdadm: cannot open /dev/sdb: Permission denied
MDadmExamine.dev.sdc: Error: command ['/sbin/mdadm', '-E', '/dev/sdc'] failed with exit code 1: mdadm: cannot open /dev/sdc: Permission denied
MachineType: Sony Corporation VPCZ21C5E
Package: mdadm 3.2.3-2ubuntu1
PackageArchitecture: amd64
ProcEnviron:
 LANGUAGE=en_US:en
 TERM=xterm
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.5.0-6-generic root=/dev/mapper/volumegrp01-root ro ^ kopt=root=/dev/mapper/volumegrp01-root quiet splash video=intelfb pcie_aspm=force i915.i915_enable_rc6=1 i915.lvds_downclock=1 i915.i915_enable_fbc=1 vt.handoff=7
ProcMDstat:
 Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
 unused devices: <none>
ProcVersionSignature: Ubuntu 3.5.0-6.6~precise1-generic 3.5.0
Tags: precise
Uname: Linux 3.5.0-6-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm admin audio cdrom debian-tor dialout lpadmin plugdev sambashare
dmi.bios.date: 02/09/2012
dmi.bios.vendor: INSYDE
dmi.bios.version: R0172H5
dmi.board.asset.tag: N/A
dmi.board.name: VAIO
dmi.board.vendor: Sony Corporation
dmi.board.version: N/A
dmi.chassis.asset.tag: N/A
dmi.chassis.type: 10
dmi.chassis.vendor: Sony Corporation
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnINSYDE:bvrR0172H5:bd02/09/2012:svnSonyCorporation:pnVPCZ21C5E:pvrJ004PS0X:rvnSonyCorporation:rnVAIO:rvrN/A:cvnSonyCorporation:ct10:cvrN/A:
dmi.product.name: VPCZ21C5E
dmi.product.version: J004PS0X
dmi.sys.vendor: Sony Corporation
etc.blkid.tab: Error: [Errno 2] No such file or directory: '/etc/blkid.tab'

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Please clarify what do you mean by pseudo raid. Is it ISMS or DDF external metadata raid? or is it dmraid?

Changed in mdadm (Ubuntu):
status: New → Incomplete
Revision history for this message
Eike Thaden (ethaden) wrote :

Unfortunately I have not idea how to find out wether dmraid or mdadm is used (or even both). Btw. as far as I can see the system is using lvm on top of the raid with an unencrypted boot partition and encrypted root and swap partitions. This was not my idea but was proposed by the ubuntu installer.

I attached the output of dmesg and hope this helps to solve that riddle.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Can you please do

   $ apport-collect 1030292

Thanks.

Revision history for this message
Steve Langasek (vorlon) wrote :

In addition to the above apport-collect command, since you mention you're using crypted lvm, it would be helpful if you would also attach the output of the following commands:

cat /etc/crypttab
sudo vgdisplay
sudo lvdisplay

Revision history for this message
Eike Thaden (ethaden) wrote :

Of course I can provide that information. But first I have a dumb question: Currently I don't know how to run apport-collect with the new mdadm in place because as reported the boot process does not complete and I'm dropped to a busybox command line. With the previous version of mdadm I can boot the system as usual.

Is it sufficient to run apport-collect with that previous version of mdadm? If not, how can I continue booting the system? Is it safe to continue with degraded partitions?

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 1030292] Re: initramfs does not boot with mdadm (3.2.5-1ubuntu0.1)

On Sun, Jul 29, 2012 at 10:53:09AM -0000, Eike Thaden wrote:
> Of course I can provide that information. But first I have a dumb
> question: Currently I don't know how to run apport-collect with the new
> mdadm in place because as reported the boot process does not complete
> and I'm dropped to a busybox command line. With the previous version of
> mdadm I can boot the system as usual.

> Is it sufficient to run apport-collect with that previous version of
> mdadm?

Yes, booting with the previous mdadm should be fine for these purposes.
Mostly what we're looking for from apport-collect is information about the
system configuration.

> If not, how can I continue booting the system? Is it safe to
> continue with degraded partitions?

The system is designed such that booting with degraded partitions should be
safe. But we can't really make any promises in the case where it's entering
degraded mode due to some as-yet-unidentified bug, and we don't know
anything about your RAID topology.

Revision history for this message
Eike Thaden (ethaden) wrote : BootDmesg.txt

apport information

tags: added: apport-collected precise
description: updated
Revision history for this message
Eike Thaden (ethaden) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : Dependencies.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : Lspci.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : Lsusb.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : ProcModules.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : ProcMounts.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : ProcPartitions.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : UdevDb.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : UdevLog.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : initrd.files.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote : mdadm.conf.txt

apport information

Revision history for this message
Eike Thaden (ethaden) wrote :

$ cat /etc/crypttab
isw_bebgaaiie_Volume0p6_crypt UUID=36089f68-8ade-44a4-9ee4-1a9164874993 none luks

$ sudo vgdisplay
  --- Volume group ---
  VG Name volumegrp01
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 7
  VG Access read/write
  VG Status resizable
  MAX LV 0
  Cur LV 2
  Open LV 2
  Max PV 0
  Cur PV 1
  Act PV 1
  VG Size 69.72 GiB
  PE Size 4.00 MiB
  Total PE 17848
  Alloc PE / Size 17848 / 69.72 GiB
  Free PE / Size 0 / 0
  VG UUID bIgQZP-M8PV-dlc8-1QkX-WQNm-Jxwb-FZH2x9

$ sudo lvdisplay
  --- Logical volume ---
  LV Name /dev/volumegrp01/SWAP
  VG Name volumegrp01
  LV UUID SNGMwx-3guV-JhOn-WRJx-qaO7-Kgen-OKeO7A
  LV Write Access read/write
  LV Status available
  # open 2
  LV Size 8.58 GiB
  Current LE 2197
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 252:8

  --- Logical volume ---
  LV Name /dev/volumegrp01/root
  VG Name volumegrp01
  LV UUID SpYW5X-E62p-49NN-x9id-CYzj-UsqA-DQdSBC
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 61.14 GiB
  Current LE 15651
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 252:9

Revision history for this message
Steve Langasek (vorlon) wrote :

mdadm.conf shows mention of imsm. And udev thinks there's a dmraid device here.

UDEV [11.091082] add /devices/virtual/block/dm-0 (block)
ACTION=add
DEVLINKS=/dev/disk/by-id/dm-name-isw_bebgaaiie_Volume0 /dev/disk/by-id/dm-uuid-D
MRAID-isw_bebgaaiie_Volume0 /dev/mapper/isw_bebgaaiie_Volume0
DEVNAME=/dev/dm-0
DEVPATH=/devices/virtual/block/dm-0
DEVTYPE=disk
DM_NAME=isw_bebgaaiie_Volume0
DM_STATE=ACTIVE
DM_SUSPENDED=0
DM_TABLE_STATE=LIVE
DM_TYPE=raid

And /proc/mdstat appears to show no arrays.

Dmitrijs, it looks like maybe the imsm support and dmraid are fighting over the device?

Revision history for this message
Steve Langasek (vorlon) wrote :

here's the exact bit from mdadm.conf:

# definitions of existing MD arrays
ARRAY metadata=imsm UUID=e44d6ca8:d2446b48:d6cdedd2:153ecfc5
ARRAY /dev/md/Volume0 container=e44d6ca8:d2446b48:d6cdedd2:153ecfc5 member=0 UUID=8fdb592d:c433e357:c4c0232b:71639fce

But this array does NOT appear in the udev.log.

Steve Langasek (vorlon)
Changed in mdadm (Ubuntu Quantal):
status: Incomplete → Triaged
Changed in mdadm (Ubuntu Precise):
status: New → In Progress
assignee: nobody → Steve Langasek (vorlon)
importance: Undecided → High
Changed in mdadm (Ubuntu Quantal):
importance: Undecided → Critical
Steve Langasek (vorlon)
description: updated
Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Eike, or anyone else affected,

Accepted mdadm into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mdadm/3.2.5-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mdadm (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Steve Langasek (vorlon) wrote :

Eike, can you please confirm that the newer mdadm in precise-proposed works on your system?

Revision history for this message
Eike Thaden (ethaden) wrote :

Sorry for my late response. I can confirm that mdadm version 3.2.5-1ubuntu0.2 is now working for both the latest kernel in precise (3.2.0-27-generic) and the upstream kernel 3.5.0-6-generic.

Revision history for this message
Eike Thaden (ethaden) wrote :

Thank you for your great work!

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Thanks a lot for reporting & testing this!!!!

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mdadm - 3.2.5-1ubuntu0.2

---------------
mdadm (3.2.5-1ubuntu0.2) precise-proposed; urgency=low

  * Patch udev-md-raid.rules to not auto-start arrays based on detection
    of ddf or isw (imsm) signatures; this conflicts with dmraid usage in
    precise and requires more analysis before it can be enabled in SRU.
    LP: #1030292.

mdadm (3.2.5-1ubuntu0.1) precise-proposed; urgency=low

  * Stable Micro Point Bug Fix release:
    - SRU master bug (LP: #1009973)
    - Fixes segfault upon update-initramfs (LP: #969384)
    - Fixes does not allow internal bitmap on version 1.2 arrays (LP: #1022915)
  * Preserved previous behaviour, by reverting:
    - bitmap chunk size of 0 will continue to mean 512B
    - Modify the below fix, to accept previous syntax
      "b10c663 config: fix handing of 'homehost' in AUTO line."
  * Use upstream version of udev rules, instead of three slightly
    different ubuntu udev rules (LP: #968074) (LP: #1002357)
  * Add missing mdmon utility. This enables external metadata RAID
    formats: DDF and Intel Matrix Storage Management (IMSM). (LP: #957494)
  * Copy udev rule from /etc/udev/rules.d/ as well as the
    /lib/udev/rules.d/, to allow local administrator to override mdadm
    rules file (LP: #1017407)
  * debian/initramfs/local-premount: add call wait_for_udev to wait a
    little longer for RAID devices to appear. This improves
    boot reliability. (LP: #942106)

mdadm (3.2.5-1) unstable; urgency=low

  [ Michael Tokarev ]
  * new upstream (bugfix) release, fixing regression when --add'ing
    device to an array, introduced in 3.2.4, plus other minor fixes
    (Closes: #673104, #673344)
  * new patch: sha1-includes.diff to fix #include mess in new sha1.h
  * added a check into debian/checkarray to skip checking arrays created
    less than 2 weeks ago (Closes: #624273)

  [ Dmitrijs Ledkovs ]
  * Remove obsolete documentation dating back to ~etch release
  * Remove reference to obsolete documention from debconf templates
  * Update debconf templates translations
  * Remove compatability with acient initramfs-tools
  * Remove debian-specific mdadm-startall.8 in clean target

mdadm (3.2.4-1) unstable; urgency=low

  * new upstream (bugfix) release (Closes: #664088, #661552)
  * removed debian-run-udev.diff (applied upstream), and
    all RUNDIR handling from debian/rules (it is the default now)
  * add build-arch and build-indep targets to debian/rules, and
    bump Standards-Version to 3.9.3

mdadm (3.2.3-3) unstable; urgency=low

  * switch from topgit to plain 3.0 (quilt) layout, creating
    debian/patches. Don't build-depend on quilt as patching
    is done automatically by dpkg-source.
  * debian/patches/debian-run-udev.diff by Roger Leigh (Closes: #644319, #627774)
  * update debian/mdadm.logcheck.ignore.server to recognize "k" in
    addition of "blocks" in kernel messages. Thanks to Frédéric Brière
    for the patch (Closes: #656038)
 -- Steve Langasek <email address hidden> Fri, 03 Aug 2012 23:08:39 -0700

Changed in mdadm (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mdadm - 3.2.5-1ubuntu3

---------------
mdadm (3.2.5-1ubuntu3) quantal; urgency=low

  * Applying relevant changes from precise SRU:
    udev-md-raid.rules to not auto-start arrays based on detection
    of ddf or isw (imsm) signatures; this conflicts with dmraid usage in
    precise and requires more analysis before it can be enabled in SRU.
    LP: #1030292. Thanks to Steve Langasek.
 -- Dmitrijs Ledkovs <email address hidden> Tue, 21 Aug 2012 14:40:43 +0100

Changed in mdadm (Ubuntu Quantal):
status: Triaged → Fix Released
Revision history for this message
JK (j-c-k) wrote :

dmraid is essentially broken for large drives (unable to set up an imsm raid1 array on 3TB, 'fakeraid'), there is no way around mdadm. Simply disabling mdadm is not a fix. See also #957494 and #957494.
Since the mdadm package is currently severly broken when the system root is part of an array, this should really get a lot more attention for quantal.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

with external metadata formats in mdadm (ISW / DDF) one needs to activate the "container" (the controller essentially) and then activate raid devices configured with it (e.g. isw_raid_members). I see udev events/rules for the arrays themself, but not for the controller.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.