mdadm assembled the other superblock thus failing to boot

Bug #1164008 reported by Austin on 2013-04-03
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
mdadm (Ubuntu)
Low
Unassigned

Bug Description

I just upgraded a working 10.04 LTS server to 12.04 LTS. The server has two non-boot drives in a software RAID1 configuration. This array worked fine in 10.04 LTS.

When the machine boots I should get one active array, /dev/md0 consisting of /dev/sdb1 and /dev/sdc1. Instead, I get dumped to a busybox session and after quitting that and booting I get two inactive arrays: md0 (consisting of /dev/sdc1 as spare) and md127 (consisting of /dev/sdb as spare). If I run mdadm --detail /dev/md0 or mdadm --detail /dev/md127 I get that the device does not appear to be active.

Once I've finished booting, I can assemble my array manually by running mdadm --stop /dev/md0 and mdadm --stop /dev/md127 and then assembling the array with mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1.

Below are the results of some troubleshooting:
$ lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04

$ apt-cache policy mdadm
mdadm:
  Installed: 3.2.5-1ubuntu0.2
  Candidate: 3.2.5-1ubuntu0.2
  Version table:
 *** 3.2.5-1ubuntu0.2 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     3.2.3-2ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : inactive sdc1[1](S)
      1953513408 blocks

md127 : inactive sdb[2](S)
      1953514496 blocks

unused devices: <none>

$ mdadm --examine /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : a4176cee:096f0ba6:40663db4:2b8b6d2c
  Creation Time : Thu Jul 7 09:32:01 2011
     Raid Level : raid1
  Used Dev Size : 1953513408 (1863.02 GiB 2000.40 GB)
     Array Size : 1953513408 (1863.02 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Wed Apr 3 10:32:36 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : d685ab7e - correct
         Events : 722

      Number Major Minor RaidDevice State
this 0 8 17 0 active sync /dev/sdb1

   0 0 8 17 0 active sync /dev/sdb1
   1 1 8 33 1 active sync /dev/sdc1

$ mdadm --examine /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : a4176cee:096f0ba6:40663db4:2b8b6d2c
  Creation Time : Thu Jul 7 09:32:01 2011
     Raid Level : raid1
  Used Dev Size : 1953513408 (1863.02 GiB 2000.40 GB)
     Array Size : 1953513408 (1863.02 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Wed Apr 3 10:32:36 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : d685ab90 - correct
         Events : 722

      Number Major Minor RaidDevice State
this 1 8 33 1 active sync /dev/sdc1

   0 0 8 17 0 active sync /dev/sdb1
   1 1 8 33 1 active sync /dev/sdc1

Austin (ninjamonic) wrote :

I'm digging into this some more and it seems that mdadm in Ubuntu 12.04 is detecting an old unused superblock on /dev/sdb that mdadm in Ubuntu 10.04 ignored.

$ mdadm --examine --scan --verbose
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=5fb6928e:89f09cde:d0aebb6f:e3c24c31
   spares=1 devices=/dev/sdb
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=a4176cee:096f0ba6:40663db4:2b8b6d2c
   devices=/dev/sdc1,/dev/sdb1

When we examine /dev/sdb, we see there's an old superblock there (note the update date is Thu Jul 7 09:00:21 2011).

$ mdadm --examine /dev/sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 5fb6928e:89f09cde:d0aebb6f:e3c24c31
  Creation Time : Fri Jul 1 10:00:58 2011
     Raid Level : raid1
  Used Dev Size : 1953513408 (1863.02 GiB 2000.40 GB)
     Array Size : 1953513408 (1863.02 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1

    Update Time : Thu Jul 7 09:00:21 2011
          State : clean
 Active Devices : 1
Working Devices : 2
 Failed Devices : 1
  Spare Devices : 1
       Checksum : 57d747d3 - correct
         Events : 24

      Number Major Minor RaidDevice State
this 2 8 16 2 spare /dev/sdb

   0 0 8 33 0 active sync /dev/sdc1
   1 1 0 0 1 faulty removed
   2 2 8 16 2 spare /dev/sdb

Austin (ninjamonic) wrote :

It looks like zeroing the superblock on /dev/sdb with mdadm --zero-superblock /dev/sdb fixed the issue with the array not assembling on boot.

Dimitri John Ledkov (xnox) wrote :

I'm glad it's fixed for you, but I do wonder how many other people hit this issue.

summary: - mdadm devices not recognized correctly at boot
+ mdadm assembled the other superblock thus failing to boot
Changed in mdadm (Ubuntu):
status: New → Confirmed
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers