mdadm RAID10 arrays cannot be rebuilt, will not use available spare drives

[bexamous]

Binary package hint: mdadm

Rebuild degraded RAID1 or RAID5 no problem, rebuild degraded RAID10 fails.

Here is basic problem, one drive fails leaving RAID10 array and clean/degraded. Add hot spare. Drive shows up as hot spare yet auto-rebuild does not start. Only occurs with RAID10 array. I try removing the spare drive, delete its super block, and then re-add it, stll fails. Do the same thing with RAID1/RAID5 and auto-rebuild starts on its own.

This is using Ubuntu 8.10 and mdadm 2.6.7-3ubuntu7.

Here is example:
bexamous@nine:~/bin$ sudo mdadm /dev/md4 --remove /dev/sdb4
mdadm: hot removed /dev/sdb4
bexamous@nine:~/bin$ sudo mdadm --zero-superblock /dev/sdb4
bexamous@nine:~/bin$ sudo mdadm /dev/md4 --add /dev/sdb4
mdadm: added /dev/sdb4
bexamous@nine:~/bin$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid0 sdb2[0] sdd2[2] sdc2[1]
      733767168 blocks 512k chunks

md3 : active raid5 sdb3[0] sdc3[1] sdd3[6] sdh3[5] sdg3[4] sdf3[3] sde3[2]
      4300824576 blocks level 5, 512k chunk, algorithm 2 [7/7] [UUUUUUU]

md4 : active raid10 sdb4[7](S) sdc4[1] sdh4[6] sdg4[5] sdf4[4] sde4[3] sdd4[2]
      53781280 blocks 32K chunks 2 near-copies [7/6] [_UUUUUU]

unused devices: <none>
bexamous@nine:~/bin$

Here is array --detail
/dev/md4:
        Version : 00.90
  Creation Time : Mon Oct 13 02:01:54 2008
     Raid Level : raid10
     Array Size : 53781280 (51.29 GiB 55.07 GB)
  Used Dev Size : 15366080 (14.65 GiB 15.73 GB)
   Raid Devices : 7
  Total Devices : 7
Preferred Minor : 4
    Persistence : Superblock is persistent

    Update Time : Thu Oct 16 11:52:39 2008
          State : clean, degraded
 Active Devices : 6
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 1

         Layout : near=2, far=1
     Chunk Size : 32K

           UUID : c16f9559:800b499b:18ceddb4:39b7cf2e (local to host nine)
         Events : 0.32

    Number Major Minor RaidDevice State
       0 0 0 0 removed
       1 8 36 1 active sync /dev/sdc4
       2 8 52 2 active sync /dev/sdd4
       3 8 68 3 active sync /dev/sde4
       4 8 84 4 active sync /dev/sdf4
       5 8 100 5 active sync /dev/sdg4
       6 8 116 6 active sync /dev/sdh4

       7 8 20 - spare /dev/sdb4

Here is another example of fail to rebuild RAID10 and then success in rebuilting as RAID1:
root@nine:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md9 : active raid10 loop4[3] loop3[2] loop2[1] loop1[0]
      40832 blocks 64K chunks 2 near-copies [4/4] [UUUU]

unused devices: <none>
root@nine:~# mdadm /dev/md9 -f /dev/loop3
mdadm: set /dev/loop3 faulty in /dev/md9
root@nine:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md9 : active raid10 loop4[3] loop3[4](F) loop2[1] loop1[0]
      40832 blocks 64K chunks 2 near-copies [4/3] [UU_U]
root@nine:~# mdadm /dev/md9 --remove /dev/loop3
mdadm: hot removed /dev/loop3
root@nine:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md9 : active raid10 loop4[3] loop2[1] loop1[0]
      40832 blocks 64K chunks 2 near-copies [4/3] [UU_U]
root@nine:~# mdadm --zero-superblock /dev/loop3
root@nine:~# mdadm /dev/md9 --add /dev/loop3
mdadm: added /dev/loop3
root@nine:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md9 : active raid10 loop3[4](S) loop4[3] loop2[1] loop1[0]
      40832 blocks 64K chunks 2 near-copies [4/3] [UU_U]
root@nine:~#

root@nine:~# mdadm -S /dev/md9
mdadm: stopped /dev/md9
root@nine:~# mdadm --zero-superblock /dev/loop1
root@nine:~# mdadm --zero-superblock /dev/loop2
root@nine:~# mdadm --zero-superblock /dev/loop3
root@nine:~# mdadm --zero-superblock /dev/loop4
root@nine:~# mdadm -C --level=raid1 -n 4 /dev/md9 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4
mdadm: array /dev/md9 started.
root@nine:~# cat /proc/mdstat
md9 : active raid1 loop4[3] loop3[2] loop2[1] loop1[0]
      20416 blocks [4/4] [UUUU]
root@nine:~# mdadm /dev/md9 -f /dev/loop3
mdadm: set /dev/loop3 faulty in /dev/md9
root@nine:~# mdadm /dev/md9 --remove /dev/loop3
mdadm: hot removed /dev/loop3
root@nine:~# mdadm /dev/md9 --zero-superblock /dev/loop3
root@nine:~# mdadm /dev/md9 --add /dev/loop3
mdadm: added /dev/loop3
root@nine:~# cat /proc/mdstat
md9 : active raid1 loop3[2] loop4[3] loop2[1] loop1[0]
      20416 blocks [4/4] [UUUU]

[Tessa (unit3)]

I'm also experiencing this issue on intrepid/amd64, and it appears to have been noticed upstream as well (Debian Bug #495580).

This is a pretty severe bug for anyone using RAID10, as it means that arrays can't be rebuilt and recovered after a drive failure.

[Tessa (unit3)]

Oh, meant to attach my current md status, which shows the problem very well:

md1 : active raid10 sdc3[3](S) sda3[0] sdb3[1]
      113859584 blocks 512K chunks 2 near-copies [3/2] [UU_]

I've added a new drive, md knows it exists, md knows there's a drive missing... but it doesn't sync. No amount of removing and re-adding the drive makes it sync.

[Tessa (unit3)]

The other note I was going to make is that, as per the observations upstream, it looks like this is a problem more with 2.6.27 than with mdadm, so I'm going to add that to this ticket as well. mdadm seems to do the job of re-adding failed devices fine, it's just the kernel that isn't syncing the array with the available hot spare(s).

[Tessa (unit3)]

As per suggestion on #ubuntu-server, I tried removing /dev/sdc3, "dd if=/dev/zero of=/dev/sdc3 bs=1M", and then readding it to see if that would kick off a sync, but it didn't make a difference.

Going to try replacing with a totally different hard drive here shortly, just to verify that isn't the issue.

[RK (rkbwde)]

I experienced the same problem (kernel 2.6.27.4). Fortunately, I still could boot kernel 2.6.26.3 - my raid10 is rebuilding.

[Tessa (unit3)]

There's no packages for 2.6.26 in the repos anymore, so I built myself some kernel packages from the 2.6.26-5.17 sources, which is listed as the most recent pre-2.6.27 source package in launchpad.

Rebooted, and my array is happily syncing, so it definitely seems to be a problem with 2.6.27.

[David Bronaugh (bronaugh)]

Hi folks,

This'll be fixed in 2.6.28; I was experiencing the same issue and submitted a bug. See http://bugzilla.kernel.org/show_bug.cgi?id=11967

[Tessa (unit3)]

Thanks for the upstream bug, David. It's nice to see the kernel devs have already patched this issue, looks like it was a pretty simple regression from 2.6.26.

Ubuntu kernel team: is it possible to get this patch into the Intrepid 2.6.27 kernel packages, since this is a potential data loss bug, and the patch is only one line?

[Stefan Bader (smb)]

SRU justification:

Impact: Simple typo prevents added spare disks of RAID10 arrays not to be synced after one drive failures.

Fix: Upstream commit (queued to -stable) to fix an if statement.

Testcase: (see above)

[Stefan Bader (smb)]

Commited to Intrepid

http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-intrepid.git;a=commitdiff;h=3ba60c8318a7cc415cb86df1b395705ae4fe3d9a

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 2.6.27-9.18

---------------
linux (2.6.27-9.18) jaunty; urgency=low

  [ Amit Kucheria ]

  * Add the armel architecture

  [ Leann Ogasawara ]

  * Reboot stalls after installing Intrepid from alternate on a Dell
    Optiplex 330
    - LP: #293372

  [ Stefan Bader ]

  * (no-up) buildenv: Update git-ubuntu-log

  [ Tim Gardner ]

  * SAUCE: Firmware Version is invalid in cciss (version 2)
    - LP: #284031
  * Add igb to debian installer udeb
    - LP: #290073

  [ Upstream Kernel Changes ]

  * Revert "HID: Invert HWHEEL mappings for some Logitech mice"
  * md: fix bug in raid10 recovery.
    - LP: #285156
  * SCSI: sd: Fix handling of NO_SENSE check condition
  * S390: Fix sysdev class file creation.
  * sysfs: Fix return values for sysdev_store_{ulong, int}
  * USB: fix crash when URBs are unlinked after the device is gone
  * ALSA: hda - Add reboot notifier
  * kbuild: mkspec - fix build rpm
  * x86: fix /dev/mem mmap breakage when PAT is disabled
  * atl1: fix vlan tag regression
  * libertas: fix buffer overrun
  * libata: initialize port_task when !CONFIG_ATA_SFF
  * syncookies: fix inclusion of tcp options in syn-ack
  * tcpv6: fix option space offsets with md5
  * pkt_sched: sch_generic: Fix oops in sch_teql
  * sparc64: Fix race in arch/sparc64/kernel/trampoline.S
  * math-emu: Fix signalling of underflow and inexact while packing result.
  * firewire: fix setting tag and sy in iso transmission
  * firewire: fix ioctl() return code
  * firewire: Survive more than 256 bus resets
  * firewire: fix struct fw_node memory leak
  * firewire: fw-sbp2: delay first login to avoid retries
  * firewire: fw-sbp2: fix races
  * ACPI: Always report a sync event after a lid state change
  * powerpc: fix i2c on PPC linkstation / kurobox machines
  * powerpc: Reserve in bootmem lmb reserved regions that cross NUMA nodes
  * powerpc/numa: Make memory reserve code more robust
  * powerpc: Don't use a 16G page if beyond mem= limits
  * i2c: The i2c mailing list is moving
  * scx200_i2c: Add missing class parameter
  * ALSA: use correct lock in snd_ctl_dev_disconnect()
  * V4L: pvrusb2: Keep MPEG PTSs from drifting away
  * DVB: s5h1411: bugfix: Setting serial or parallel mode could destroy
    bits
  * DVB: s5h1411: Perform s5h1411 soft reset after tuning
  * DVB: s5h1411: Power down s5h1411 when not in use
  * PCI: fix 64-vbit prefetchable memory resource BARs
  * sched: disable the hrtick for now
  * sched_clock: prevent scd->clock from moving backwards
  * x86: avoid dereferencing beyond stack + THREAD_SIZE
  * rtc-cmos: look for PNP RTC first, then for platform RTC
  * USB: storage: Avoid I/O errors when issuing SCSI ioctls to JMicron
    USB/ATA bridge
  * sata_promise: add ATA engine reset to reset ops
  * sata_nv: fix generic, nf2/3 detection regression
  * ACPI: EC: Rename some variables
  * ACPI: EC: Check for IBF=0 periodically if not in GPE mode
  * libata: Fix LBA48 on pata_it821x RAID volumes.
  * ACPI: Ingore the RESET_REG_SUP bit when using ACPI reset mechanism
  * ACPI: Clear WAK_STS on resume
  * hfsplus: fix Buffer overflow with a corrupted image
  * hfsplus: check read_mapping_page() retur...

[Tessa (unit3)]

not a bug in mdadm

[Tessa (unit3)]

Hrm, I thought this was going to get patched for Intrepid as well as jaunty?

[Andy Whitcroft (apw)]

On Tue, Nov 18, 2008 at 06:16:30PM -0000, Graeme Humphries wrote:
> Hrm, I thought this was going to get patched for Intrepid as well as
> jaunty?

According to the launchpad bug a fix for this problem was committed to
Intrepid. It should appear in the 2.6.27-9.18 release which is
currently in proposed, basically in testing until for a few weeks. You
can enable proposed in update-manager to get this update.

[Tessa (unit3)]

Thanks Andy. I think I got confused from the bug posting where it listed that kernel for jaunty, not intrepid. I assumed that meant it wouldn't be released to intrepid, but I guess that's incorrect. Thanks for clearing it up.

[Martin Pitt (pitti)]

Accepted linux into intrepid-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Tessa (unit3)]

Yep, looks like it works fine here, thanks for the fix guys.

[LCID Fire (lcid-fire)]

Yeah it works again - super great thanks to you guys :)

[Loïc Minier (lool)]

Uploaded to intrepid-proposed with 2.6.27-10.20:
https://lists.ubuntu.com/archives/intrepid-changes/2008-November/009508.html

Current version in intrepid-updates/-security is 2.6.27.16.20 and according to its changelog:
http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_2.6.27-16.44/changelog

It contains the changes:
  [ Upstream Kernel Changes ]
  * md: fix bug in raid10 recovery.
    - LP: #285156