Boot hang, apparently blocking on SW RAID rebuild

[Brian Elliott Finley]

I am experiencing this issue on a new Sun Fire V40z Server in a well cooled
machine room. (please refer to http://bugzilla.ubuntu.com/show_bug.cgi?id=10916)

Hardware:
- Quad CPU amd64 -- AMD Opteron(tm) Processor 852
- 32G memory
- 2x SCSI disks

RAID config:
- /dev/md0 RAID1
  Mount point: /boot
  Partitions: /dev/sda1, /dev/sdb1
- /dev/md1 RAID1
  Physical device for LVM vg0
  $ mount | grep vg0
  /dev/mapper/vg0-root on / type ext3 (rw,errors=remount-ro)
  /dev/mapper/vg0-tmp on /tmp type ext3 (rw)
  /dev/mapper/vg0-var on /var type ext3 (rw)

I didn't try the "acpi=off" option, but was able to temporarily resolve the
situation in this way:
- multiple boots failed with hoary 2.6.10-5-amd64-k8-smp in the way described below
  - normal, just hit <Enter> boot failed
  - append "init=/bin/bash" boot boot failed
  - append "single" boot failed
- boot from "live" CD, then "watch cat /proc/mdstat" showed /dev/md1 re-syncing
- after the re-sync was complete, I was able to reboot with kernel 2.6.12.2-bef
  without incident (smp kernel)
- then tried booting again from 2.6.10-5-amd64-k8-smp, and also had success

Another point of potential interest is the file "/script" on the initrd, as this
is where the RAID arrays are assembled. It contains the following for my system:

   mdadm -A /devfs/md/1 -R -u 55f3a23c:a0ef4950:a0a11bfe:250e3f63 /dev/sda2
/dev/sdb2
   mkdir /devfs/vg0
   mount_tmpfs /var
   if [ -f /etc/lvm/lvm.conf ]; then
   cat /etc/lvm/lvm.conf > /var/lvm.conf
   fi
   mount_tmpfs /etc/lvm
   if [ -f /var/lvm.conf ]; then
   cat /var/lvm.conf > /etc/lvm/lvm.conf
   fi
   mount -nt devfs devfs /dev
   vgchange -a y vg0
   umount /dev
   umount -n /var
   umount -n /etc/lvm
   ROOT=/dev/mapper/vg0-root
   mdadm -A /devfs/md/1 -R -u 55f3a23c:a0ef4950:a0a11bfe:250e3f63 /dev/sda2
/dev/sdb2

The machine is in use now, and I am unable to perform further tests on it.
However, I will be receiving another one soon (identical, I believe), and will
be able to re-create
this problem and do further testing on it. Please let me know if there are
tests you would like me to perform.

[Matt Zimmerman (mdz)]

Can you describe the way in which the boot failed?

[Ben Collins (ben-collins)]

Apparently it was hanging trying to complete the sw raid rebuild, correct?
Probably would have completed booting if had just been left to complete that task?

Anyway, I belive this to be the case. But I think a message to the console would
help alleviate this. Perhaps initramfs needs to juice up the script some.

[Jeff Bailey (jbailey)]

Hmm. I just call mdrun, so it's a bit of a black box to me right now. Which
tool would be best to get the info from? It might be best to put it right into
mdrun anyway.

[Brian Elliott Finley (finley)]

> Apparently it was hanging trying to complete the sw raid rebuild, correct?

This is my best guess, but there was no output so I can't say for sure. I can
say that the machine was not wedged, as it did respond to my keystrokes (Ie.:
Enter, etc. moved the cursor, but didn't provoke any other interaction).

> Probably would have completed booting if had just been left to complete that task?

Also my best guess, but can't say.

> I think a message to the console would help alleviate this.

Yes. If I had a rebuild status (frequently updated until completion) on the
console, this would have been a non-issue (assuming the two guesses above are true).

> Which tool would be best to get the info from?

You could simply do the equivalent of "watch cat /proc/mdstat".

[Jack Wasey (jackwasey)]

I have a completely different setup, but the same problem.

Resync gets to about 75% of 300Gb in RAID1 on two SATA disks. The crash brings down the whole machine with no logging.

The only slightly unusual thing is that the output of mdadm --examine /dev/md0 shows that the number of the two devices sda5 and sdb5 are not "0" and "1", but "2" and "0". This happened after I booted a gutsy kernel, it assigned one drive to sdc, and despite UUIDs, this messed up the array, so I added sdc5 instead of sdb5. I then switched back to feisty kernel, and sda reappeared, but again not in the array (now composed of sdb and sdc), and again degraded.

Finally, I readded sda, so I should have been back to the original situation, however, I now have sda listed as drive "2" (as shown in square brackets on first line of mdadm --examine /dev/md0 output) and sdb as drive "0".

This is not good, as it now hangs the machine completely between 60% and 80% resync. I'm now running a second resync using 'watch', so I'll see if it is at exactly the same spot that it crashes.

[Brian Elliott Finley (finley)]

For what it's worth, upgrading to the next release solved my problem.
It included improved initrd functionality, and the problem "just went away".

Cheers, -Brian

Waster wrote:
> I have a completely different setup, but the same problem.
>
>
> Resync gets to about 75% of 300Gb in RAID1 on two SATA disks. The crash brings down the whole machine with no logging.
>
> The only slightly unusual thing is that the output of mdadm --examine
> /dev/md0 shows that the number of the two devices sda5 and sdb5 are not
> "0" and "1", but "2" and "0". This happened after I booted a gutsy
> kernel, it assigned one drive to sdc, and despite UUIDs, this messed up
> the array, so I added sdc5 instead of sdb5. I then switched back to
> feisty kernel, and sda reappeared, but again not in the array (now
> composed of sdb and sdc), and again degraded.
>
>
> Finally, I readded sda, so I should have been back to the original situation, however, I now have sda listed as drive "2" (as shown in square brackets on first line of mdadm --examine /dev/md0 output) and sdb as drive "0".
>
> This is not good, as it now hangs the machine completely between 60% and
> 80% resync. I'm now running a second resync using 'watch', so I'll see
> if it is at exactly the same spot that it crashes.
>
>

--
Brian Elliott Finley
Linux Strategist
Argonne National Lab, CIS
Office: 630.252.4742
Mobile: [phone]

[Jeremy Mikola (jmikola)]

Waster wrote:
>I have a completely different setup, but the same problem.
>
>Resync gets to about 75% of 300Gb in RAID1 on two SATA disks. The crash brings down the whole machine with no logging.

Waster wrote:
>This is not good, as it now hangs the machine completely between 60% and 80% resync. I'm now running a second resync using 'watch', so I'll see if it is at exactly the same spot that it crashes.

I'm having the same problem using Ubuntu 7.04 while rebuilding a 4-disk Raid 5 array. The array originally became out of sync after a system crash about two months ago (due to some previous bug that has since been fixed in Ubuntu), and I have not been able to rebuild the array since. I was able to start it in degraded mode with 3 drives, and backup my data (thankfully), but any attempts to resync leads to what you described above. I can leave the machine on with "watch" polling the output of "/proc/mdstat", and I noticed that it crashes in the 70% range. No logging is generated.

After backing up my data, I decided to wipe the md superblocks from the drives and start fresh by creating a new raid 5 array. Unfortunately, I'm experiencing the same problem when constructing the array as it does the initial sync.

I'm lead to believe this is a software problem, as tests from smartctl and Western Digital's own boot disk show no problems with the drives (250gb SATA's - WD2500KS-00MJB0).

[Dustin Kirkland  (kirkland)]

Hello-

Can anyone retest and confirm this data on Intrepid (ideally), or even Hardy?

I believe this to be fixed, as I have a 4-disk RAID5 on a Hardy that is able to re-sync without a problem, and I've tested this in Intrepid virtual machines, which are also able to re-sync without a problem.

I'm going to mark this 'Fix Released', for now. Please reopen if you can confirm this behavior in Hardy or Intrepid.

Thanks!
:-Dustin