Boot hang, apparently blocking on SW RAID rebuild

Bug #22673 reported by Brian Elliott Finley on 2005-09-27
8
Affects Status Importance Assigned to Milestone
initramfs-tools (Ubuntu)
Medium
Unassigned

Bug Description

I am experiencing this issue on a new Sun Fire V40z Server in a well cooled
machine room. (please refer to http://bugzilla.ubuntu.com/show_bug.cgi?id=10916)

Hardware:
- Quad CPU amd64 -- AMD Opteron(tm) Processor 852
- 32G memory
- 2x SCSI disks

RAID config:
- /dev/md0 RAID1
  Mount point: /boot
  Partitions: /dev/sda1, /dev/sdb1
- /dev/md1 RAID1
  Physical device for LVM vg0
  $ mount | grep vg0
  /dev/mapper/vg0-root on / type ext3 (rw,errors=remount-ro)
  /dev/mapper/vg0-tmp on /tmp type ext3 (rw)
  /dev/mapper/vg0-var on /var type ext3 (rw)

I didn't try the "acpi=off" option, but was able to temporarily resolve the
situation in this way:
- multiple boots failed with hoary 2.6.10-5-amd64-k8-smp in the way described below
  - normal, just hit <Enter> boot failed
  - append "init=/bin/bash" boot boot failed
  - append "single" boot failed
- boot from "live" CD, then "watch cat /proc/mdstat" showed /dev/md1 re-syncing
- after the re-sync was complete, I was able to reboot with kernel 2.6.12.2-bef
  without incident (smp kernel)
- then tried booting again from 2.6.10-5-amd64-k8-smp, and also had success

Another point of potential interest is the file "/script" on the initrd, as this
is where the RAID arrays are assembled. It contains the following for my system:

   mdadm -A /devfs/md/1 -R -u 55f3a23c:a0ef4950:a0a11bfe:250e3f63 /dev/sda2
/dev/sdb2
   mkdir /devfs/vg0
   mount_tmpfs /var
   if [ -f /etc/lvm/lvm.conf ]; then
   cat /etc/lvm/lvm.conf > /var/lvm.conf
   fi
   mount_tmpfs /etc/lvm
   if [ -f /var/lvm.conf ]; then
   cat /var/lvm.conf > /etc/lvm/lvm.conf
   fi
   mount -nt devfs devfs /dev
   vgchange -a y vg0
   umount /dev
   umount -n /var
   umount -n /etc/lvm
   ROOT=/dev/mapper/vg0-root
   mdadm -A /devfs/md/1 -R -u 55f3a23c:a0ef4950:a0a11bfe:250e3f63 /dev/sda2
/dev/sdb2

The machine is in use now, and I am unable to perform further tests on it.
However, I will be receiving another one soon (identical, I believe), and will
be able to re-create
this problem and do further testing on it. Please let me know if there are
tests you would like me to perform.

Matt Zimmerman (mdz) wrote :

Can you describe the way in which the boot failed?

Ben Collins (ben-collins) wrote :

Apparently it was hanging trying to complete the sw raid rebuild, correct?
Probably would have completed booting if had just been left to complete that task?

Anyway, I belive this to be the case. But I think a message to the console would
help alleviate this. Perhaps initramfs needs to juice up the script some.

Jeff Bailey (jbailey) wrote :

Hmm. I just call mdrun, so it's a bit of a black box to me right now. Which
tool would be best to get the info from? It might be best to put it right into
mdrun anyway.

Brian Elliott Finley (finley) wrote :

> Apparently it was hanging trying to complete the sw raid rebuild, correct?

This is my best guess, but there was no output so I can't say for sure. I can
say that the machine was not wedged, as it did respond to my keystrokes (Ie.:
Enter, etc. moved the cursor, but didn't provoke any other interaction).

> Probably would have completed booting if had just been left to complete that task?

Also my best guess, but can't say.

> I think a message to the console would help alleviate this.

Yes. If I had a rebuild status (frequently updated until completion) on the
console, this would have been a non-issue (assuming the two guesses above are true).

> Which tool would be best to get the info from?

You could simply do the equivalent of "watch cat /proc/mdstat".

Jack Wasey (jackwasey) wrote :

I have a completely different setup, but the same problem.

Resync gets to about 75% of 300Gb in RAID1 on two SATA disks. The crash brings down the whole machine with no logging.

The only slightly unusual thing is that the output of mdadm --examine /dev/md0 shows that the number of the two devices sda5 and sdb5 are not "0" and "1", but "2" and "0". This happened after I booted a gutsy kernel, it assigned one drive to sdc, and despite UUIDs, this messed up the array, so I added sdc5 instead of sdb5. I then switched back to feisty kernel, and sda reappeared, but again not in the array (now composed of sdb and sdc), and again degraded.

Finally, I readded sda, so I should have been back to the original situation, however, I now have sda listed as drive "2" (as shown in square brackets on first line of mdadm --examine /dev/md0 output) and sdb as drive "0".

This is not good, as it now hangs the machine completely between 60% and 80% resync. I'm now running a second resync using 'watch', so I'll see if it is at exactly the same spot that it crashes.

For what it's worth, upgrading to the next release solved my problem.
It included improved initrd functionality, and the problem "just went away".

Cheers, -Brian

Waster wrote:
> I have a completely different setup, but the same problem.
>
>
> Resync gets to about 75% of 300Gb in RAID1 on two SATA disks. The crash brings down the whole machine with no logging.
>
> The only slightly unusual thing is that the output of mdadm --examine
> /dev/md0 shows that the number of the two devices sda5 and sdb5 are not
> "0" and "1", but "2" and "0". This happened after I booted a gutsy
> kernel, it assigned one drive to sdc, and despite UUIDs, this messed up
> the array, so I added sdc5 instead of sdb5. I then switched back to
> feisty kernel, and sda reappeared, but again not in the array (now
> composed of sdb and sdc), and again degraded.
>
>
> Finally, I readded sda, so I should have been back to the original situation, however, I now have sda listed as drive "2" (as shown in square brackets on first line of mdadm --examine /dev/md0 output) and sdb as drive "0".
>
> This is not good, as it now hangs the machine completely between 60% and
> 80% resync. I'm now running a second resync using 'watch', so I'll see
> if it is at exactly the same spot that it crashes.
>
>

--
Brian Elliott Finley
Linux Strategist
Argonne National Lab, CIS
Office: 630.252.4742
Mobile: 630.631.6621

Jeremy Mikola (jmikola) wrote :

Waster wrote:
>I have a completely different setup, but the same problem.
>
>Resync gets to about 75% of 300Gb in RAID1 on two SATA disks. The crash brings down the whole machine with no logging.

Waster wrote:
>This is not good, as it now hangs the machine completely between 60% and 80% resync. I'm now running a second resync using 'watch', so I'll see if it is at exactly the same spot that it crashes.

I'm having the same problem using Ubuntu 7.04 while rebuilding a 4-disk Raid 5 array. The array originally became out of sync after a system crash about two months ago (due to some previous bug that has since been fixed in Ubuntu), and I have not been able to rebuild the array since. I was able to start it in degraded mode with 3 drives, and backup my data (thankfully), but any attempts to resync leads to what you described above. I can leave the machine on with "watch" polling the output of "/proc/mdstat", and I noticed that it crashes in the 70% range. No logging is generated.

After backing up my data, I decided to wipe the md superblocks from the drives and start fresh by creating a new raid 5 array. Unfortunately, I'm experiencing the same problem when constructing the array as it does the initial sync.

I'm lead to believe this is a software problem, as tests from smartctl and Western Digital's own boot disk show no problems with the drives (250gb SATA's - WD2500KS-00MJB0).

Adam Conrad (adconrad) on 2008-01-23
Changed in initramfs-tools:
assignee: adconrad → nobody
Dustin Kirkland  (kirkland) wrote :

Hello-

Can anyone retest and confirm this data on Intrepid (ideally), or even Hardy?

I believe this to be fixed, as I have a 4-disk RAID5 on a Hardy that is able to re-sync without a problem, and I've tested this in Intrepid virtual machines, which are also able to re-sync without a problem.

I'm going to mark this 'Fix Released', for now. Please reopen if you can confirm this behavior in Hardy or Intrepid.

Thanks!
:-Dustin

Changed in initramfs-tools:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.