Software raid intermittently fails to start at boot time

[Tom Poindexter]

Binary package hint: upstart

This problem began in 10.04, and continues to be a problem in 10.10

I have a software raid configuration, from /etc/mdadm/mdadm.conf:

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=cc4f61b6:e18d4f12:9b71af0b:b5f1b4e8
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=6f28c043:b295a219:1a6d1efe:10bfaf94

which are mounted, from /etc/fstab:

# /dev/md1
UUID=88fcde7d-3cbb-4dfa-b0d9-f93ca9a5b3cd /extramd ext3 relatime 0 2
# /dev/md0
UUID=f033d1d3-4643-42d5-9c64-05df46412129 /home ext3 relatime 0 2

During system boot (about 50% of the time) one or both of the raid devices are not found, and
thus, the mount fails. The situation will often correct itself on a subsequent power-off/power-on or
system reset.

I have this system for several years, and first installed 7.10 on it, with the same software
raid configuration, and upgraded to each Ubuntu release. I had no problems with software
raid failing to start until 10.04 was installed.

I have found a work around that cures the symptom. In /etc/init/mountall.conf
I have added:

   mdadm --assemble --scan

just before the command "exec mountall ....."

I am guessing that this is an 'upstart' problem.

See line 38:

------/etc/int/mountall.conf---------------------------------
# mountall - Mount filesystems on boot
#
# This helper mounts filesystems in the correct order as the devices
# and mountpoints become available.

description "Mount filesystems on boot"

start on startup
stop on starting rcS

expect daemon
task

emits virtual-filesystems
emits local-filesystems
emits remote-filesystems
emits all-swaps
emits filesystem
emits mounting
emits mounted

# temporary, until we have progress indication
# and output capture (next week :p)
console output

script
    . /etc/default/rcS
    [ -f /forcefsck ] && force_fsck="--force-fsck"
    [ "$FSCKFIX" = "yes" ] && fsck_fix="--fsck-fix"

    # set $LANG so that messages appearing in plymouth are translated
    if [ -r /etc/default/locale ]; then
        . /etc/default/locale
        export LANG LANGUAGE LC_MESSAGES LC_ALL
    fi

    # - make sure md devices are up
    mdadm --assemble --scan
    exec mountall --daemon $force_fsck $fsck_fix
end script

post-stop script
    rm -f /forcefsck 2>dev/null || true
end script

lspci output:

00:00.0 Host bridge: ALi Corporation M1695 K8 Northbridge [PCI Express and HyperTransport]
00:01.0 PCI bridge: ALi Corporation PCI Express Root Port
00:02.0 PCI bridge: ALi Corporation PCI Express Root Port
00:04.0 Host bridge: ALi Corporation M1689 K8 Northbridge [Super K8 Single Chip]
00:05.0 PCI bridge: ALi Corporation AGP8X Controller
00:06.0 PCI bridge: ALi Corporation M5249 HTT to PCI Bridge
00:07.0 ISA bridge: ALi Corporation M1563 HyperTransport South Bridge (rev 70)
00:07.1 Bridge: ALi Corporation M7101 Power Management Controller [PMU]
00:11.0 Ethernet controller: ALi Corporation ULi 1689,1573 integrated ethernet. (rev 40)
00:12.0 IDE interface: ALi Corporation M5229 IDE (rev c7)
00:12.1 Mass storage controller: ALi Corporation ULi 5289 SATA (rev 10)
00:13.0 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.1 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.2 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.3 USB Controller: ALi Corporation USB 2.0 Controller (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
03:00.0 VGA compatible controller: nVidia Corporation NV40 [GeForce 6800] (rev a1)
04:06.0 Multimedia audio controller: C-Media Electronics Inc CM8738 (rev 10)
04:07.0 FireWire (IEEE 1394): Agere Systems FW322/323 (rev 61)

lsb_release -rd:

Description: Ubuntu 10.10
Release: 10.10

apt-cache policy upstart:

upstart:
  Installed: 0.6.6-3
  Candidate: 0.6.6-3
  Version table:
 *** 0.6.6-3 0
        500 http://us.archive.ubuntu.com/ubuntu/ maverick/main i386 Packages
        100 /var/lib/dpkg/status

lsmod output:

Module Size Used by
binfmt_misc 6599 1
vboxnetadp 6454 0
vboxnetflt 15184 0
vboxdrv 190231 2 vboxnetadp,vboxnetflt
nls_iso8859_1 3261 1
nls_cp437 4931 1
vfat 9201 1
fat 48240 1 vfat
nfs 275158 0
lockd 65605 1 nfs
fscache 46361 1 nfs
nfs_acl 2257 1 nfs
auth_rpcgss 34001 1 nfs
sunrpc 193114 5 nfs,lockd,nfs_acl,auth_rpcgss
nvidia 9329739 38
snd_cmipci 30469 2
snd_pcm 71475 1 snd_cmipci
snd_page_alloc 7120 1 snd_pcm
snd_opl3_lib 8850 1 snd_cmipci
snd_hwdep 5040 1 snd_opl3_lib
snd_mpu401_uart 5661 1 snd_cmipci
snd_seq_midi 4588 0
snd_rawmidi 17783 2 snd_mpu401_uart,snd_seq_midi
snd_seq_midi_event 6047 1 snd_seq_midi
snd_seq 47174 2 snd_seq_midi,snd_seq_midi_event
snd_timer 19067 3 snd_pcm,snd_opl3_lib,snd_seq
sbp2 19332 0
snd_seq_device 5744 4 snd_opl3_lib,snd_seq_midi,snd_rawmidi,snd_seq
ns558 3068 0
ppdev 5556 0
parport_pc 26058 1
gameport 9327 3 snd_cmipci,ns558
snd 49006 13 snd_cmipci,snd_pcm,snd_opl3_lib,snd_hwdep,snd_mpu401_uart,snd_rawmidi,snd_seq,snd_timer,snd_seq_device
ieee1394 81069 1 sbp2
amd64_agp 7433 1
ali_agp 3729 0
k8temp 3228 0
i2c_ali15x3 5190 0
shpchp 29886 0
soundcore 880 1 snd
agpgart 32011 3 nvidia,amd64_agp,ali_agp
i2c_ali1563 5642 0
i2c_ali1535 4865 0
lp 7342 0
parport 31492 3 ppdev,parport_pc,lp
raid10 22394 0
raid456 53514 0
async_raid6_recov 4883 1 raid456
async_pq 3038 2 raid456,async_raid6_recov
raid6_pq 80029 2 async_raid6_recov,async_pq
async_xor 2298 3 raid456,async_raid6_recov,async_pq
async_memcpy 1045 2 raid456,async_raid6_recov
async_tx 2135 5 raid456,async_raid6_recov,async_pq,async_xor,async_memcpy
raid1 20969 2
raid0 8547 0
multipath 5955 0
linear 3822 0
usbhid 36882 0
dm_raid45 81721 0
firewire_ohci 21106 0
hid 67742 1 usbhid
floppy 54311 0
firewire_core 46643 1 firewire_ohci
sata_uli 2573 11
xor 15136 2 async_xor,dm_raid45
crc_itu_t 1383 1 firewire_core
uli526x 13298 0
pata_ali 7976 0

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: upstart 0.6.6-3
ProcVersionSignature: Ubuntu 2.6.35-23.40-generic 2.6.35.7
Uname: Linux 2.6.35-23-generic i686
NonfreeKernelModules: nvidia
Architecture: i386
Date: Tue Nov 30 21:50:47 2010
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.utf8
 SHELL=/bin/bash
SourcePackage: upstart

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jean-Philippe Guérard (fevrier)]

I've had this issue on 2 different computers. Including after a fresh Ubuntu install.

It's a timing problem. Sometimes, the system won't boot at all, including after several restarts.