11.04 and up cannot start from RAID/LVM2

[rihad]

I have two identical disks set up in a fake RAID configuration through the PC BIOS, over which a LVM2 volume where all my partitions live except /boot, which is on its dedicated RAID partition.

After the upgrade to 11.04, when doing a cold restart, /dev/dm-* files aren't found, so the booting halts with an error. Stock 11.04 kernel is used as per uname:
Linux rihad 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

Apparently the new kernel also messes up the RAID configuration, because the array members are then shown as OFFLINE in the BIOS making the disks unusable, and nothing short of a cold restart brings them back to life.

Fortunately I was wise enough to have saved the previous distro's kernel vmlinuz-2.6.35-28-generic in my GRUB menu, booting into which can correctly initialize the LVM2/DM subsystem, and then after doing a warm reboot into the newer kernel everything works. So I must do a two-phased boot every morning, which sucks.

Problem persists with:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2-rc1-oneiric/linux-image-3.2.0-030200rc1-generic_3.2.0-030200rc1.201111071935_amd64.deb

This is a kernel regression:
bad: 2.6.37-12.26
good: 2.6.37-11.25

---
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 1.23-0ubuntu3
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: HDA Generic [HDA Generic]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: rihad 3074 F.... pulseaudio
 /dev/snd/controlC0: rihad 3074 F.... pulseaudio
 /dev/snd/pcmC0D0p: rihad 3074 F...m pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xfb100000 irq 46'
   Mixer name : 'Realtek ID 885'
   Components : 'HDA:10ec0885,1458a002,00100101'
   Controls : 15
   Simple ctrls : 8
Card1.Amixer.info:
 Card hw:1 'NVidia'/'HDA NVidia at 0xf6000000 irq 17'
   Mixer name : 'Nvidia GPU 0d HDMI/DP'
   Components : 'HDA:10de000d,10de0101,00100100'
   Controls : 16
   Simple ctrls : 4
DistroRelease: Ubuntu 11.10
InstallationMedia: Kubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
MachineType: Gigabyte Technology Co., Ltd. P35C-DS3R
NonfreeKernelModules: nvidia
Package: linux (not installed)
ProcEnviron:
 LANGUAGE=
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: root=/dev/mapper/vg0-root ro quiet splash
ProcVersionSignature: Ubuntu 3.0.0-12.20-generic 3.0.4
RelatedPackageVersions:
 linux-restricted-modules-3.0.0-12-generic N/A
 linux-backports-modules-3.0.0-12-generic N/A
 linux-firmware 1.60
RfKill:

Tags: oneiric
Uname: Linux 3.0.0-12-generic x86_64
UpgradeStatus: Upgraded to oneiric on 2011-10-14 (29 days ago)
UserGroups: adm admin cdrom dialout lpadmin plugdev sambashare tty
dmi.bios.date: 11/16/2007
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: F8
dmi.board.name: P35C-DS3R
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrF8:bd11/16/2007:svnGigabyteTechnologyCo.,Ltd.:pnP35C-DS3R:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnP35C-DS3R:rvr:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: P35C-DS3R
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[rihad (rihad)]

$ ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 236 2011-05-29 10:45 control
lrwxrwxrwx 1 root root 7 2011-05-29 11:04 isw_dacdafdfg_MYDISK -> ../dm-1
lrwxrwxrwx 1 root root 7 2011-05-29 10:45 isw_dacdafdfg_MYDISK1 -> ../dm-2
lrwxrwxrwx 1 root root 7 2011-05-29 10:45 isw_dacdafdfg_MYDISK2 -> ../dm-3
lrwxrwxrwx 1 root root 7 2011-05-29 10:45 isw_dacdafdfg_MYDISK3 -> ../dm-4
lrwxrwxrwx 1 root root 7 2011-05-29 10:45 isw_dacdafdfg_MYDISK4 -> ../dm-5
lrwxrwxrwx 1 root root 7 2011-05-29 10:45 vg0-home -> ../dm-7
lrwxrwxrwx 1 root root 8 2011-05-29 10:46 vg0-migrate -> ../dm-11
lrwxrwxrwx 1 root root 7 2011-05-29 10:45 vg0-root -> ../dm-0
lrwxrwxrwx 1 root root 8 2011-05-29 10:45 vg0-swap -> ../dm-12
lrwxrwxrwx 1 root root 7 2011-05-29 10:45 vg0-tmp -> ../dm-6
lrwxrwxrwx 1 root root 7 2011-05-29 10:45 vg0-usr -> ../dm-9
lrwxrwxrwx 1 root root 7 2011-05-29 10:45 vg0-var -> ../dm-8
lrwxrwxrwx 1 root root 8 2011-05-29 10:46 vg0-vm -> ../dm-10

MYDISK is the "whole" disk cfdisk can be run on to create PC partitions
MYDISK1 is multiboot to WinXP (irrelevant)
MYDISK2 is the Kubuntu boot partition
MYDISK3 is the big LVM2 physical disk that the remaining vg0-* partitions live on
MYDISK4 is multiboot to FreeBSD (irrelevant)

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 880646

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[rihad (rihad)]

apport information

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . If possible, please test the latest v3.2-rcN kernel (Not a kernel in the daily directory). Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag(Only that one tag, please leave the others). This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed by the mainline kernel, please add the following tag 'kernel-fixed-upstream-KERNEL-VERSION'. For example, if kernel version 3.2-rc1 fixed and issue, the tag would be: 'kernel-fixed-upstream-v3.2-rc1'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Thanks in advance.

[rihad (rihad)]

Hi, Joseph, I've just installed this image:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2-rc1-oneiric/linux-image-3.2.0-030200rc1-generic_3.2.0-030200rc1.201111071935_amd64.deb

but the same problem still persists. Added the "kernel-bug-exists-upstream" tag.
"needs-upstream-testing" wasn't among them.

[rihad (rihad)]

But still seen after the minor upgrade to
linux-headers-3.0.0-13-generic 3.0.0-13.22

[rihad (rihad)]

Bug still present in
linux-image-3.0.0-14-generic 3.0.0-14.23 Linux kernel image for version 3.0.0 on x86/x86_64

[rihad (rihad)]

Perhaps Ubuntu kernel config after 2.6.35-28 started setting an incompatible option in its "one-for-all" type of kernel config that fails to initialize LVM2/DM? Maybe I should try building my own lean and mean custom kernel as I used to back in Debian?

[rihad (rihad)]

But still present with stock kernel 3.2.8. I think this is a bug in [K]ubuntu because latest Debian boots LVM2 w/ kernel 3.2.8 perfectly.
Maybe the OS fails to include DM modules in the initrd image?

[penalvch (penalvch)]

rihad, thank you for reporting this and helping make Ubuntu better. This bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? Can you try with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you run the following command in the development release from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux <replace-with-bug-number>

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.

Please let us know your results. Thanks in advance.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[rihad (rihad)]

I'm afraid the bug still exists in latest upstream linux-image-3.0.0-22-generic
Something broke after version linux-image-2.6.35-28-generic.
So I still need to keep both versions around: first, boot into 2.6.35. It initalizes FakeRAID hardware properly. Then, reboot into 3.0.0.
I'm marking it as active.

[penalvch (penalvch)]

rihad, could you please test http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.5-rc6-quantal/ ?

[rihad (rihad)]

Same ole. Tried them just now:
ii linux-headers-3.5.0-030500rc6 3.5.0-030500rc6.201207072135 Header files related to Linux kernel version 3.5.0
iU linux-headers-3.5.0-030500rc6-generic 3.5.0-030500rc6.201207072135 Linux kernel headers for version 3.5.0 on 64 bit x86 SMP
ii linux-image-3.5.0-030500rc6-generic 3.5.0-030500rc6.201207072135 Linux kernel image for version 3.5.0 on 64 bit x86 SMP

/dev/mapper/* is still empty (except for "control") when first booting into 3.5.

[penalvch (penalvch)]

rihad, the next step is to perform a kernel bisect from 2.6.35-28-generic to 2.6.38-8-generic, in order to identify the offending commit(s). Could you please do this following https://wiki.ubuntu.com/Kernel/KernelBisection ?

[rihad (rihad)]

Uhm, I'm having a hard time pulling the right commits.
Working on ubuntu-natty clone:
$ git log --oneline Ubuntu-2.6.35-22.33..Ubuntu-2.6.38-8.42
fatal: ambiguous argument 'Ubuntu-2.6.35-22.33..Ubuntu-2.6.38-8.42': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions

Natty lacks 2.6.35-28

[penalvch (penalvch)]

rihad, what I would do is to go through the Ubuntu kernels first via https://launchpad.net/ubuntu/natty/+source/linux, then once you find which two kernels are the good one and the next is the regression, perform the commit bisect.

[rihad (rihad)]

Ok, using binary search I tracked the good and the bad (and the evil) versions in 5-6 attempts. Here they are:
good: 2.6.37-11.25
bad: 2.6.37-12.26

$ git bisect start Ubuntu-2.6.37-12.26 Ubuntu-2.6.37-11.25
Bisecting: a merge base must be tested
[90a8a73c06cc32b609a880d48449d7083327e11a] Linux 2.6.37-rc7
$

As per the howto, git should have told that there are still N bisections to be made, yet it didn't.
Did I do something wrong? I followed all its steps.

Here's the output of "git log --oneline Ubuntu-2.6.37-11.25..Ubuntu-2.6.37-12.26"
http://pastebin.com/v2gtZk3k

[rihad (rihad)]

Posting so this thread doesn't expire...

[penalvch (penalvch)]

rihad, sorry I have not gotten back to you sooner, I am in the middle of moving from one coast to another. If you need help bisecting, you are welcome seek more immediate, granular support via Ubuntu Kernel Team mailinglist or IRC https://wiki.ubuntu.com/KernelTeam#Communications, or via Ubuntu Community support http://www.ubuntu.com/support .

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[rihad (rihad)]

Relaunching the bug report as it is not yet fixed in mainline kernel.

I've tried getting some help at Ubuntu forums as suggested here, but no one has ever replied.
http://ubuntuforums.org/showthread.php?p=12156399

Once again, the bug was born in this release:
bad: 2.6.37-12.26
good: 2.6.37-11.25

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[rihad (rihad)]

Thanks for updating the report. I can confirm the bug is still present in current distro kernel:
Linux rihad 3.0.0-28-generic #45-Ubuntu SMP Wed Nov 14 21:57:26 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

I'm willing to try mainline kernel as soon as an attempt is made at fixing it.

[penalvch (penalvch)]

rihad, regarding your comments https://bugs.launchpad.net/ubuntu/+source/linux/+bug/880646/comments/47 :
>"I'm willing to try mainline kernel as soon as an attempt is made at fixing it."

This is an inverted understanding on how bug reporting works. As per Ubuntu Kernel Team policy, unless a bug reporter has tested for the problem in the mainline kernel, it would not be marked Triaged, or a state in which a developer would begin working on this as per https://wiki.ubuntu.com/Bugs/Status .

Hence, could you please test http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8-rc1-raring/ and comment to the results?

[rihad (rihad)]

Sure, I thought I already tried the latest version in July (!) as per your request in posting 36. I'll let you know later how 3.8 goes. Thanks.

[rihad (rihad)]

vmlinuz-3.8.0-030800rc1-generic hasn't fixed the problem.

[penalvch (penalvch)]

rihad, thank you for testing the newest mainline kernel. Regarding your comments https://bugs.launchpad.net/ubuntu/+source/linux/+bug/880646/comments/45 :
>"I've tried getting some help at Ubuntu forums as suggested here, but no one has ever replied. http://ubuntuforums.org/showthread.php?p=12156399"

I would contact the Ubuntu Kernel Team exclusively via https://wiki.ubuntu.com/KernelTeam#Communications , as they would know how to bisect kernel commits. When doing so, please provide a URL so we may track the communication.

Thank you for your understanding.

[rihad (rihad)]

Thanks, Christopher, I just re-posted the forum question to <email address hidden> mailing list.
This month's archive seems to be here: https://lists.ubuntu.com/archives/kernel-team/2012-December/thread.html

[rihad (rihad)]

And my probem: https://lists.ubuntu.com/archives/kernel-team/2012-December/024257.html

[Joseph Salisbury (jsalisbury)]

Version 2.6.37-11.25 was based off of upstream v2.6.37-rc7.
Version 2.6.37-12.62 was rebased to upstream v2.6.37-rc8 and again to v2.6.37 final.

We first need to identify if v2.6.37-rc8 or v2.6.37 final introduced the regression. Can you test the following two kernels and report back which(Or if both) have the bug:

v2.6.37-rc8: http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.37-rc8-natty/
v2.6.37 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.37-natty/

[rihad (rihad)]

Thanks for chiming in, Joseph. The rc8 version has the bug. I took the liberty to guess there's no need to test the release after that, just let me know if I need to.

[penalvch (penalvch)]

rihad, as per https://bugs.launchpad.net/ubuntu/+source/linux/+bug/880646/comments/54 :
>" Can you test the following two kernels..."

[rihad (rihad)]

Christopher, both have the problem ;-)

[Joseph Salisbury (jsalisbury)]

Thanks for testing, rihad. I assume the bug was introduced in v2.6.37-rc7. Can you test that kernel to confirm it does not have the bug? It can be downloaded from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.37-rc7-natty/

[Joseph Salisbury (jsalisbury)]

I meant the bug was introduced in v2.6.37-rc8 not rc7

[rihad (rihad)]

Hi, Joseph. Sorry, no, rc7 doesn't work either.

[rihad (rihad)]

rc6?

[Joseph Salisbury (jsalisbury)]

Interesting. I would thing rc7 would not have the bug, since 2.6.37-11.25 is based off of rc7.

Can you also test rc6:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.37-rc6-natty/

[rihad (rihad)]

Nope, didn't work out.

[Joseph Salisbury (jsalisbury)]

The may be an Ubuntu SAUCE patch in 2.6.37-11.25 that fixes this and was dropped in 2.6.37-12.26.

Can you test the following kernel and confirm it does not have the bug:
https://launchpad.net/~canonical-kernel-team/+archive/ppa/+build/4186164

That is the latest Lucid(10.04) kernel.

[rihad (rihad)]

Uhm, the link you gave is for 2.6.32. Is it ok?
There's no *.deb file for kernel image, only udeb. Does it matter?

[Joseph Salisbury (jsalisbury)]

Yes, 2.6.32 is ok. I'd like to see if there was an Ubuntu SAUCE patch that fixes this bug and was dropped in 2.6.37-12.26 and if this bug exists in Lucid. The name of the kernel file on that page is:
linux-image-2.6.32-45-generic_2.6.32-45.102_amd64.deb

It can be downloaded directoy from:
https://launchpad.net/~canonical-kernel-team/+archive/ppa/+build/4186164/+files/linux-image-2.6.32-45-generic_2.6.32-45.102_amd64.deb

[rihad (rihad)]

2.6.32-45 seemed to boot fine (judging by disk activity), but it finally froze while showing the blue Kubuntu screen and no disk activity. Perhaps kernel is too old for current FS/software/whatever. Toggling Num Lock on the keyboard was fine, so it wasn't a hard lock up. Ctrl+Alt-F1-F2 etc didn't work. Ctrl+Alt-Del worked and the machine rebooted. RAID members weren't marked as OFFLINE, so it's a good sign.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[penalvch (penalvch)]

rihad, as per http://www.gigabyte.com/products/product-page.aspx?pid=2551#bios an update is available for your BIOS F12. If you update to this, does it change anything?

As well, once complete, please provide the output of the following terminal command:
sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date

Thank you for your understanding.

[rihad (rihad)]

Hi, Christopher, I don't know why, but I just did this risky operation. Now it's ver. F12 (was F8). It didn't change anything, only this time there's an extra annoying 3-second lag before showing up the GRUB menu :) How on earth could this bug be BIOS related if kernel up to 2.6.35, Windows XP & FreeBSD can ALL detect the FakeRAID array?

[A1an (alan-b)]

Just got a regression on RAID boot:

2.6.32-46 works
2.6.32-47 breaks RAID 1 drive

See #1190295

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]