Drive not found since 3.2.0-21... kernel panic, unable to mount root fs

Bug #974765 reported by James Shackleford on 2012-04-06
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
ubiquity (Ubuntu)
High
Unassigned

Bug Description

Since 3.2.0-21 the kernel does not locate my hard drive. Consequently, the kernel cannot locate the system partition, which results in a kernel panic -- unable to mount root filesystem.

The suggested valid boot devices following this error include only my CD-ROM drive (sr0).

Kernel 3.2.0-20 boots fine.

Kernel 3.2.0-22 suffers the same problem as 3.2.0-21.

Please let me know if I can supply any further/more specific information.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-generic 3.2.0.22.24
ProcVersionSignature: Ubuntu 3.2.0-20.33-generic 3.2.12
Uname: Linux 3.2.0-20-generic x86_64
NonfreeKernelModules: nvidia
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0-0ubuntu4
Architecture: amd64
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Thu Apr 5 20:44:16 2012
HibernationDevice: RESUME=UUID=35553fcd-9ae3-4e61-8142-e496f4d885d4
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
MachineType: Dell Inc. Precision WorkStation T7500
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-20-generic root=UUID=862bcf01-fc44-4c29-8d08-f8be78763567 ro quiet splash
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/20/2011
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A10
dmi.board.name: 06FW8P
dmi.board.vendor: Dell Inc.
dmi.board.version: A01
dmi.chassis.type: 7
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA10:bd04/20/2011:svnDellInc.:pnPrecisionWorkStationT7500:pvr:rvnDellInc.:rn06FW8P:rvrA01:cvnDellInc.:ct7:cvr:
dmi.product.name: Precision WorkStation T7500
dmi.sys.vendor: Dell Inc.

James Shackleford (tshack) wrote :

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

You can update to the latest development kernel by simply running the following commands in a terminal window:

    sudo apt-get update
    sudo apt-get dist-upgrade

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

If you want this bot to quit automatically requesting kernel tests, add a tag named: bot-stop-nagging.

 Thank you for your help, we really do appreciate it.

Changed in linux (Ubuntu):
status: New → Confirmed
status: Confirmed → Incomplete
tags: added: kernel-request-3.2.0-22.35
Changed in linux (Ubuntu):
status: Incomplete → Confirmed

James Shackleford, thank you for reporting this and helping make Ubuntu better. Could you please provide the results following https://wiki.ubuntu.com/DebuggingKernelBoot ? As well, if you could also please test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
tags: added: bot-stop-nagging needs-bisect needs-upstream-testing regression-release
removed: kernel-request-3.2.0-22.35
Changed in linux (Ubuntu):
importance: Undecided → High
James Shackleford (tshack) wrote :

Here are the results of the "debugging kernel boot." Due to the nature of the issue, the best I could do is camera shots.

If there is a trick in this situation for obtaining a full boot log, I would be interested to hear.

-------------------------------
Case 1: (default as per https://wiki.ubuntu.com/DebuggingKernelBoot)

recordfail
gfxmode text
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd0,msdos3)'
search --no-floppy --fs-uuid --set=root 862bcf01-fc44-4c29-8d08-f8be78763567
linux /boot/vmlinuz-3.2.0-22-generic root=/dev/sda3 ro

http://www.tshack.net/tmp/debug.jpg [158 KB]

-------------------------------
Case 2: (case 1 with raid=noautodetect)

recordfail
gfxmode text
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd0,msdos3)'
search --no-floppy --fs-uuid --set=root 862bcf01-fc44-4c29-8d08-f8be78763567
linux /boot/vmlinuz-3.2.0-22-generic raid=noautodetect root=/dev/sda3 ro

http://www.tshack.net/tmp/debug_noraidauto.jpg [187 KB]
-------------------------------

Note that using root=UUID=378367a0-4aac-4e08-b078-e708b24ac4d9 does not work either. For kernel 3.2.0-20, update-grub still generates root= using UUID on the kernel line. For 3.2.0.21+ update, root= is more explicitly specified using /dev/xxxyy entries for some reason. Neither specification works for kernels >= 3.2.0.21 on my system.

I will perform an upstream kernel build as soon as time permits.

James Shackleford (tshack) wrote :
James Shackleford (tshack) wrote :

Sorry, the note at the bottom of #4 accidentally included the wrong UUID, but you get the idea.

Corrected post reads as follows:

Note that using root=UUID=862bcf01-fc44-4c29-8d08-f8be78763567 does not work either. For kernel 3.2.0-20, update-grub still generates root= using UUID on the kernel line. For 3.2.0.21+ update, root= is more explicitly specified using /dev/xxxyy entries for some reason. Neither specification works for kernels >= 3.2.0.21 on my system.

James Shackleford (tshack) wrote :

Tried the pre-compiled mainline daily for April 6th 3.4.0 :
(from: http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/current/)
 * linux-headers-3.4.0-999_3.4.0-999.201204060408_all.deb
 * linux-headers-3.4.0-999-generic_3.4.0-999.201204060408_amd64.deb
 * linux-image-3.4.0-999-generic_3.4.0-999.201204060408_amd64.deb

The results are similar. (See attachment)

tags: removed: needs-upstream-testing

James Shackleford, thank you for providing the pictures and testing the mainline kernel. The next step is to perform a kernel bisect to identify the offending commit. Could you please do this following https://wiki.ubuntu.com/Kernel/KernelBisection ?

James Shackleford (tshack) wrote :

Hi Christopher. No problem. However, I am currently traveling so I will not be able to perform the kernel bisection until I return on Tuesday. I apologize for this delay.

James Shackleford (tshack) wrote :

I have performed some more work towards this issue and I have found that it is an issue not with the kernel but with the generation of initrd.img.

Ubuntu-3.2.0-20-generic is the kernel I received upon initial install via CD-ROM. This kernel came with the proper initrd.img-3.2.0-20-generic image. update-grub detects both kernel and ramdisk image correctly, and the system boots fine.

Ubuntu-3.2.0-21-generic and Ubuntu-3.2.0-22-generic will installed via repository (normal system update). initrd.img does not exist for either of these kernels. Consequently, update-grub does not generate initrd lines for these kernels and they fail to mount the system partition due to lack of necessary modules.

I built Ubuntu-3.2.0-23 from the Ubuntu kernel git repository via:
$ fakeroot debian/rules clean
$ fakeroot debian/rules binary-headers binary-generic

and installed the resulting debs:
 * linux-headers-3.2.0-23_3.2.0-23.36_all.deb
 * linux-headers-3.2.0-23-generic_3.2.0-23.36_amd64.deb
 * linux-image-3.2.0-23-generic_3.2.0-23.36_amd64.deb

Again, initrd.img-3.2.0-23-generic was never generated for some reason. So, naturally, booting this also failed.

I was able to get Ubuntu-3.2.0-23-generic to boot by:
 1) making a copy of my working initrd.img-3.2.0-20-generic
 2) unpacking it
 3) rename lib/modules/3.2.0-20-generic to lib/modules/3.2.0-23-generic
 4) repacking
 5) moving to /boot/initrd.img-3.2.0-23-generic

I know this is a terrible hack, but I am not sure how to generate a proper initrd.img by hand.

In short, initrd.img is not being generated after kernel installation by initramfs-tools on my system for some reason. I am not familiar enough with the generation of initrd.img to know where the failure could be. Help is, of course, appreciated in tracking the source of the error.

Regardless, this is no longer really a kernel bug and should probably be reassigned or closed/re-opened as appropriate.

James Shackleford (tshack) wrote :

Yes, I can confirm that on my system the problem is with update-initramfs.

After kernel installation, update-initramfs is invoked via /etc/kernel/postinst.d/initramfs-tools. However, initrd.img is not created. In fact, update-initramfs seems to do nothing.

Running update-initramfs manually via:
$ sudo update-initramfs -c -t -k 3.2.0-21-generic -v

Produces no output.

Running:
$ update-initramfs

also produces no output. Not even the usage as in 10.04 LTS.

However, I am able to use /usr/sbin/update-initramfs from my working 10.04 LTS partition to generate the initrd.img images. So, it looks like a problem with update-initramfs

I tried:
$ sudo apt-get clean
$ sudo apt-get autoclean
$ sudo apt-get --reinstall install initramfs-tools

but this did not seem to change anything.

James Shackleford (tshack) wrote :

Final update.

After further investigation, it seems that update-initramfs on my 12.04 LTS partition was merely a symlink to /bin/true

The "real" update-initramfs was diverted to /usr/sbin/update-initramfs.distrib

I am guessing there was just a hiccup in ubiquity during the install, perhaps?

James Shackleford (tshack) wrote :

Final-Final update.

No guessing. It was ubiquity:

$ sudo dpkg-divert --list
. . .
diversion of /usr/sbin/update-initramfs to /usr/sbin/update-initramfs.distrib by ubiquity
. . .

Changed in linux (Ubuntu):
status: Incomplete → Invalid
Changed in linux (Ubuntu):
status: Invalid → Confirmed
Richard Senior (r-senior) wrote :

I've just done a clean install of 12.04 Beta 2 and got this problem after apt-get dist-upgrade.

I see exactly the same symptoms as James describes with the messed-up grub entries and kernel panic. My /usr/sbin/update-initramfs is also linked to /bin/true.

I've worked around it by choosing the 3.2.0-20 kernel from the grub menu, then I removed the 3.2.0-23 series with Synaptic and ran update-grub.

My laptop has kernel 3.2.0-22 and all looks OK:

richard@calder:/boot$ ls -l /boot
total 45204
-rw-r--r-- 1 root root 789425 Mar 27 18:22 abi-3.2.0-20-generic
-rw-r--r-- 1 root root 791023 Apr 3 20:13 abi-3.2.0-22-generic
-rw-r--r-- 1 root root 140211 Mar 27 18:22 config-3.2.0-20-generic
-rw-r--r-- 1 root root 140279 Apr 3 20:13 config-3.2.0-22-generic
drwxr-xr-x 3 root root 12288 Apr 10 12:23 grub
-rw-r--r-- 1 root root 14159604 Apr 6 02:13 initrd.img-3.2.0-20-generic
-rw-r--r-- 1 root root 14168869 Apr 11 17:12 initrd.img-3.2.0-22-generic
-rw-r--r-- 1 root root 176764 Nov 27 10:00 memtest86+.bin
-rw-r--r-- 1 root root 178944 Nov 27 10:00 memtest86+_multiboot.bin
-rw------- 1 root root 2884032 Mar 27 18:22 System.map-3.2.0-20-generic
-rw------- 1 root root 2884270 Apr 3 20:13 System.map-3.2.0-22-generic
-rw------- 1 root root 4965584 Mar 27 18:22 vmlinuz-3.2.0-20-generic
-rw------- 1 root root 4966000 Apr 3 20:13 vmlinuz-3.2.0-22-generic
richard@calder:/boot$ ls -l /usr/sbin/update-initramfs
-rwxr-xr-x 1 root root 9164 Sep 19 2011 /usr/sbin/update-initramfs

Richard Senior (r-senior) wrote :

So, given that my laptop is OK and fully up to date, the fact I got the problem on another machine by doing dist-upgrade suggests the problem is in this update:

The following packages have been kept back:
  linux-generic linux-headers-generic linux-image-generic
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

James Shackleford (tshack) wrote :

For reference, I will post my solution as well.

First, I undid the divert performed by ubiquity via:
$ sudo rm /usr/sbin/update-initramfs
$ sudo dpkg-divert --rename --remove /usr/sbin/update-initramfs

Then I was able to generate initrd images for all kernels via:
$ sudo update-initramfs -c -t -k all

And then updated grub as usual:
$ sudo update-grub

I have since pulled down a kernel via "Update Manager" and update-initramfs is now being triggered as intended by /etc/kernel/postinst.d/initramfs-tools, which now generates working boot configurations.

Richard Senior (r-senior) wrote :

I've followed these two steps of James's solution

$ sudo rm /usr/sbin/update-initramfs
$ sudo dpkg-divert --rename --remove /usr/sbin/update-initramfs

and can confirm that my problem machine also boots correctly with the 3.2.0-23 kernel reinstated by reinstalling linux-generic, linux-headers-generic and linux-image-generic.

James Shackleford (tshack) wrote :

I am removing the 'needs-bisect' tag.

It should be noted that in Post #10 I only mentioned building 3.2.0-23, but I also built 3.2.0-9, 3.2.0-20, 3.2.0-21, and 3.2.0-22. All of these build failed to boot due to lack of initrd.img. Naturally, the bisection failed because I could not find a "good" version since the problem was related to update-initranfs and initrd.img generation... not the kernel.

Somebody more knowledgeable than myself should probably move this bug somewhere more appropriate (ubiquity, perhaps?) since it has been shown that this is not a kernel issue, despite being kernel related.

tags: removed: needs-bisect
Richard Senior (r-senior) wrote :

Agree with James that this looks like an installer issue. On my fresh install everything was OK. It was only when I triggered an update that introduced a new kernel that the problem with update-initramfs became apparent.

Switched to ubiquity (Ubuntu)

affects: linux (Ubuntu) → ubiquity (Ubuntu)
Richard Senior (r-senior) wrote :

I just downloaded a fresh version of the Precise Beta 2 ISO and re-installed on my problem machine. I did the same dist-upgrade that I did yesterday and all is well. No issues with update-initramfs.

I'd suggest that this is most likely a problem with one of the old ISOs and this bug should be marked invalid if James agrees?

James Shackleford (tshack) wrote :

Richard, thank you for your contributions. Good to know I was not the only person with this bug.

I agree. Setting this bug back to invalid.

Changed in ubiquity (Ubuntu):
status: Confirmed → Invalid
Brian Ronald (brianetta) wrote :

I just got hit by this bug after installing the release. I haven't been using any betas at all.

Brian Ronald (brianetta) wrote :

To clarify: I used the Xubuntu installer, after the prompted upgrade crashed half way through, and that's when I encountered this problem.

Changed in ubiquity (Ubuntu):
status: Invalid → Confirmed
To post a comment you must log in.