Hardy kernel panic after upgrade

[karaluh]

I've got a problem with the new kernel. After upgrade Gutsy->Hardy which went fine reboot welcomes me with Kernel panic - not syncing: VFS: unable to mount root on unknown-block(0,0). Trying to boot Gutsy kernel ends in BusyBox. Quick look around in it showed no hard drive devices (sda/sdb/hdc/hdd in /dev). I did boot with System Rescue CD, mounted /boot and rolled back initrd to the one generated during gutsy install for 2.6.22. I successfully booted Gutsy kernel then. update-initramfs successfully actualized initrd for 2.6.22 so there has to be some problem with the kernel. I've tried to reinstall it, install 2.6.22-16-386, server and pci=nomsi option with kernel panic everytime. Ubuntu Live CD recognizes all drives correctly, but overwriting 2.6.24 kernel with the one from Live CD doesn't help. Recompilation of Hardy kernel doesnt either. This bug is also valid for vanilla 2.6.24.3 and 2.6.25 kernels. I've got lvm on software raid, but that shouldn't matter here. System installed on sata drives, and besides those two, I've got one hdd and dvd-rw on pata. Chipset is nforce2, sata controller PDC20376.

[Michaël Witrant (mike-lepton)]

I had the same problem : sata disk on nforce2, gutsy->hardy then unable to mount root at boot.

The problem was in menu.lst: the initrd line was missing for the new kernel.
Previous kernel (from gutsy) had a valid initrd line and was working fine, but hardy kernel didn't have any initrd line.

I simply ran update-grub and the kernel booted. The only diff is the initrd line.

[karaluh (karaluh)]

How could I be so blind and/or stupid? I owe you beer or two, Michael :-). Thank you.

[Rich (rincebrain)]

I agree that this is silly, but I feel like dist-upgrade doing the wrong thing here is still not okay.

(I had this problem too. SiS chipset, PATA disk.)

[Michaël Witrant (mike-lepton)]

Yes. There's an easy fix, but the bug is still there.

And it is not the first time I've to run update-grub to boot my upgraded ubuntu.

Should we reopen the bug and its package to "upgrade system" or something ?
We can't reproduce it...

[André Heynatz (ombatar)]

It would be nice to let it get fixed. The user should not be required to know about "update-grub" at all. Usability is an area where MacOS and Windows still are superior, and according to bug #1 about Windows market share we should do everything to make using Ubuntu a no-brainer. The user should be able to update his system without need for intervention. A solid fix would be to do "update-grub" as last step in the Ubuntu Upgrade component.

[Michaël Witrant (mike-lepton)]

Reopening the bug

[Lindsay (fmouse)]

This bug is still outstanding as of July 22, and has hit me as well. Please see my report at http://ubuntuforums.org/showthread.php?p=5440237#post5440237

[Lindsay (fmouse)]

I should also note here that I'm tracking the "universe / multiverse" packages in my sources.list. Perhaps this apparently incomplete kernel install comes from that repository, in which case it may be beyond the scope of the Ubuntu bug trackers.

[Lindsay (fmouse)]

My bad. The 2.6.24.3 kernel in my case was apparently not a stock Ubuntu kernel, but one I probably compiled myself. During a recent upgrade, some process or other probably ran update-grub and several unwanted stanzas got inserted into menu.lst.

[Michael Vogt (mvo)]

Please attach the files in /var/log/dist-upgrade to this bugreport.
 Thanks Michael

[Michael Vogt (mvo)]

Here is what is causing the dpkg error during the upgrade:

Paramétrage de gforge-common (4.6.99+svn6347-1) ...^M
Calculating defaults^M
Reading defaults from /etc/gforge/gforge.conf^M
[: 104: ==: unexpected operator^M
Creating /etc/gforge/gforge.conf^M
[: 107: ==: unexpected operator^M
[: 107: ==: unexpected operator^M
[: 107: ==: unexpected operator^M
SSL Enabled^M
Creating /etc/gforge/httpd.conf^M
Creating /etc/gforge/httpd.secrets^M
Creating /etc/gforge/local.inc^M
Creating other includes^M
^M
Paramétrage de postgresql-common (87) ...^M
Installation de la nouvelle version du fichier de configuration /etc/postgresql-common/autovacuum.conf ...^M
Warning: The following devices contain databases and have write^M
caching enabled: /dev/hda^M
This could destroy the integrity of your databases in the event of power^M
failure. Consider disabling the write cache with "hdparm -W 0 <device>".^M
^M
Paramétrage de postgresql-8.2 (8.2.7-1) ...^M
Creating new cluster (configuration: /etc/postgresql/8.2/main, data: /var/lib/postgresql/8.2/main)...^M
Moving configuration file /var/lib/postgresql/8.2/main/postgresql.conf to /etc/postgresql/8.2/main...^M
Moving configuration file /var/lib/postgresql/8.2/main/pg_hba.conf to /etc/postgresql/8.2/main...^M
Moving configuration file /var/lib/postgresql/8.2/main/pg_ident.conf to /etc/postgresql/8.2/main...^M
Configuring postgresql.conf to use port 5433...^M
 * Starting PostgreSQL 8.2 database server^M
   ...done.^M
^M
Paramétrage de gforge-db-postgresql (4.6.99+svn6347-1) ...^M
Installation de la nouvelle version du fichier de configuration /etc/cron.d/gforge-db-postgresql ...^M
Calculating defaults^M
Reading defaults from /etc/gforge/gforge.conf^M
[: 104: ==: unexpected operator^M
Creating /etc/gforge/gforge.conf^M
[: 107: ==: unexpected operator^M
[: 107: ==: unexpected operator^M
[: 107: ==: unexpected operator^M
SSL Enabled^M
Creating /etc/gforge/httpd.conf^M
Creating /etc/gforge/httpd.secrets^M
Creating /etc/gforge/local.inc^M
Creating other includes^M
Replacing config file /etc/postgresql/7.4/main/pg_hba.conf with new version^M
 * Reloading PostgreSQL 7.4 database server^M
   ...done.^M
Procedural language on gforge already enabled^M
You'll see some debugging info during this installation.^M
Do not worry unless told otherwise.^M
Upgrading with 20050812.sql^M
Updating debian_meta_data table.^M
Committing.^M
Upgrading with 20050822.sql^M
Updating debian_meta_data table.^M
Committing.^M
Upgrading with 20050823.sql^M
Updating debian_meta_data table.^M
Committing.^M
Upgrading with 20050824.sql^M
Updating debian_meta_data table.^M
Committing.^M
Upgrading with 20050831.sql^M
Updating debian_meta_data table.^M
Committing.^M
Upgrading with 20060113.sql^M
Updating debian_meta_data table.^M
Committing.^M
Upgrading with 20060214.sql^M
Updating debian_meta_data table.^M
Committing.^M
Upgrading with 20060216-nocommit.sql^M
Updating debian_meta_data table.^M
Committing.^M
Fixing past mistakes in role naming^M
Updating debian_meta_data table.^M
Committing.^M
Upgrading with 20051103_transiciel_motscle_document.sql^M
Updating debian_meta_data table.^M
Committing.^M
Upgrading with 20070924-forum-perm.sql^M
DBD::Pg::st exec...

[Michael Vogt (mvo)]

The kernel upgrade seems to have worked, so if the system does not boot anymore, then this is clearly a kernel bug too.

[Michaël Witrant (mike-lepton)]

The upgrade *did* work. I remember having troubles with gforge, but I fixed them (I've probably removed the package). I don't think it's related to the current issue.

The upgrade process worked, but the upgraded system didn't boot because of the missing Initrd line.

[Michael Vogt (mvo)]

Thanks, I reassign to grub if that has failed to write a correct menu.lst

[Steve Langasek (vorlon)]

update-grub handles the case where an initrd doesn't exist by providing a kernel boot stanza without an initrd line. There are users who have kernels that don't use initrds, and update-grub doesn't have enough information to distinguish this use case from the case where a kernel package has failed to create an initrd image.

If the kernel package listed itself as 'installed', but the initrd was missing, this is a bug in the kernel package. Was the kernel in state 'installed' at the end of this upgrade?

[Michaël Witrant (mike-lepton)]

How can I find this information now ?

[Michaël Witrant (mike-lepton)]

I've upgraded to 8.10 today and it's asking me to reboot the system.

Again, there's no initrd line for the new kernels in /boot/grub/menu.lst
Here's the first auto generated kernel entry:
title Ubuntu 8.10, kernel 2.6.27-7-generic
root (hd0,2)
kernel /boot/vmlinuz-2.6.27-7-generic root=UUID=924b1a6f-3d78-4fac-9957-55ffd2ddd0f8 ro
savedefault

The kernel seems installed:
$ dpkg -l |grep linux-image-2.6.27-7-generic
ii linux-image-2.6.27-7-generic 2.6.27-7.16

The initrd image exists:
$ ls -l /boot/*-2.6.27-7-generic
-rw-r--r-- 1 root root 507665 2008-11-04 22:00 /boot/abi-2.6.27-7-generic
-rw-r--r-- 1 root root 91364 2008-11-04 22:00 /boot/config-2.6.27-7-generic
-rw-r--r-- 1 root root 9261441 2008-11-13 21:30 /boot/initrd.img-2.6.27-7-generic
-rw-r--r-- 1 root root 1029585 2008-11-04 22:00 /boot/System.map-2.6.27-7-generic
-rw-r--r-- 1 root root 1073 2008-11-04 22:02 /boot/vmcoreinfo-2.6.27-7-generic
-rw-r--r-- 1 root root 2244464 2008-11-04 22:00 /boot/vmlinuz-2.6.27-7-generic

According to term.log, update-grub was only run once, when memtest86 was updated. It was not run after the kernel install (ie, after "update-initramfs: Generating /boot/initrd.img-2.6.27-7-generic")

I've just checked kernel-img.conf:
$ cat /etc/kernel-img.conf
do_symlinks = yes
relative_links = yes
do_bootloader = no
do_bootfloppy = no
do_initrd = yes
link_in_boot = no

Looks like I disabled grub update... So the missing update-grub is right. I will enable it again (I don't remember exactly why I did that).

But the update-grub run on memtest86 is probably wrong. Should we disable it when do_bootloader is "no"? Or should we run it after the kernel images are fully installed?

Right now, upgrading at the same time memtest86 and a kernel image, while having do_bootloader set to "no" will leave the system unbootable.

I'll let the system in this state for some hours, if anyone want fresh data. But then I'll run update-grub, reboot and use it again.

[Michaël Witrant (mike-lepton)]

Also, bug #222421 looks similar to this one.

[Michaël Witrant (mike-lepton)]

After further investigation in the memtest86+ package, I discovered the bug affecting me has already been fixed.

The update-grub command run during my upgrade process was done by the old package after it was removed (by the postrm file). The new memtest86+ package doesn't run update-grub any more.

See also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=460080

So, this bug seems fixed for future upgrades. But any other user upgrading to 8.10 with memtest86+ and do_bootloader=no will encounter the same problem and will have a unbootable system until he runs update-grub manually.

[Steve Langasek (vorlon)]

thanks for the follow-up, Michael. Since this does appear to be a problem with interaction between memtest86+ and a non-default kernel-img.conf, I'm reassigning to memtest86+ and marking as resolved.

I'm rather concerned by the reason that it's resolved, though, since memtest86+ still provides integration for grub2 - just not for grub1; so this memtest86+ bug may resurface later.

[Michaël Witrant (mike-lepton)]

Actually, it's resolved for users who have already switched to 8.10. Anyone running a previous version may still be affected.