Bug #222421 “initrd not configured in menu.lst after upgrade fro...” : Bugs : grub package : Ubuntu

psl (slansky) on 2008-04-26

description:

updated

Revision history for this message

Steve Langasek (vorlon) wrote on 2008-04-30:

#1

Hi psl,

Thank you for taking the time to report this bug and help to improve Ubuntu.

Can you tell me whether /boot/initrd-2.6.24-16-generic exists on your system? The only case I see where update-grub would fail to add the initrd line to menu.lst is if the initrd itself doesn't exist for some reason.

Changed in grub:
status:	New → Incomplete

Revision history for this message

psl (slansky) wrote on 2008-05-01:

#2

Hi Steve,

you are right, initrd-2.6.24-16-generic was ready in the system, ONLY record for initrd in menu.lst was missing. I added initrd line to menu.lst and system started as expected.

Revision history for this message

Paul Taylor (paul-taylor-london) wrote on 2008-05-02:

#3

part of /var/log/installer/syslog Edit (13.1 KiB, text/plain)

Xubuntu 8.04 (24 April 2008) CD image downloaded

My only differences from "standard" set-up:
(1) Old hardware
(2) DIY disk partitioning.

On completion of the intallation, it failed to boot, with messages like this:
VFS: cannot open root deveice "UUID=<hex stuff>" or unknown block (0,0)
Please append a correct "root=" boot option; here are the available partitions:
kernel panic - not syncing: VFS: unable to mount root fs on unknown-block (0,0)

Eventually I worked out that
- the "initrd" command is missing from /boot/grub/menu.lst
- the file /boot/initrd.img-2.6.24-16-generic is missing
(actially, it's there, but called /boot/initrd.img-2.6.24-16-generic.bak)
After correcting this, the new system booted correctly.

Another thing that confused matters was that my internal IDE disk
that had been called /dev/hda was now called /dev/sda which I thought
meant a SCSI disk.

On further investigation, I found that the "installer" failed to create initrd.img
The relevant part of /var/log/installer/syslog is attached.

For the enlightenment of novices like me:
"grub" is part of the bootstrapping process, and loads the operating system kernel;
to do this, it needs the executable binary of the kernel itself
and also a minimal filesystem, which is what initrd.img provides
(this is a gzip-compressed cpio-image of /bin, /conf, /etc, /init, /lib, /modules, /sbin, /scripts, /usr and /var)
There are tools such as update_initramfs and others to create this image,
but I could not work out how to do so unless you already have the target filesystem in place,
which seems to require the target kernel too; in particular, I don't know how to set things
up to use chroot.

Since there is an initrd.img...bak file, somebody was presumably aware of the possibility
that creation of the new initrd.img might fail during the installation process. There
should be a check for this, and in this case the .bak file should be installed.

As psl says, this is a critical bug, and one that could easily put people off using Linux for life.
Ubuntu sells itself as "Linux for human beings", which this kind of failure clearly is not.

However, what makes me angry about this is not the bug itself, but the enthusiasm that
the installation cd has for trashing my disk and pre-existing system (which was Ubuntu 6.10).
Even though I had backed everything up, it took me about 20 hours work from giving the
go-ahead to the Xubuntu 8.04 installation to recovering my old system in an approximately
usable state. (cpio had very kindly changed many of the directory permissions to drwx------,
making much the system unusable.)

Ubuntu and other Linux distributions go to some length to provide "dual boot" with
Micros**t, so why not a dual boot with the pre-existing Unix (Ubuntu) system? Then,
after a failure like this one, the user can revert to the old but working system. If that's
too difficult, then at least please move the old / /usr and /var to OLD, so that they can be
restored using some other tool like Knoppix.

If I hadn't had a Knoppix CD for fixing things, and a laptop to look stuff up on the Web,
It would have had no way of recovering my computer.

Xubuntu 8.04 (24 April 2008) CD image downloaded

My only differences from "standard" set-up:
(1) Old hardware
(2) DIY disk partitioning.

On completion of the intallation, it failed to boot, with messages like this:
VFS: cannot open root deveice "UUID=<hex stuff>" or unknown block (0,0)
Please append a correct "root=" boot option; here are the available partitions:
kernel panic - not syncing: VFS: unable to mount root fs on unknown-block (0,0)

Eventually I worked out that
- the "initrd" command is missing from /boot/grub/menu.lst
- the file /boot/initrd.img-2.6.24-16-generic is missing
  (actially, it's there, but called /boot/initrd.img-2.6.24-16-generic.bak)
After correcting this, the new system booted correctly.

Another thing that confused matters was that my internal IDE disk
that had been called /dev/hda was now called /dev/sda which I thought
meant a SCSI disk.

On further investigation, I found that the "installer" failed to create initrd.img
The relevant part of /var/log/installer/syslog is attached.

For the enlightenment of novices like me:
"grub" is part of the bootstrapping process, and loads the operating system kernel;
to do this, it needs the executable binary of the kernel itself
and also a minimal filesystem, which is what initrd.img provides
(this is a gzip-compressed cpio-image of  /bin, /conf, /etc, /init, /lib, /modules, /sbin, /scripts, /usr and /var)
There are tools such as update_initramfs and others to create this image,
but I could not work out how to do so unless you already have the target filesystem in place,
which seems to require the target kernel too;  in particular, I don't know how to set things
up to use chroot.

Since there is an initrd.img...bak file, somebody was presumably aware of the possibility
that creation of the new initrd.img might fail during the installation process.   There
should be a check for this, and in this case the .bak file should be installed.

As psl says, this is a critical bug, and one that could easily put people off using Linux for life.
Ubuntu sells itself as "Linux for human beings", which this kind of failure clearly is not.

However, what makes me angry about this is not the bug itself, but the enthusiasm that
the installation cd has for trashing my disk and pre-existing system (which was Ubuntu 6.10).
Even though I had backed everything up, it took me about 20 hours work from giving the
go-ahead to the Xubuntu 8.04 installation to recovering my old system in an approximately
usable state.  (cpio had very kindly changed many of the directory permissions to drwx------,
making much the system unusable.)

Ubuntu and other Linux distributions go to some length to provide "dual boot" with
Micros**t,  so why not a dual boot with the pre-existing Unix (Ubuntu) system?   Then, 
after a failure like this one, the user can revert to the old but working system.    If that's
too difficult, then at least please move the old / /usr and /var to OLD, so that they can be
restored using some other tool like Knoppix.

If I hadn't had a Knoppix CD for fixing things, and a laptop to look stuff up on the Web,
It would have had no way of recovering my computer.

Revision history for this message

Paul Taylor (paul-taylor-london) wrote on 2008-05-02:

#4

lspci -vvnn Edit (5.5 KiB, text/plain)

Description of my hardware:

Mainboard Gigabyte GA-7IX ATX SOYO 5EMA+ 100Mz (Skt7)
Processor AMD K6-2-550
Memory 128Mb DIMM PC100

NB this is less RAM than the Xubuntu 8.04 documentation says that I need.

Revision history for this message

Paul Taylor (paul-taylor-london) wrote on 2008-06-18:

#5

I find it shocking that nothing has been done about the fact that the installation CD can leave someone's computer in an unusable state, from which it is only possible to recover using other tools and considerable Linux experience.

Revision history for this message

Paul Taylor (paul-taylor-london) wrote on 2008-06-22:

#6

to Steve Langasek:

How do you know that (occurrence of) this bug is "rare"? It renders the computer unusable, especially if it occurs during use of the installation CD. If this happens to someone who is "just trying out" Linux for the first time, they'll (re)install Windows Vista and never come back to Linux. If it happens to a more experienced Unix user, they'll switch to another Linux distro and never come back to Ubuntu. In neither case is the user likely to go to the trouble of submitting a big report.

I can understand your intellectual frustration in not understanding what "triggers" the failure to create initrd.img, but the priority is surely to fail safely, or find a roundabout way of succeeding, as I eventually did myself, by using initrd.img.bak.

I would suggest that the program that rewrites /boot/grub/menu.lst should refuse to do so unless it can find an initrd.img (or an initrd.img.bak). It could then also generate whatever debugging information you need, and ask the user to submit it here.

I would also suggest that the installation CD should not trash the existing system, but leave it in a state that allows the user to revert to it, ideally via the GRUB menu.

Revision history for this message

Paul Taylor (paul-taylor-london) wrote on 2008-06-26:

#7

This dialogue seems to be split between two pages - it is not possible to merge them?

> if the initrd.img has not been created, then that's certainly a bug in its own right,
> though not a bug in grub, and we should still try to get that fixed.

I would point out that in my case the initrd.img WAS created - that was how I eventually got
8.04 to work on my machine. But (1) it was called /boot/initrd.img-2.6.24-16-generic.bak
and (2) it was not mentioned in menu.lst.

The absolute priority here is that the user should at least be informed of what has happened before
the installation CD quits, and provided at least with some advice as to how to get out of the mess
and back to the old system. In particular, the installer should NOT TRASH THE OLD SYSTEM.

The program that writes menu.lst (I am assuming that it is separate from the one that creates
initrd.img) could do this check, try to make use of what is available, give an error report to the
user and save debugging information for you.

Apparently I have not been able to stress adequately the seriousness with which I view this bug.
The installer left my machine in an unusable state. I will not do another Ubuntu installation or
recommend Ubuntu to anyone else until this is fixed. In particular, I am considering putting Linux
on my Mac laptop (instead of MACOS 10.3), but there is no way that I would risk a repetition of this
experience, as I would have no equivalent of Knoppix to get me out of the mess.

Revision history for this message

Steve Langasek (vorlon) wrote on 2008-06-26: Re: [Bug 222421] Re: initrd not configured in menu.lst after upgrade from Ubuntu 7.10 to 8.04

#8

Download full text (3.9 KiB)

On Thu, Jun 26, 2008 at 09:44:07PM -0000, Paul Taylor wrote:
> This dialogue seems to be split between two pages - it is not possible
> to merge them?

Not without explicitly copying from one to the other. Sorry, there's no
perfect way to merge bug reports. (The 'perfect' way is to make sure
duplicate bug reports are never filed. :)

> I would point out that in my case the initrd.img WAS created - that was
> how I eventually got 8.04 to work on my machine. But (1) it was called
> /boot/initrd.img-2.6.24-16-generic.bak and (2) it was not mentioned in
> menu.lst.

Right; the fact that you only had a .bak image available indicates that
something went wrong in the upgrade of the kernel package or a related
package, and therefore the initrd was not available under the correct name.
Under those circumstances, update-grub could not possibly do anything other
than what it did; although not the normal use case with Ubuntu-provided
kernel packages, it is valid to have a kernel which is set up to boot
without the use of an initrd, and update-grub must not fail to set up the
boot entry in that case.

There's also no available information channel we could use for the kernel
to pass information to update-grub about whether an initrd is required.
That would require substantial redesign of the interface, and it really is
much simpler to ensure that a failure to create the initramfs file is
treated as a failure at the kernel level and stops the process there, rather
than asking grub to detect this scenario after the fact.

> The absolute priority here is that the user should at least be informed of
> what has happened before the installation CD quits, and provided at least
> with some advice as to how to get out of the mess and back to the old
> system. In particular, the installer should NOT TRASH THE OLD SYSTEM.

I agree that the failure to properly set up the kernel is an error that
needs to be communicated to the user - and by all rights, this already
*should* happen. As you're the only user who has reported this error as
part of an install, we're largely dependent on your help to figure out how
to reproduce this bug and figure out why this error is not being
communicated correctly.

As for not trashing the old system, installing an operating system carries
inherent risk of rendering a system unbootable; there is no way to prove
with 100% certainty that this won't happen to some set of users. We try to
make the installer as robust and bug-free as possible, but at the same time
the Wubi installer has been introduced in Ubuntu 8.04 precisely to provide
an alternative install method that doesn't have this risk.

The particular problem you've described is one that certainly should be
preventable - and prevented! - but again, we need to understand why the
initrd.img was missing in order to do that.

From the log file you provided, I notice the following:

Apr 27 19:12:12 ubuntu ubiquity: update-initramfs: Generating /boot/initrd.img-2.6.24-16-generic
Apr 27 19:14:00 ubuntu ubiquity: cpio: write error: Broken pipe
Apr 27 19:14:00 ubuntu ubiquity: Segmentation fault
Apr 27 19:14:00 ubuntu kernel: [ 7286.703810] gzip[6126]: segfault at 080a6002 eip ...

On Thu, Jun 26, 2008 at 09:44:07PM -0000, Paul Taylor wrote:
> This dialogue seems to be split between two pages - it is not possible
> to merge them?

Not without explicitly copying from one to the other.  Sorry, there's no
perfect way to merge bug reports.  (The 'perfect' way is to make sure
duplicate bug reports are never filed. :)

> I would point out that in my case the initrd.img WAS created - that was
> how I eventually got 8.04 to work on my machine.   But (1) it was called
> /boot/initrd.img-2.6.24-16-generic.bak and (2) it was not mentioned in
> menu.lst.

Right; the fact that you only had a .bak image available indicates that
something went wrong in the upgrade of the kernel package or a related
package, and therefore the initrd was not available under the correct name.
Under those circumstances, update-grub could not possibly do anything other
than what it did; although not the normal use case with Ubuntu-provided
kernel packages, it is valid to have a kernel which is set up to boot
without the use of an initrd, and update-grub must not fail to set up the
boot entry in that case.

There's also no available information channel we could use for the kernel
to pass information to update-grub about whether an initrd is required. 
That would require substantial redesign of the interface, and it really is
much simpler to ensure that a failure to create the initramfs file is
treated as a failure at the kernel level and stops the process there, rather
than asking grub to detect this scenario after the fact.

> The absolute priority here is that the user should at least be informed of
> what has happened before the installation CD quits, and provided at least
> with some advice as to how to get out of the mess and back to the old
> system.   In particular, the installer should NOT TRASH THE OLD SYSTEM.

I agree that the failure to properly set up the kernel is an error that
needs to be communicated to the user - and by all rights, this already
*should* happen.  As you're the only user who has reported this error as
part of an install, we're largely dependent on your help to figure out how
to reproduce this bug and figure out why this error is not being
communicated correctly.

As for not trashing the old system, installing an operating system carries
inherent risk of rendering a system unbootable; there is no way to prove
with 100% certainty that this won't happen to some set of users.  We try to
make the installer as robust and bug-free as possible, but at the same time
the Wubi installer has been introduced in Ubuntu 8.04 precisely to provide
an alternative install method that doesn't have this risk.

The particular problem you've described is one that certainly should be
preventable - and prevented! - but again, we need to understand why the
initrd.img was missing in order to do that.

From the log file you provided, I notice the following:

Apr 27 19:12:12 ubuntu ubiquity: update-initramfs: Generating /boot/initrd.img-2.6.24-16-generic
 Apr 27 19:14:00 ubuntu ubiquity: cpio: write error: Broken pipe
 Apr 27 19:14:00 ubuntu ubiquity: Segmentation fault
 Apr 27 19:14:00 ubuntu kernel: [ 7286.703810] gzip[6126]: segfault at 080a6002 eip 0804ed9c esp bfca18a0 error 4
 Apr 27 19:14:00 ubuntu ubiquity: update-initramfs: failed for /boot/initrd.img-2.6.24-16-generic

The segfault itself looks like a kernel problem, or an out-of-memory
condition.  We would need to investigate further to understand why this
happened.

More directly, we see that ubiquity noticed the update-initramfs failure
(because it reported it), but did not stop the install at that time.  So
this should be treated as an ubiquity bug; adding that task here.

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@ubuntu.com                                     vorlon@debian.org

Evan (ev) on 2008-07-03

Changed in ubiquity:
importance:	Undecided → Medium
status:	New → Confirmed

Revision history for this message

norbsoft (norbsoft) wrote on 2008-07-12:

#9

>As you're the only user who has reported this error...

I think Paul is not the only one. It should be thousends. But probably most of the is a newbie like me. What should they do? I needed hours to find this page. And still I don't know what to do. I wanted simply to try the new 8.04 - definetely no update. Medium checked - OK. Still got the problem.

FYI my configuration:
- MS Windows XP Prof SP3
- Asus Pundit2-M2A690G
- AMD Athlon X2 BE2400
- Pioneer BDC-S02 (SATA-II)
- Samsung HD161HJ 160 (SATA-II)

Paul is right: you must make this problem public asap.

Unlike others I will try it again - if you have any solution Steve.

Thanks
norb

Revision history for this message

arruah (arruah) wrote on 2008-08-03:

#10

I've got same problem

Revision history for this message

Robert Fleming (fleming) wrote on 2008-08-25:

#11

I can confirm this bug.. in fact, I'm two-for-two on the Ubuntu systems I have. Maybe relevant: I use software RAID-1 for the root filesystems of both systems. The initrd.img was properly created; the menu.lst was just missing the "initrd" line. BUT: the menu.lst entries for the old 2.6.22 kernels DO still have the correct initrd lines.

My hardware is not especially old.

Revision history for this message

Ante Karamatić (ivoks) wrote on 2008-10-03:

#12

I can confirm this happend on 8.04->8.10Beta. I did upgrade with update-manager. Initrd was generated, but grub didn't pick it up. It looks like grub is configured before initrd was created:

2008-10-03 02:04:09 configure grub 0.97-29ubuntu36 0.97-29ubuntu36
...
2008-10-03 02:11:07 status triggers-pending initramfs-tools 0.92bubuntu13
...
2008-10-03 02:12:44 configure linux-image-generic 2.6.27.4.4 2.6.27.4.4
...
2008-10-03 02:13:32 status installed initramfs-tools 0.92bubuntu13

An easy fix would be to run 'update-grub' on the end of upgrade :/

Revision history for this message

Michaël Witrant (mike-lepton) wrote on 2008-11-14:

#13

Duplicate of bug #222799 ?

Revision history for this message

Steve Langasek (vorlon) wrote on 2009-01-07:

#14

Ante, do you have the full dpkg log from this upgrade? From what I see in /usr/sbin/update-initramfs, only initramfs updates (-u) are deferred as a trigger; initramfs creation (-c) is still processed immediately. So the initramfs should still have been created at the right point in time, but perhaps I'm overlooking something.

Revision history for this message

Ante Karamatić (ivoks) wrote on 2009-01-07:

#15

I don't have that system anymore :(

Maybe the problem is that initramfs-tools are installed after the kernel. Maybe intrepid's kernel won't work with hardy's initramfs-tools. In that situation, installation of new kernel wouldn't generate initrd, but it would update grub menu.lst. And then, after initramfs-tools is installed and initrd is created, grub never picks it up again. Thus, it looks like everything is in place, but not in menu.lst. This, of course, is just a wild guess. Too bad I don't have that system anymore, but I'll try recreating this problem in VM.

Revision history for this message

Michaël Witrant (mike-lepton) wrote on 2009-01-07:

#16

Did you set "do_bootloader = no" in /etc/kernel-img.conf ?

That was the reason my initrd.img was not generated after upgrade (see bug #222799).

Revision history for this message

Ante Karamatić (ivoks) wrote on 2009-01-08: Re: [Bug 222421] Re: initrd not configured in menu.lst after upgrade from Ubuntu 7.10 to 8.04

#17

On Wed, 07 Jan 2009 18:56:34 -0000
Michael <email address hidden> wrote:

> Did you set "do_bootloader = no" in /etc/kernel-img.conf ?

I didn't change anything in that file.

> That was the reason my initrd.img was not generated after upgrade (see
> bug #222799).

initrd was generated. It just wasn't in grub's menu.lst, making system
unbootable.

Anyway, i did release upgrade in VM, but the bug didn't reappear. I'll
do more testing tomorrow.

Revision history for this message

Steve Langasek (vorlon) wrote on 2009-01-09: Re: [Bug 222421] Re: initrd not configured in menu.lst after upgrade from Ubuntu 7.10 to 8.04

#18

Yes, the sequence here that results in this kind of breakage must be:

- kernel is unpacked
- update-grub is called
- initramfs is generated (in kernel package postinst)
- update-grub is *not* called
- kernel package postinst ends successfully, leaving the kernel configured
but menu.lst broken

There are any number of ways this could come to pass, but I haven't found
any way this would happen as a result of bugs in the kernel package
maintainer scripts shipped in any of the relevant kernel versions, nor in
initramfs-tools. It's also possible this could happen as a result of a
misconfigured /etc/kernel-img.conf (postinst_hook unset, postrm_hook left in
place), or it's possible to get in this state because the kernel package
postinst did *not* complete successfully - but in that case the user should
get a very visible error from the package manager.

I would note that it's not a bug for update-grub to be called by something
else after the new kernel is unpacked and before it's configured. There are
perfectly legitimate reasons this would be the case, either due to direct
user action, as part of the postrm of another kernel that's being
uninstalled or upgraded at the same time, or as part of the maintainer
script of another package (e.g., grub itself - though this would not be the
case for hardy->intrepid upgrades - or memtest86+). The bug is in
update-grub *not* being called at the end of the kernel postinst, and I
don't see why that's happening in any of these cases.

Revision history for this message

Paul Taylor (paul-taylor-london) wrote on 2009-01-09:

#19

Download full text (3.6 KiB)

Thanks to Steve Langasek for his thinking allowed about this important bug.

In his faulty sequence, and on the occasion that the bug hit me, the crucial failure is that
** there is kernel entry in menu.lst with no accompanying initrd **
(in particular, the default entry had this).

Looking at the code in /usr/sbin/update-grub, I see that write_kernel_entry() and the loop
in output_kernel_list() where it is called seem to have been written for the explicit possibility
that initrd does not exist.

First, is this a legitimate possibility? Even if it is, could it be eliminated by generating a dummy
initramfs, even where it is not needed, or a file with the appropriate name but some cipher as
its content to indicate to write_kernel_entry() that initramfs is INTENTIONALLY absent?

What I propose is that update-grub should ONLY generate a kernel entry in menu.lst when
the kernel and initramfs actually exist. (Maybe it could also call some program to verify
that they are intact.) Failing this, it could print an error message to STDERR and leave a
comment in menu.lst concerning the defective kernel.

Bugs are like problems with one's house: some are like torn wallpaper, which can be left until
you have both the time and the degree of irritation fix them, whilst others are like rainwater
dripping on to your bed from a broken rooftile, which have to be fixed straight away.

Now, I use my computer for email and wordprocessing, and wouldn't bother to make much
fuss about bugs in packages. But, as I said above, this one left my machine in an unusable
state, that I could only fix because I had had a bit of experience as a sysadmin. This is like
water coming through the roof, and it should have the highest classification in launchpad.

So I consider that there should be a belt-and-braces solution in this case. Ubiquity has
overall responsibility for installation, and that should double-check, after the other programs
have done their job, whether the system is safe and will be able to boot. If not, it should
scream loudly to the user, try to fix the problem, and give copious debugging information,
including the address of launchpad.

I am not actually sure that Steve's sequence is correct, because in my case there was a .bak
version of the initramfs (and not the properly named one). I think that my logfile above
shows that mkinitramfs failed. I don't know why the .bak file existed, but it worked when
I renamed it and rewrote menu.lst manually. Maybe mkinitramfs created the image correctly,
with the .bak name, but failed just before it changed the name.

I think that my logfile above shows that initramfs failed because of a lack of memory. When
the next kernel came along, I was unable to install it for exactly this reason, and apt-get
got into a confused state in which it would neither install nor remove packages. This forced
me to buy some more RAM. (I also had far too small a partition for /, and later fixed this too.)
However, this problem has still not competely gone away - the machine hangs whenever
I run any of the Ubuntu GUI pacakage management tools, and so I jsut use apt-get from the
terminal instead.

But, once again, even if m...

Thanks to Steve Langasek for his thinking allowed about this important bug.

In his faulty sequence, and on the occasion that the bug hit me, the crucial failure is that
**  there is kernel entry in menu.lst with no accompanying initrd **
(in particular, the default entry had this).

Looking at the code in /usr/sbin/update-grub,  I see that write_kernel_entry() and the loop
in output_kernel_list() where it is called seem to have been written for the explicit possibility
that initrd does not exist.

First, is this a legitimate possibility?  Even if it is, could it be eliminated by generating a dummy
initramfs, even where it is not needed, or a file with the appropriate name but some cipher as
its content to indicate to write_kernel_entry() that initramfs is INTENTIONALLY absent?

What I propose is that update-grub should ONLY generate a kernel entry in menu.lst when
the kernel and initramfs actually exist.  (Maybe it could also call some program to verify
that they are intact.)  Failing this, it could print an error message to STDERR and leave a 
comment in menu.lst concerning the defective kernel.

Bugs are like problems with one's house:  some are like torn wallpaper, which can be left until
you have both the time and the degree of irritation fix them, whilst others are like rainwater
dripping on to your bed from a broken rooftile, which have to be fixed straight away.

Now, I use my computer for email and wordprocessing, and wouldn't bother to make much
fuss about bugs in packages.  But, as I said above, this one left my machine in an unusable
state, that I could only fix because I had had a bit of experience as a sysadmin.   This is like
water coming through the roof, and it should have the highest classification in launchpad.

So I consider that there should be a belt-and-braces solution in this case.  Ubiquity has
overall responsibility for installation, and that should double-check, after the other programs
have done their job, whether the system is safe and will be able to boot.  If not, it should
scream loudly to the user,  try to fix the problem, and give copious debugging information,
including the address of launchpad.

I am not actually sure that Steve's sequence is correct, because in my case there was a .bak
version of the initramfs (and not the properly named one).   I think that my logfile above
shows that mkinitramfs failed.  I don't know why the .bak file existed, but it worked when 
I renamed it and rewrote menu.lst manually.  Maybe mkinitramfs created the image correctly,
with the .bak name, but failed just before it changed the name.

I think that my logfile above shows that initramfs failed because of a lack of memory. When
the next kernel came along, I was unable to install it for exactly this reason, and apt-get
got into a confused state in which it would neither install nor remove packages.  This forced
me to buy some more RAM.   (I also had far too small a partition for /, and later fixed this too.)
However, this problem has still not competely gone away - the machine hangs whenever 
I run any of the Ubuntu GUI pacakage management tools, and so I jsut use apt-get from the
terminal instead.

But, once again, even if my hardware falls below the recommended spec, this is no excuse
for leaving my machine in an unusable state without any explanation of what has really gone
wrong.  Presumably it is part of the Ubuntu philosophy that people without the money to
buy the latest and sexiest computers still have the right to use a decent operating system.

Finally, I noticed from a search of lauchpad that there have actually been lots of reports of
this bug, dating back to 2006.

Revision history for this message

Paul Taylor (paul-taylor-london) wrote on 2009-01-09:

#20

Sorry, "thinking aloud".

And I do hate text boxes on web forms, and their assumption that "carriage return" means "new paragraph". A "preview" button would help, to give one the opportunity to correct the line breaking.

Revision history for this message

Steve Langasek (vorlon) wrote on 2009-01-11:

#21

On Fri, Jan 09, 2009 at 08:04:57PM -0000, Paul Taylor wrote:
> First, is this a legitimate possibility?

Yes, it is.

> Even if it is, could it be eliminated by generating a dummy initramfs, even
> where it is not needed

No, because the entire problem is that update-grub is being called (for a
reason that I presume is legitimate) before the kernel postinst has a chance
to do its job, and then kernel package installation is somehow failing to
call update-grub afterwards. So the only reliable point at which we would
generate the dummy initramfs is the same point at which we generate the real
one.

> Finally, I noticed from a search of lauchpad that there have actually been
> lots of reports of this bug, dating back to 2006.

It is probable that most of these are false-positives due to broken
configurations (/etc/kernel-img.conf) or other bugs that have already been
fixed. For that matter, the bug you describe is not the same as the one
originally reported by psl and confirmed by Ante; conflating the two issues
in a single bug report makes it harder to get to the root of either bug.
The ubiquity bug really should have been opened as a separate bug report.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Revision history for this message

Michaël Witrant (mike-lepton) wrote on 2009-01-11:

#22

Since this problem
* is known to happen sometimes (whether the circumstances are clear or not),
* happens on valid systems (for example using do_bootloader=no is not an error),
* may reappear in the future (any package can call update-grub after kernel unpacking and the kernel package may not call it during postinst for whatever reason),
* will break the whole system (by making it unbootable),
* is very hard to figure out and fix once it happened (the problem is not obvious, you only get a generic kernel panic and the recovery boot option doesn't work),
* is very easy to detect and prevent before reboot (simply check menu.lst and call update-grub),

I'd suggest a sanity check after upgrade, or after any kernel installation. Since the kernel package does know it needs an initrd line, it could check menu.lst : if the installed kernel is present without an initrd line, raise an error, ask the user, call update-grub, or do anything that will warn the user.

Another easy fix would be to generate the initramfs just after unpacking. Or to install the kernel in /boot only once the initramfs has been generated.

Revision history for this message

Paul Taylor (paul-taylor-london) wrote on 2009-01-11:

#23

Download full text (4.2 KiB)

With all due respect, Steve, I fear that you are never going to solve
this/these bug(s) unless you are more critical of your own reasoning.
Whilst I now use computers primarily for email and word processing,
you can see from my web page that I was a computer science lecturer and
have taught reasoned programming, based on methods due to Floyd, Hoare
and Dijkstra.

Correctness of a program depends on a clear statement of what you intend
it to do, and on correctness of the parts. This is expressed by means
of pre-, post- and mid-conditions, and by invariants for loops and data
structures. It seems to me that these are missing here, and in particular
the implementation
CONFLATES FAILURE TO CREATE THE INITRAMFS WITH THE INTENDED ABSENCE
that you claim is legitimate.

Let me work backwards from the headline problem, namely failure to boot
after a new Ubiquity installation. The precondition for booting is to
have a valid (kernel,initramfs) pair. Ubiquity, and in particular
update-grub, failed to satisfy this as their post-condition. The
pre-condition for update-grub is that each kernel be accompanied by any
initramfs that it needs; update-grub works according to the presence or
absence of files whose names match certain patterns. It assumes that
the absence of an initramfs indicates that none is needed. This assumption
is wrong.

> the entire problem is that update-grub is being called (for a reason
> that I presume is legitimate) before the kernel postinst has a chance
> to do its job, and then kernel package installation is somehow failing
> to call update-grub afterwards. So the only reliable point at which
> we would generate the dummy initramfs is the same point at which
> we generate the real one.

So we come to the data invariant. menu.lst is a representation of a data
structure consisting of a list (with a preference order defined by the
version numbers) of (kernel,initramfs) pairs. This is obtained from
the representation in the filesystem, which is in turn modified by the
programs that install kernels and generate initramfses.

THESE PROGRAMS ARE NOT MAINTAINING THE DATA INVARIANT.

You need to treat them as a single operation, and not two. Or, given
that the data structure is encoded using filenames, they should be given
temporary names and then renamed by a program that first verifies that
both are valid, before putting them both in place at the same time.

You claim that a kernel can legimately do without an initramfs. I know
next to nothing about kernels, and acknowledge that you may be right,
but I suggest that you examine that claim very carefully. Presumably
it pertains to a multi-boot situation, given that the choice of the
Linux kernel in Ubuntu is under the control of the Ubuntu design. This
means that non-initramfs kernels come from somewhere else, not the program
that instals the normal Ubuntu Linux kernel. So you can adjust the data
invariant and the way that you maintain it accordingly.

Given the severity of the consequences, I, like Micheal, ALSO think that
there should be a subsequent check and fix in Ubiquity before it finishes
the installation.

Finally, there is the Ubuntu philosophy. As I unders...

With all due respect, Steve, I fear that you are never going to solve
this/these bug(s) unless you are more critical of your own reasoning.
Whilst I now use computers primarily for email and word processing,
you can see from my web page that I was a computer science lecturer and
have taught reasoned programming, based on methods due to Floyd, Hoare
and Dijkstra.

Correctness of a program depends on a clear statement of what you intend
it to do, and on correctness of the parts.   This is expressed by means
of pre-, post- and mid-conditions, and by invariants for loops and data
structures.  It seems to me that these are missing here, and in particular
the implementation
   CONFLATES FAILURE TO CREATE THE INITRAMFS WITH THE INTENDED ABSENCE
that you claim is legitimate.

Let me work backwards from the headline problem, namely failure to boot
after a new Ubiquity installation.   The precondition for booting is to
have a valid (kernel,initramfs) pair.  Ubiquity, and in particular
update-grub, failed to satisfy this as their post-condition.   The
pre-condition for update-grub is that each kernel be accompanied by any
initramfs that it needs; update-grub works according to the presence or
absence of files whose names match certain patterns.  It assumes that
the absence of an initramfs indicates that none is needed.  This assumption
is wrong.

> the entire problem is that update-grub is being called (for a reason
> that I presume is legitimate) before the kernel postinst has a chance
> to do its job, and then kernel package installation is somehow failing
> to call update-grub afterwards. So the only reliable point at which
> we would generate the dummy initramfs is the same point at which
> we generate the real one.

So we come to the data invariant.  menu.lst is a representation of a data
structure consisting of a list (with a preference order defined by the
version numbers) of (kernel,initramfs) pairs.   This is obtained from 
the representation in the filesystem, which is in turn modified by the
programs that install kernels and generate initramfses.

THESE PROGRAMS ARE NOT MAINTAINING THE DATA INVARIANT.

You need to treat them as a single operation, and not two.  Or, given
that the data structure is encoded using filenames, they should be given
temporary names and then renamed by a program that first verifies that
both are valid, before putting them both in place at the same time.

You claim that a kernel can legimately do without an initramfs.  I know
next to nothing about kernels, and acknowledge that you may be right,
but I suggest that you examine that claim very carefully.   Presumably
it pertains to a multi-boot situation, given that the choice of the
Linux kernel in Ubuntu is under the control of the Ubuntu design.  This
means that non-initramfs kernels come from somewhere else, not the program
that instals the normal Ubuntu Linux kernel.  So you can adjust the data
invariant and the way that you maintain it accordingly.

Given the severity of the consequences, I, like Micheal, ALSO think that
there should be a subsequent check and fix in Ubiquity before it finishes
the installation.

Finally, there is the Ubuntu philosophy.  As I understand it, the word
means "I am here because of everyone else".   The whole open source world
relies on the cooperation of users, both for good will and for information.

With regard to reporting bugs in Launchpad, you have now said both
> The 'perfect' way is to make sure duplicate bug reports are never filed.
and
> The ubiquity bug really should have been opened as a separate bug report.

No database implementation can achieve this, except for the empty database.
Mathematically, it depends on "decidable equality" of records, but saying
whether one bug manifestation is the same as another is subjective.  
Usually, I have failed to report bugs at all because I cannot see my way
through the morass of entries in Lauchpad.  For those of you on the receiving
end of the reports, you must either indulge ignorant users like me, or
just not get my reports.

Also, for me to submit a report that is diagnostically useful to you,
your programs must first provide me with the diagnostic information,
and then tell me politely where to submit it.   This comes back to the
formulation and verification of correctness conditions in the program.

Revision history for this message

Paul Taylor (paul-taylor-london) wrote on 2009-01-29:

#24

Download full text (6.8 KiB)

Ubuntu is a Good Thing, and I would like to recommend it to my friends, giving them CDs. However, I cannot do this when I know that the installation CD or a routine kernel upgrade could leave their computer in an unbootable state. Especially when it is now clear where the error lies. Please can we fix it?

Earlier, Steve spelt out the sequence
   1. kernel is unpacked
   2. update-grub is called
   3. initramfs is generated (in kernel package postinst)
   4. update-grub is *not* called
   5. kernel package postinst ends successfully, leaving the kernel configured but menu.lst broken
suggesting that the fault lies in step 4 or 5.

This is incorrect. The error is at step 2. As I said before, this fails to maintain the data invariant.

If you don't like the theoretical description, let me spell it out with reference to the kernel upgrade that Ubtunu/Hardy has just done on my machine. I ran "sudo apt-get update" and then "upgrade" from a terminal, and a number of other packages arrived at the same time:
  acpid libcamel1.2-11 libebook1.2-9 libecal1.2-7 libedataserver1.2-9
  libldap-2.4-2 linux-headers-2.6.24-23 linux-headers-2.6.24-23-generic
  linux-image-2.6.24-23-generic linux-libc-dev python-apt python-gobject
  vim-common vim-runtime vim-tiny
...
Fetched 35.1MB in 1min57s (300kB/s)
(Reading database ... 153425 files and directories currently installed.)
Preparing to replace linux-image-2.6.24-23-generic 2.6.24-23.46 (using .../linux-image-2.6.24-23-generic_2.6.24-23.48_i386.deb) ...
Done.
Unpacking replacement linux-image-2.6.24-23-generic ...
Running postrm hook script /sbin/update-grub.
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-2.6.24-23-generic
Found kernel: /boot/vmlinuz-2.6.24-22-generic
Found kernel: /boot/vmlinuz-2.6.24-21-generic
Found kernel: /boot/vmlinuz-2.6.24-19-generic
Found kernel: /boot/memtest86+.bin
Replacing config file /var/run/grub/menu.lst with new version
Updating /boot/grub/menu.lst ... done

Now we have in /boot --- please notice the dates:
422838 2009-01-26 04:30 abi-2.6.24-23-generic
80051 2009-01-26 04:30 config-2.6.24-23-generic
7504607 2009-01-20 17:19 initrd.img-2.6.24-23-generic
7504424 2009-01-15 15:43 initrd.img-2.6.24-23-generic.bak
905809 2009-01-26 04:30 System.map-2.6.24-23-generic
1922904 2009-01-26 04:30 vmlinuz-2.6.24-23-generic

So if I had had a power failure at this point, a reboot would try to run the kernel (vmlinuz) of 26th Jan with the (presumably incompatible) image (initrd) of 20th Jan. Alternatively, if a kernel with a new version number had been installed, there would have been no initrd at all.

Either way, menu.lst is now CORRUPT, because it contains an invalid kernel configuration.

There are a lot more installations, and therefore opportunities for crashes and power failures, before

Unpacking replacement linux-headers-2.6.24-23 ...
Preparing to replace linux-headers-2.6.24-23-generic 2.6.24-23...

Ubuntu is a Good Thing, and I would like to recommend it to my friends, giving them CDs.  However, I cannot do this when I know that the installation CD or a routine kernel upgrade could leave their computer in an unbootable state.  Especially when it is now clear where the error lies.   Please can we fix it?

Earlier, Steve spelt out the sequence
   1. kernel is unpacked
   2. update-grub is called
   3. initramfs is generated (in kernel package postinst)
   4. update-grub is *not* called
   5. kernel package postinst ends successfully, leaving the kernel configured but menu.lst broken
suggesting that the fault lies in step 4 or 5.

This is incorrect.  The error is at step 2.  As I said before, this fails to maintain the data invariant.

If you don't like the theoretical description, let me spell it out with reference to the kernel upgrade that Ubtunu/Hardy has just done on my machine.   I ran "sudo apt-get update" and then "upgrade" from a terminal, and a number of other packages arrived at the same time:
  acpid libcamel1.2-11 libebook1.2-9 libecal1.2-7 libedataserver1.2-9
  libldap-2.4-2 linux-headers-2.6.24-23 linux-headers-2.6.24-23-generic
  linux-image-2.6.24-23-generic linux-libc-dev python-apt python-gobject
  vim-common vim-runtime vim-tiny
...
Fetched 35.1MB in 1min57s (300kB/s)                                            
(Reading database ... 153425 files and directories currently installed.)
Preparing to replace linux-image-2.6.24-23-generic 2.6.24-23.46 (using .../linux-image-2.6.24-23-generic_2.6.24-23.48_i386.deb) ...
Done.
Unpacking replacement linux-image-2.6.24-23-generic ...
Running postrm hook script /sbin/update-grub.
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-2.6.24-23-generic
Found kernel: /boot/vmlinuz-2.6.24-22-generic
Found kernel: /boot/vmlinuz-2.6.24-21-generic
Found kernel: /boot/vmlinuz-2.6.24-19-generic
Found kernel: /boot/memtest86+.bin
Replacing config file /var/run/grub/menu.lst with new version
Updating /boot/grub/menu.lst ... done

Now we have in /boot --- please notice the dates:
  422838 2009-01-26 04:30 abi-2.6.24-23-generic
    80051 2009-01-26 04:30 config-2.6.24-23-generic
7504607 2009-01-20 17:19 initrd.img-2.6.24-23-generic
7504424 2009-01-15 15:43 initrd.img-2.6.24-23-generic.bak
 905809 2009-01-26 04:30 System.map-2.6.24-23-generic
1922904 2009-01-26 04:30 vmlinuz-2.6.24-23-generic

So if I had had a power failure at this point, a reboot would try to run the kernel (vmlinuz) of 26th Jan with the (presumably incompatible) image (initrd) of 20th Jan.   Alternatively, if a kernel with a new version number had been installed, there would have been no initrd at all.

Either way, menu.lst is now CORRUPT, because it contains an invalid kernel configuration.

There are a lot more installations, and therefore opportunities for crashes and power failures, before

Unpacking replacement linux-headers-2.6.24-23 ...
Preparing to replace linux-headers-2.6.24-23-generic 2.6.24-23.46 (using .../linux-headers-2.6.24-23-generic_2.6.24-23.48_i386.deb) ...
Unpacking replacement linux-headers-2.6.24-23-generic ...

Yet more installations take place before

Setting up linux-image-2.6.24-23-generic (2.6.24-23.48) ...
Running depmod.
update-initramfs: Generating /boot/initrd.img-2.6.24-23-generic
Not updating initrd symbolic links since we are being updated/reinstalled 
(2.6.24-23.46 was configured last, according to dpkg)
Not updating image symbolic links since we are being updated/reinstalled 
(2.6.24-23.46 was configured last, according to dpkg)
Running postinst hook script /sbin/update-grub.
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-2.6.24-23-generic
Found kernel: /boot/vmlinuz-2.6.24-22-generic
Found kernel: /boot/vmlinuz-2.6.24-21-generic
Found kernel: /boot/vmlinuz-2.6.24-19-generic
Found kernel: /boot/memtest86+.bin
Updating /boot/grub/menu.lst ... done

Finally /boot/grub/menu.lst is back in a safe state.

So this is what I propose:

update-grub should merely CONCATENATE (some list of grub options and) kernel configuration items that are in individual files, and not try to implement any of the "logic" of matching kernels with initrds.

I don't understand what the circumstances might be in which a kernel might not want an initrd, but let us imagine a very general situation in which there are Unix kernels from different Linux distros, MacOSX, Salaris, BSD, etc etc.  Each of them is built and installed in a different way, and needs different options in grub.  So, as the last stage of the installation of a particular kernel, some script (belonging to that kernel installation package) checks that all of its bits are in place, and generates a file that update-grub will subsequently add to menu.lst.

There is the opportunity here to use this file for other purposes, communicating between the installation steps, and telling the user where to find log files for debugging purposes. All of this information would be "commented out" with # so that grub doesn't treat it as an active kernel, EXCEPT where it actually provides a valid kernel configuration.

For the purposes of disaster recovery, grub itself should have facilities for showing menu.lst and any other files, in particular debugging log files, and should also give urls of Linux help pages and of a launchpad page where advice can be found and bugs reported.

By this mechanism, you will be able to get debugging information back about why mkinitramfs fails, for example.

update-grub may find NO valid kernel configurations.   We avoid this by adopting the usual methods of preventing the user from deleting the last remaining valid kernel unless s/he really intends to do so.

This leaves the case of a fresh installation (Ubiquity).  Of necessity, this must be running under SOME kernel, so Ubiquity should begin by installing this, and creating the initial menu.lst containing just it.  Alternatively, it should copy a minimal kernel straight from disk.

I couldn't follow the source code for grub and ubiquity, but I did notice that ubiquity duplicates code that is like that in update-grub.   Duplication of code is a bad idea for correctness reasons - it should CALL update-grub instead.

Besides, this, I have said before that Ubiquity should not trash the disk, but try to leave any existing Unix system in a recoverable state.   It should also have, in addition to  LIVE CD running and installation, a recovery mode for fixing a previous broken installation attempt.

Revision history for this message

psl (slansky) wrote on 2009-01-29:

#25

I created new bug #322949. I think it is related to this one from some point of view. initrd.img is build during kernel removal.

Revision history for this message

Steve Langasek (vorlon) wrote on 2012-03-23:

#26

This bug was rediscovered recently as bug #959700 and should now be fixed in 12.04.

Changed in grub (Ubuntu):
status:	Incomplete → Fix Released

Revision history for this message

Steve Langasek (vorlon) wrote on 2012-03-23:

#27

Sorry, I mean bug #957262, not 959700.

Changed in ubiquity (Ubuntu):
status:	Confirmed → Invalid

Affects		Status	Importance	Assigned to	Milestone
	grub (Ubuntu)	Fix Released	Undecided	Unassigned
	ubiquity (Ubuntu)	Invalid	Medium	Unassigned

Ubuntu
grub package

initrd not configured in menu.lst after upgrade from Ubuntu 7.10 to 8.04

Bug Description

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntugrub package

initrd not configured in menu.lst after upgrade from Ubuntu 7.10 to 8.04

Bug Description

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
grub package