Doesn't check for failure due to full filesystem

Bug #31126 reported by Erich Pawlik
66
This bug affects 2 people
Affects Status Importance Assigned to Milestone
initramfs-tools (Ubuntu)
High
Unassigned

Bug Description

After upgrading to version 2.6.15.14, the system halted during the boot process after displaying the following messages:

[4294668.836000] RAMDISK: ran out of compressed data
[4294668.836000] invalid compressed format (err=1)
[4294668.867000] Kernel panic
[4294668.867000]

It was possible to boot the previous kernel version.

Diagnosis:

The problem appears to be a problem of the kernel update procedure. I have a small boot partition which did run out of disk space. The problem could be fixed by removing some older kernel versions left over from previous kernel updates, editing /boot/grub/menu.lst and reinstalling version 2.16.15.14.

I feel its a good feature to have the latest working kernel available in case of problems with kernel updates. However, I feel it is approbriate to remove older kernel versions during the update process. Collecting old kernels forever also means that the position of entries in /boot/grub/menu.lst changes after each update this is annoying if you are by default booting another operating system.

Revision history for this message
Ben Collins (ben-collins) wrote :

Maybe initramfs can offer to do this (or atleast suggest it to the user).

Revision history for this message
Matt Zimmerman (mdz) wrote :

initramfs-tools is already cautious about this; it doesn't move the initramfs into place until it has been successfully created.

Are you sure that the package installation didn't fail? Do you have a log?

Revision history for this message
Adam Conrad (adconrad) wrote :

Actually, Matt, it's a longstanding bug in initramfs-tools (and one I refuse to let myself forget to fix before dapper) that initramfs-tools doesn't have much in the way of useful error conditions, and certainly doesn't fail it if runs out of space. This bug if very real (and very annoying, and very getting fixed next week, with several other "Eek, initramfs-tools is broken in weird ways, but we never noticed until now" bugs.

Changed in initramfs-tools:
status: Unconfirmed → Confirmed
Revision history for this message
Matt Zimmerman (mdz) wrote : Re: [Bug 31126] Re: Kernel panic after installation of linux-386 [Dapper]

On Tue, Apr 04, 2006 at 02:23:49AM -0000, Adam Conrad wrote:
> Actually, Matt, it's a longstanding bug in initramfs-tools (and one I
> refuse to let myself forget to fix before dapper) that initramfs-tools
> doesn't have much in the way of useful error conditions, and certainly
> doesn't fail it if runs out of space. This bug if very real (and very
> annoying, and very getting fixed next week, with several other "Eek,
> initramfs-tools is broken in weird ways, but we never noticed until now"
> bugs.

I see; so the kernel is dutifully paying attention to error returns from
initramfs-tools...which it doesn't provide. ;-)

--
 - mdz

Revision history for this message
Erich Pawlik (erichpawlik) wrote : Re: Kernel panic after installation of linux-386 [Dapper]

To Matt's question: Do you still need a log and if yes, where can I find it?

I am not sure whether the installation failed. Synaptic didn't bring up an error. However what happened (and this happened several times on several machines with Hoary and Dapper - I never did get deep into Breezy because I perceived it too buggy):
- /boot partition (quite small, I have taken over my partitioning from another distribution that recommends small /boot partitions) is full
- The error message occurs when booting into the latest kernel
- Booting into older kernel works
- After deinstallation of an older kernel version and reinstallation of the latest kernel version, everything is fine.

I am not sure whether I like the idea that the installation of the new kernel aborts in case of missing disk space. I rather would prefer that an older kernel version gets deinstalled (I don't need four or five kernel versions, two is enough, the new one and the latest working)

Regards

Erich

Revision history for this message
Matt Zimmerman (mdz) wrote : Re: [Bug 31126] Re: Kernel panic after installation of linux-386 [Dapper]

On Tue, Apr 04, 2006 at 06:36:23AM -0000, Erich Pawlik wrote:
> To Matt's question: Do you still need a log and if yes, where can I find it?

No, thanks. It seems likely that indeed the problem was caused by running
out of disk space on /boot, and Adam has acknowledged that that case isn't
handled correctly at the moment.

> I am not sure whether I like the idea that the installation of the new
> kernel aborts in case of missing disk space. I rather would prefer that an
> older kernel version gets deinstalled (I don't need four or five kernel
> versions, two is enough, the new one and the latest working)

It is important to keep the solution simple in these cases in order to
maintain robustness; the installation should simply abort.

Note that a separate /boot is only needed in very special configurations
today (e.g., RAID-5 or LVM root), so in general this should be a rare
occurrence.

--
 - mdz

Revision history for this message
Erich Pawlik (erichpawlik) wrote : Re: Kernel panic after installation of linux-386 [Dapper]

I don't think that aborting the installation is a robust behavior - it is simple to program, but robustness to my opinion is not a question of programming simplicity, but of ease of use. If a kernel update fails, a lot of people with not few technical skills (the Ubuntu target group I guess) will be out in the dark and don't know where to proceed from there. In addition, aborting a program is only simple it you accept to leave your garbage lying around all over the place. Again not very appropriate for the target group.

On the other hand, I agree that this error is very unlikely to happen to somebody else. The usual suspects such as Google and the Ubuntu forums didn't help.So, aborting will not hurt a lot of people. I am not very pleased that a lot of kernels are lying around, but can deal with this manually.

Revision history for this message
Matt Zimmerman (mdz) wrote : Re: [Bug 31126] Re: Kernel panic after installation of linux-386 [Dapper]

On Tue, Apr 04, 2006 at 08:09:49PM -0000, Erich Pawlik wrote:
> I don't think that aborting the installation is a robust behavior - it is
> simple to program, but robustness to my opinion is not a question of
> programming simplicity, but of ease of use. If a kernel update fails, a
> lot of people with not few technical skills (the Ubuntu target group I
> guess) will be out in the dark and don't know where to proceed from there.
> In addition, aborting a program is only simple it you accept to leave your
> garbage lying around all over the place. Again not very appropriate for
> the target group.

In practical terms, there are cases where an error unwind is simply
inherently riskier than aborting. Re-entrantly invoking the package manager
to try to dynamically free up space during the installation of another
package...that is one of these cases.

An error message explaining that available space is exhausted, with a button
to retry, would go a lot further toward a robust AND usable solution.

> I am not very pleased that a lot of kernels are lying around, but can deal
> with this manually.

We are aware of this issue, but have not solved it yet.

--
 - mdz

Revision history for this message
Ben Collins (ben-collins) wrote :

The only thing that I wanted to add to this bug report is that if the installation of the vmlinux fails because of space problems, that, IMO, is a bug in dpkg or other portions of package management, not a bug in the package.

The initramfs issue of there not being enough space for the initrd is a seperate issue (since it is created at install time, not as part of the package itself).

Revision history for this message
eloj (srm-dfr) wrote : /boot full not checked -- suggestions

>"and one I refuse to let myself forget to fix before dapper"

Someone fell asleep at the wheeeeel :-)

I've been hit by this several times, due to having at one time created a too small boot partition. Just now, it happened again. The only reason I noticed is because the rattling of the disk suddenly stopped and CPU usage went up at a time that was unexpected.

Here's an excerpt from my /boot after the failed upgrade:
-rw-r--r-- 1 root root 6962511 2006-06-01 14:39 initrd.img-2.6.15-23-686
-rw-r--r-- 1 root root 6967716 2006-06-28 03:59 initrd.img-2.6.15-25-686
-rw-r--r-- 1 root root 1019904 2006-07-12 00:27 initrd.img-2.6.15-26-686

Notice the b0rken initrd-26 (file size)

What's so sinister about this is that the Update Manager will conclude with a "Upgrade Completed!" and a helpful hint that maybe rebooting would be a good idea :-\

While I believe the bug really covers a wider range of packets (the Upgrade Manager for sure!), here are some feature suggestions:

1. Have the Ubuntu Installer warn about creating too small /boot partitions. Currently, about 10MiB (9.2MiB for me) is needed _per kernel_. A message box should inform any user creating a /boot smaller than say 50MiB about how the system can keep kernels around and how this might become an issue.

2. The default grub configuration seems to be "keep all kernels"? I don't know what the actual limit if any is, but the "howmany" entry in /boot/grub/menu.lst ought to be set to something sane instead of "all", or whatever the default is. Personally I think '4' would be more than enough, in which case /boot would have to be at least 50MiB.

3. At no time can the Upgrade Manager be allowed to report success, if a kernel package fails. Never. Don't know whose fault this is (initrd, apt/dpkg, upgrade manager, whatever), but it's Got. To. Be. Fixed.

Revision history for this message
Sascha Silbe (sascha-ubuntu-launchpad) wrote :

Just happened for me again, too:

[...]
Setting up linux-image-2.6.15-27-386 (2.6.15-27.48) ...

gzip: stdout: No space left on device
Searching for GRUB installation directory ... found: /boot/grub
[...]
root@caravan:~# gzip -t /boot/initrd.img-2.6.15-27-386

gzip: /boot/initrd.img-2.6.15-27-386: unexpected end of file

So now the default grub entry points to an unbootable kernel. :(
If it's hard to fix initramfs-tools, please at least use "gzip -t" to check if the initrd has been successfully created.

Revision history for this message
ucsblaw (armen8) wrote :

hi everyone...i am very new to the linux world and i need some help.

I have a sony vaio s580. 2 gigs of ram and 100gig sata HDD.
I installed feisty using wubi... i managed to get everything up and running but then i got cocky.
I tried to fix the hibernation/ suspend problem using a guide i found on this forum

In terminal put:

Quote:
sudo apt-get install uswsusp

it gave me some blue box that said that i cant install uswsusp (i think that's what it said) and then i pressed ok. Since Im an idiot...I then decided i should remove what i just tried to install so I typed

Quote:
sudo aptitude remove uswsusp

i then restarted my computer and got this

Quote:
wubi/boot/initrd.img-2.6.20-16 low latency
[Linux-initrd @ 0x1f9af000, 0x640000 bytes]
boot
[16.297381] RAMDISK: ran out of compressed data
[16.297448] invalid compressed format (err=1)
[16.298258] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown block (8,3)
[16.298323]

Now, i need to remove my battery just to shut off the computer...i cant power off. I can get into the grub menu but i dont have a 'recovery mode' option. I dont know what i could do. This seems like a similar problem to the original post so i posted my problem here...can anyone walk me through this?

best regards

Revision history for this message
Matt Zimmerman (mdz) wrote : Re: [Bug 31126] Re: Doesn't check for failure due to full filesystem

On Mon, Jun 18, 2007 at 12:10:22AM -0000, ucsblaw wrote:
> Now, i need to remove my battery just to shut off the computer...i cant
> power off. I can get into the grub menu but i dont have a 'recovery
> mode' option. I dont know what i could do. This seems like a similar
> problem to the original post so i posted my problem here...can anyone
> walk me through this?

This isn't a support forum; please use this area only to discuss the precise
nature of the bug described. If you need help, you can file a support
request at https://answers.launchpad.net/ubuntu/+addquestion or via one of
the other resources at http://www.ubuntu.com/support

--
 - mdz

Revision history for this message
Colin Watson (cjwatson) wrote :

This was fixed in Ubuntu 8.04 (sorry we didn't follow up on this bug before now):

initramfs-tools (0.85eubuntu24) hardy; urgency=low

  * Implement the initramfs-tools part of the initramfs error handling spec
  * update-initramfs:
    - Make a hard link to the original initramfs image, rather than moving
      it out of the way.
    - Create a new initramfs image to ${initramfs}.new, to ensure we still
      have a functional initramfs in case of failure. The original initramfs
      only gets replaced when a new image is successfully created.
  * scripts/functions:
    - Added add_mountroot_fail_hook function to allow scripts in
      init-premount to register a hook to allow extra information
      to be given to the user, in the event of a non-existant root
      device.
    - The panic function now runs any registered mountroot fail hooks that
      were previously registered, and only does so when passed the -r
      argument from the calling function.
  * scripts/local: Call the panic function with -r to run any registered
    mountroot fail hooks when a root device cannot be found.

 -- Luke Yelavich <email address hidden> Tue, 05 Feb 2008 13:38:51 +1100

Removing old kernels is still outstanding, but is expected to be handled by way of system-cleaner (or whatever it gets renamed to).

Changed in initramfs-tools:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers