2.6.32-26 unbootable: does not find root file system

[David Henningsson]

When trying to boot 2.6.32-26, the following error occurs:

udevadm trigger is not permitted while udev is unconfigured.
Gave up waiting for root device.
/.. etc ../
ALERT! /dev/disk/by-uuid/0927db9a-0528-49c5-bdc5-ff3d5a9eb9c5 does not exist. Dropping to a shell!

2.6.32-25 boots fine.

SRU TEST CASE:
--------------
This is a somewhat synthetic, but IMHO sufficient test case:

- Pretend that udev is currently being upgraded:
sudo /var/lib/dpkg/info/udev.preinst upgrade `dpkg-query -W -f '${Version}' udev`

- Now update initramfs:
sudo update-initramfs -u

If you do this in a VM and can apply a previous snapshot, or are able to boot an older kernel, then you can reboot now and confirm that the initramfs is broken. With the fixed package this should produce a working initramfs. If you don't want to reboot, then you can just try

   gzip -cd /boot/initrd.img-`uname -r` | cpio -i --to-stdout sbin/udevadm | file -

A working initramfs has an "ELF LSB shared object", a broken initramfs has a POSIX shell script (the wrapper).

- Finally, finish the fake udev upgrade:
sudo /var/lib/dpkg/info/udev.postinst configure `dpkg-query -W -f '${Version}' udev`

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: linux-image-2.6.32-25-generic 2.6.32-25.45
Regression: Yes
Reproducible: Yes
ProcVersionSignature: Ubuntu 2.6.32-25.45-generic 2.6.32.21+drm33.7
Uname: Linux 2.6.32-25-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.21.
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC3: david 1680 F.... pulseaudio
 /dev/snd/controlC1: david 1680 F.... pulseaudio
 /dev/snd/controlC0: david 1680 F.... pulseaudio
CRDA: Error: [Errno 2] Filen eller katalogen finns inte
Card0.Amixer.info:
 Card hw:0 'Juli'/'ESI Juli@ at 0xcf00, irq 21'
   Mixer name : 'ICE1724 - multitrack'
   Components : ''
   Controls : 36
   Simple ctrls : 14
Card1.Amixer.info:
 Card hw:1 'SB'/'HDA ATI SB at 0xfe024000 irq 16'
   Mixer name : 'Realtek ALC888'
   Components : 'HDA:10ec0888,1458e601,00100001'
   Controls : 37
   Simple ctrls : 21
Card2.Amixer.info:
 Card hw:2 'HDMI'/'HDA ATI HDMI at 0xfdffc000 irq 29'
   Mixer name : 'ATI R6xx HDMI'
   Components : 'HDA:1002aa01,00aa0100,00100100'
   Controls : 4
   Simple ctrls : 1
Card2.Amixer.values:
 Simple mixer control 'IEC958',0
   Capabilities: pswitch pswitch-joined penum
   Playback channels: Mono
   Mono: Playback [on]
Card3.Amixer.info:
 Card hw:3 'Headset'/'Logitech Logitech USB Headset at usb-0000:00:13.0-1, full speed'
   Mixer name : 'USB Mixer'
   Components : 'USB046d:0a0c'
   Controls : 6
   Simple ctrls : 2
CheckboxSubmission: f6f6c1db2c80d23ca163011ad01858b3
CheckboxSystem: 4ed15c40009aa6f7770f606350a390a2
Date: Tue Nov 9 11:29:37 2010
HibernationDevice: RESUME=UUID=445bac10-f1e6-4ed7-90dd-cf9b237c31fe
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.

 vboxnet0 no wireless extensions.
MachineType: Gigabyte Technology Co., Ltd. GA-MA770T-UD3P
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.32-25-generic root=UUID=0927db9a-0528-49c5-bdc5-ff3d5a9eb9c5 ro crashkernel=384M-2G:64M,2G-:128M quiet splash
ProcEnviron:
 LANG=sv_SE.UTF-8
 SHELL=/bin/bash
RelatedPackageVersions: linux-firmware 1.34.1
RfKill:

SourcePackage: linux
dmi.bios.date: 06/24/2009
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: F2
dmi.board.name: GA-MA770T-UD3P
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrF2:bd06/24/2009:svnGigabyteTechnologyCo.,Ltd.:pnGA-MA770T-UD3P:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnGA-MA770T-UD3P:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: GA-MA770T-UD3P
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[Martin Pitt (pitti)]

Steve, this is a regression from the current lucid SRU. Can you please have a look at assign to the right person? Thanks!

[Tim Gardner (timg-tpi)]

David - can this be corrected by running 'update-initramfs -u' after booting the 2.6.32-25 kernel? Are you out of disk space in /boot ?

[Stefan Bader (smb)]

I can confirm that this just happened to one of my machines and that re-running update-initramfs makes the system bootable again. For some reason all of my previous initrd's also were partially remade, so I had -24 and -25 with the udevadm trigger message and -26 panic while trying to find the initrd at all.
Unfortunately dpkg.log seems not to contain any good hints (did apt-get dist-upgrade from commandline).

[Colin Watson (cjwatson)]

This is not a regression, but it is a latent bug that you were unlucky enough to hit. Something (it's not clear what) ran update-initramfs while udev was unconfigured; there are quite a few candidates and it's not clear exactly what did it. It probably doesn't matter.

I think it would be best for udev's initramfs hook to check whether /sbin/udevadm.upgrade exists. If it does, it should copy that into the initramfs as /sbin/udevadm, rather than the /sbin/udevadm wrapper script.

[Martin Pitt (pitti)]

Thanks Colin. I removed the "verification-failed"/"regression-proposed" flags.

[Launchpad Janitor (janitor)]

This bug was fixed in the package udev - 164-3

---------------
udev (164-3) natty; urgency=low

  * debian/udev.initramfs-hook: If /sbin/udevadm.upgrade exists, copy that
    into the initramfs instead of the shell script wrapper that we have during
    udev upgrades. (LP: #672964)
 -- Martin Pitt <email address hidden> Fri, 12 Nov 2010 18:35:55 +0100

[Martin Pitt (pitti)]

Lucid/Maverick SRUs uploaded as well, patches:

  http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/udev/lucid/revision/177
  http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu/maverick/udev/ubuntu/revision/2624

(same as the natty fix).

[Colin Watson (cjwatson)]

Accepted udev into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Colin Watson (cjwatson)]

Accepted udev into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[David Henningsson (diwic)]

I don't know exactly how to test this, but I did the following:
1) booted 2.6.32-25
2) sudo apt-get update && sudo apt-get dist-upgrade
3) rebooted 2.6.32-26

2.6.32-26 was still unbootable. I'm not sure it's related, but I noticed a line saying something like "updating initramfs for 2.6.35-22" (which is a maverick backport kernel I'm testing from time to time), but nothing similar for 2.6.32-26.

Marking as verification-failed for now, but if I tested things wrong, please give more specific test instructions.

[Martin Pitt (pitti)]

David, do you still have the complete screen output by any chance?

udev's postinst already calls update-initramfs -u. I wonder which kernel you were running at the time when you did the update, was it 2.6.35-22 or 2.6.32-26? I suppose it was .35, since you said that .32 didn't boot.

If you do "sudo update-initramfs -u -k all", does 2.6.32-26 boot then? I'm hesitant to add that to udev's postinst, though, since I don't really want to touch other kernels (it might actually break them). The problem here is that you already fell into the "breaks initramfs" trap. I'm pretty sure that the update here will prevent people from falling into it in the first place, but so far we do nothing to repair it after the fact.

[David Henningsson (diwic)]

> David, do you still have the complete screen output by any chance?

No, sorry.

> I wonder which kernel you were running at the time when you did the update, was it 2.6.35-22 or 2.6.32-26?

It was 2.6.32-25. (Currently I have 2.6.32-25, 2.6.32-26 and 2.6.35-22 installed. 2.6.32-26 doesn't boot.)

[David Henningsson (diwic)]

$ sudo update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-2.6.35-22-generic
update-initramfs: Generating /boot/initrd.img-2.6.32-26-generic
update-initramfs: Generating /boot/initrd.img-2.6.32-25-generic

I *think* that's what I saw, but only for 2.6.35-22, when I upgraded udev.

[David Henningsson (diwic)]

> If you do "sudo update-initramfs -u -k all", does 2.6.32-26 boot then?

Yes.

> I'm pretty sure that the update here will prevent people from falling into it in the first place, but so far we do nothing to repair it after the fact.

I guess I'm not the right person to make the decision whether we should or not, but I assume it depends on how common it is.

[Jean-Baptiste Lallement (jibel)]

verification for lucid and maverick.

I'm unable to reproduce the original issue with the provided test case. The diversion is removed and update-initramfs is run in the postinst script, copying the right udevadm to the initramfs.

But, if I run update-initramfs without removing the diversion first (running steps 1 & 2 or skipping enable_udevadm in postinst) then the current version in -updates generates a broken initramfs and the version in -proposed generates a correct initramfs.

So this fix fixes an issue, but I am not sure that it fixes the issue reported here originally.

[Charles Whittington (chw-hillbarn)]

I am suffering from this at present having upgraded from 2.6.32-25 (works) to 2.6.32-26 (gives above error). Is there anything I can do/provide to either
a) provide evidence
b) test solution?

Charles

[Martin Pitt (pitti)]

Hello Jean-Baptiste,

Jean-Baptiste Lallement [2010-11-22 11:23 -0000]:
> I'm unable to reproduce the original issue with the provided test case.
> The diversion is removed and update-initramfs is run in the postinst
> script, copying the right udevadm to the initramfs.

Ah, you mean that calling the udev postinst in step 3 updates the
initramfs _again_, and this time a correct one.

Right, so the check for the broken initramfs should be done before
calling the postinst. I updated the test case in the description
accordingly.

> But, if I run update-initramfs without removing the diversion first
> (running steps 1 & 2 or skipping enable_udevadm in postinst) then the
> current version in -updates generates a broken initramfs and the version
> in -proposed generates a correct initramfs.

That's the situation this update was meant to fix indeed. So I'll call
this a succesful verification.

> So this fix fixes an issue, but I am not sure that it fixes the issue
> reported here originally.

The original description and followups seem to indicate exactly that
problem. Unfortunately it is hard to provide a simple test case for
this, as the actual bug would only occur if multiple kernel ABIs are
involved during upgrade. I. e. the first udevadm trigger during the
update updates the initramfs for ABI 2.6.X (a broken one due to this
bug), and later on in the game a new kernel 2.6.Y gets installed, and
apparently the final udev configure then only updated the initramfs
for 2.6.Y, but not for .X any more.

Thanks,

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[Launchpad Janitor (janitor)]

This bug was fixed in the package udev - 151-12.3

---------------
udev (151-12.3) lucid-proposed; urgency=low

  * debian/udev.initramfs-hook: If /sbin/udevadm.upgrade exists, copy that
    into the initramfs instead of the shell script wrapper that we have during
    udev upgrades. (LP: #672964)
 -- Martin Pitt <email address hidden> Fri, 12 Nov 2010 18:44:54 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package udev - 162-2.2

---------------
udev (162-2.2) maverick-proposed; urgency=low

  * debian/udev.initramfs-hook: If /sbin/udevadm.upgrade exists, copy that
    into the initramfs instead of the shell script wrapper that we have during
    udev upgrades. (LP: #672964)
 -- Martin Pitt <email address hidden> Fri, 12 Nov 2010 18:37:10 +0100