Activity log for bug #332270

Date Who What changed Old value New value Message
2009-02-20 21:23:08 Michael Flaig bug added bug
2009-02-20 21:39:15 Martin Kossick None: bugtargetdisplayname Ubuntu udev (Ubuntu)
2009-02-20 21:39:15 Martin Kossick None: bugtargetname ubuntu udev (Ubuntu)
2009-02-20 21:39:15 Martin Kossick None: statusexplanation reassigning this bug to package udev
2009-02-20 21:39:15 Martin Kossick None: title Bug #332270 in Ubuntu: "[jaunty] doesn't boot anymore after udev upgrade" Bug #332270 in udev (Ubuntu): "[jaunty] doesn't boot anymore after udev upgrade"
2009-02-20 23:09:35 Martin Kossick udev: status New Confirmed
2009-02-20 23:09:35 Martin Kossick udev: statusexplanation reassigning this bug to package udev Okay, thank you. Setting status to confirmed. For future reference you can manage the status of your own bugs by clicking on the current status in the yellow line and then choosing a new status in the revealed drop down box. You can learn more about bug statuses at https://wiki.ubuntu.com/Bugs/Status .
2009-02-21 13:46:20 Albert Damen bug added attachment 'syslog.gz' (syslog.gz)
2009-02-21 20:17:54 Michael Evans udev: status Confirmed In Progress
2009-02-21 20:17:54 Michael Evans udev: assignee intuitivenipple
2009-02-21 20:17:54 Michael Evans udev: statusexplanation Okay, thank you. Setting status to confirmed. For future reference you can manage the status of your own bugs by clicking on the current status in the yellow line and then choosing a new status in the revealed drop down box. You can learn more about bug statuses at https://wiki.ubuntu.com/Bugs/Status . 20090221-12:13:33 < IntuitiveNipple> Sure... you do that, I'll do the new package and test it in the VM
2009-02-22 02:08:29 Daniel T Chen udev: importance Undecided Critical
2009-02-22 02:08:29 Daniel T Chen udev: statusexplanation 20090221-12:13:33 < IntuitiveNipple> Sure... you do that, I'll do the new package and test it in the VM
2009-02-22 03:25:27 Scott James Remnant (Canonical) udev: status In Progress Incomplete
2009-02-22 04:41:35 Anders Kaseorg udev: status Incomplete Confirmed
2009-02-22 04:41:35 Anders Kaseorg udev: assignee intuitivenipple
2009-02-22 04:41:35 Anders Kaseorg udev: statusexplanation For me: http://web.mit.edu/andersk/Public/lp-332270/dpkg_-l.txt http://web.mit.edu/andersk/Public/lp-332270/ls_sys_block.txt http://web.mit.edu/andersk/Public/lp-332270/lvm_pvs.txt http://web.mit.edu/andersk/Public/lp-332270/lvm_vgs.txt
2009-02-22 06:03:53 Marc Deslauriers bug added attachment 'dpkg_l.txt' (mdeslaur dpkg -l)
2009-02-22 06:04:16 Marc Deslauriers bug added attachment 'ls_sys_block.txt' (mdeslaur ls /sys/block)
2009-02-22 06:04:37 Marc Deslauriers bug added attachment 'lvm_pvs.txt' (mdeslaur lvm pvs)
2009-02-22 06:05:09 Marc Deslauriers bug added attachment 'lvm_vgs.txt' (mdeslaur lvm vgs)
2009-02-22 10:27:42 Luka Renko bug added attachment 'nw8440-lvm-udev-bug.tar.gz' (collected logs from LVM laptop)
2009-02-22 10:31:31 Peter Sabaini bug added attachment 'dpkg-l.txt' (dpkg-l.txt)
2009-02-22 10:31:51 Peter Sabaini bug added attachment 'ls-sys-block.txt' (ls-sys-block.txt)
2009-02-22 10:32:10 Peter Sabaini bug added attachment 'lvm-pvs.txt' (lvm-pvs.txt)
2009-02-22 10:32:25 Peter Sabaini bug added attachment 'lvm-vgs.txt' (lvm-vgs.txt)
2009-02-22 10:54:01 hunger bug added attachment '332270_info.tar.gz' (requested system information)
2009-02-22 13:54:19 TJ bug added attachment 'boot-log-udev-debug.log.gz' (Gzipped boot log with udev_log=debug)
2009-02-22 14:01:46 TJ bug added attachment 'boot-log-udev-debug.log.gz' (Gzipped boot log with udev_log=debug)
2009-02-22 14:04:19 Daniel Hahler bug added attachment 'bug332270.tar.gz' (bug332270.tar.gz)
2009-02-22 14:09:12 Albert Damen bug added attachment 'dpkgl.txt' (dpkgl.txt)
2009-02-22 14:09:12 Albert Damen bug added attachment 'lvm-pvs.txt' (lvm-pvs.txt)
2009-02-22 14:09:12 Albert Damen bug added attachment 'lvm-vgs.txt' (lvm-vgs.txt)
2009-02-22 14:09:12 Albert Damen bug added attachment 'lvm-lvs.txt' (lvm-lvs.txt)
2009-02-22 14:09:12 Albert Damen bug added attachment 'sys-block.txt' (sys-block.txt)
2009-02-22 14:09:12 Albert Damen bug added attachment '60-persistent-storage-modified-rules.txt' (60-persistent-storage-modified-rules.txt)
2009-02-22 14:09:12 Albert Damen bug added attachment 'dev-udev-watch.txt' (dev-udev-watch.txt)
2009-02-22 14:32:14 Daniel Hahler udev: status Confirmed Triaged
2009-02-22 14:32:14 Daniel Hahler udev: statusexplanation For me: http://web.mit.edu/andersk/Public/lp-332270/dpkg_-l.txt http://web.mit.edu/andersk/Public/lp-332270/ls_sys_block.txt http://web.mit.edu/andersk/Public/lp-332270/lvm_pvs.txt http://web.mit.edu/andersk/Public/lp-332270/lvm_vgs.txt Scott, I think it makes sense to provide an udev update, which includes a/the workaround - unless a proper fix is not found already. There should be enough info for debugging provided already and given the impact of this bug, it makes not much sense to upset more users using the development branch.
2009-02-22 17:03:28 TJ bug added attachment 'boot-udev_log=debug.log.gz' (udev_log=debug via serial console)
2009-02-22 17:25:56 Scott James Remnant (Canonical) title [jaunty] doesn't boot anymore after udev upgrade udev repeatedly generates "change" events for the same block device(s)
2009-02-22 19:20:43 Scott James Remnant (Canonical) udev: importance Critical High
2009-02-22 19:20:43 Scott James Remnant (Canonical) udev: statusexplanation Scott, I think it makes sense to provide an udev update, which includes a/the workaround - unless a proper fix is not found already. There should be enough info for debugging provided already and given the impact of this bug, it makes not much sense to upset more users using the development branch. Reducing Importance. This bug so far seems to affect a very small number of people, and is not replicable by installing with a configuration involving a combination of MD, LVM and encrypted swap. It's therefore likely to be a problem caused by some people's existing configuration, which may not even be a configuration our installer can create. Please note that Importance is not Priority - I will still be investigating this, but right now, we wouldn't stop the release for it. Your help is required to determine what is unique about your systems that is causing this bug. Please attach a tar file containing your /etc/udev/rules.d and /lib/udev/rules.d directories as well as /dev itself
2009-02-22 19:51:28 Scott James Remnant (Canonical) udev: statusexplanation Reducing Importance. This bug so far seems to affect a very small number of people, and is not replicable by installing with a configuration involving a combination of MD, LVM and encrypted swap. It's therefore likely to be a problem caused by some people's existing configuration, which may not even be a configuration our installer can create. Please note that Importance is not Priority - I will still be investigating this, but right now, we wouldn't stop the release for it. Your help is required to determine what is unique about your systems that is causing this bug. Please attach a tar file containing your /etc/udev/rules.d and /lib/udev/rules.d directories as well as /dev itself But do track for the release
2009-02-22 19:51:28 Scott James Remnant (Canonical) udev: milestone jaunty-alpha-5
2009-02-22 20:13:45 Jonathan Hudson bug added attachment 'jh-332270.tar.gz' (jh-332270.tar.gz)
2009-02-22 20:53:44 TJ bug added attachment 'boot-udev-storm.log.gz' (Udev event storm during init-premount)
2009-02-22 20:56:27 jayboi bug added attachment 'udev.tar.gz' (udev.tar.gz)
2009-02-22 21:32:44 TJ bug added attachment 'boot-udev-storm-handle_inotify.log' (handle_inotify log messages)
2009-02-22 22:48:09 Albert Damen bug added attachment 'udev_rules.tar.gz' (udev_rules.tar.gz)
2009-02-22 22:48:09 Albert Damen bug added attachment 'dev.tar.gz' (dev.tar.gz)
2009-02-23 02:52:20 Luke Yelavich bug added attachment 'dpkg-l.log' (Dpkg package list)
2009-02-23 02:52:57 Luke Yelavich bug added attachment 'sysblock.log' (/sys/block)
2009-02-23 02:53:23 Luke Yelavich bug added attachment 'lvm-vgs.log' (lvm vgs)
2009-02-23 02:53:42 Luke Yelavich bug added attachment 'lvm-pvs.log' (lvm pvs)
2009-02-23 11:18:10 Oleksij Rempel bug added attachment 'trace' (trace)
2009-02-23 12:39:42 Christian Neugebauer bug added attachment 'munin-df-day.png' (munin-df-day.png)
2009-02-23 16:23:44 TJ description Since todays upgrade of udev jaunty is not able to boot anymore affects also recovery mode, of course. I can see messages like /sys/devices/pci0000:00/0000:00:1f.1/host0/target0:0:0/0:0:0:0/block/sda (9367) /sys/devices/pci0000:00/0000:00:1f.1/host0/target0:0:0/0:0:0:0/block/sda/sda2 (9366) /sys/devices/pci0000:00/0000:00:1f.1/host0/target0:0:0/0:0:0:0/block/sda/sda2 (9365) the messages appear in blocks of quite many, always repeating after 5 minutes or so and the number is counting up. Harddisk LED is flashing rapidly but no hearable access to the hard disk I think it has something to do with udev as this is in the last upgrade log 2009-02-20 20:59:24 upgrade libvolume-id1 137-2 138-1 2009-02-20 20:59:24 status unpacked libvolume-id1 138-1 2009-02-20 20:59:24 status unpacked libvolume-id1 138-1 2009-02-20 20:59:25 configure libvolume-id1 138-1 138-1 2009-02-20 20:59:25 status unpacked libvolume-id1 138-1 2009-02-20 20:59:25 status half-configured libvolume-id1 138-1 2009-02-20 20:59:25 status installed libvolume-id1 138-1 2009-02-20 20:59:26 upgrade udev 137-2 138-1 2009-02-20 20:59:27 status unpacked udev 138-1 2009-02-20 20:59:27 status unpacked udev 138-1 2009-02-20 20:59:34 upgrade libudev0 137-2 138-1 2009-02-20 20:59:34 status unpacked libudev0 138-1 2009-02-20 20:59:34 status unpacked libudev0 138-1 2009-02-20 20:59:41 configure udev 138-1 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status half-configured udev 138-1 2009-02-20 20:59:41 status installed udev 138-1 2009-02-20 20:59:47 configure libudev0 138-1 138-1 2009-02-20 20:59:47 status unpacked libudev0 138-1 2009-02-20 20:59:47 status half-configured libudev0 138-1 2009-02-20 20:59:47 status installed libudev0 138-1 SYMPTOMS Early in the boot process (during initial ram-disk script processing) the PC appears to stop although there may be a lot of repeated disk activity, possibly accompanied by messages to screen of the form: UEVENT[1235217664.944992] change /devices/pci0000:00/0000:00:09.0/host0/target0:0:0/0:0:0:0/block/sda (block) This happens after an automatic package upgrade to udev to 138-1. RECOVERY PROCEDURE If the PC is unable to complete start-up, effectively locking you out, this is how to regain control and install a work-around until a solution is available. 1. When the PC starts enter the GRUB menu. 2. Select the Recovery entry and edit it (press "e") - this starts in single-user mode with minimal services. 3. Select the "kernel" line and edit it (press "e") 4. Add " break=top" to the end of the line and press Enter. 5. With the "kernel" line highlighted press "b" to boot using the edited entry. The boot process will stop when the initial ram-disk scripts begin and you'll be faced with a busybox shell. Edit the rules that contain the problematic "watch" and remove it: for rule in /lib/udev/rules.d/*; do sed -i -e '/watch/s/watch//' $rule; done Modify the init script so the boot can continue: sed -i -e '/run-init/s,${init},/bin/bash,' /init Restart the boot process by pressing Ctrl+D The system *should* now boot to a root prompt although there may still be a delay. Once there, edit the non-initrd system rules: for rule in /lib/udev/rules.d/*; do sed -i -e '/watch/s/watch//' $rule; done And rebuild the initial ram-disk image so you don't have to go through this procedure at every boot: To update just the kernel version that is currently running: update-initramfs -u -k `uname -r` or to update for all installed kernels: update-initramfs -u -k all Now restart the PC and hopefully the regular multiuser start-up should go normally. -- TJ ----------------------------- Since todays upgrade of udev jaunty is not able to boot anymore affects also recovery mode, of course. I can see messages like /sys/devices/pci0000:00/0000:00:1f.1/host0/target0:0:0/0:0:0:0/block/sda (9367) /sys/devices/pci0000:00/0000:00:1f.1/host0/target0:0:0/0:0:0:0/block/sda/sda2 (9366) /sys/devices/pci0000:00/0000:00:1f.1/host0/target0:0:0/0:0:0:0/block/sda/sda2 (9365) the messages appear in blocks of quite many, always repeating after 5 minutes or so and the number is counting up. Harddisk LED is flashing rapidly but no hearable access to the hard disk I think it has something to do with udev as this is in the last upgrade log 2009-02-20 20:59:24 upgrade libvolume-id1 137-2 138-1 2009-02-20 20:59:24 status unpacked libvolume-id1 138-1 2009-02-20 20:59:24 status unpacked libvolume-id1 138-1 2009-02-20 20:59:25 configure libvolume-id1 138-1 138-1 2009-02-20 20:59:25 status unpacked libvolume-id1 138-1 2009-02-20 20:59:25 status half-configured libvolume-id1 138-1 2009-02-20 20:59:25 status installed libvolume-id1 138-1 2009-02-20 20:59:26 upgrade udev 137-2 138-1 2009-02-20 20:59:27 status unpacked udev 138-1 2009-02-20 20:59:27 status unpacked udev 138-1 2009-02-20 20:59:34 upgrade libudev0 137-2 138-1 2009-02-20 20:59:34 status unpacked libudev0 138-1 2009-02-20 20:59:34 status unpacked libudev0 138-1 2009-02-20 20:59:41 configure udev 138-1 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status unpacked udev 138-1 2009-02-20 20:59:41 status half-configured udev 138-1 2009-02-20 20:59:41 status installed udev 138-1 2009-02-20 20:59:47 configure libudev0 138-1 138-1 2009-02-20 20:59:47 status unpacked libudev0 138-1 2009-02-20 20:59:47 status half-configured libudev0 138-1 2009-02-20 20:59:47 status installed libudev0 138-1
2009-02-23 16:38:52 Scott James Remnant (Canonical) bug assigned to lvm2 (Ubuntu)
2009-02-23 16:45:18 Scott James Remnant (Canonical) lvm2: status New Triaged
2009-02-23 16:45:18 Scott James Remnant (Canonical) lvm2: importance Undecided High
2009-02-23 16:45:18 Scott James Remnant (Canonical) lvm2: statusexplanation There are two parts to this bug. The actual bug is being caused by "lvm vgscan" opening all block devices on the system writable and then closing them again, which triggers the inotify event. There seems to be no reason they cannot be opened readonly, since the only thing it does is call fstat() and read() on them. Since the devices include any LVM PVs (which is what it's looking for), this gives you a recursive loop where LVM PVs appear to have been changed, so udev calls "lvm vgscan" again to update its tables. Part #1 of the fix will be a patch to LVM to only open block devices with O_RDONLY Part #2 of the fix will be a patch to udev to not add the inotify watch until all RUN rules are processed; this will prevent the initial onslaught on boot, and hopefully guard against anything it runs changing the device resulting in reprocessing (which is better handled with udev rules ordering anyway). It will obviously not protect against a "udevadm trigger" once booted, so it's possible that we'll trip over this with another program later and will have to fix that one in the same manner as LVM. It's not yet clear why this does not affect _everyone_, there appears to be a race involved as slowing udev down with various log levels - or using only one core - is enough to hide the bug for some people. I need to work out why that is before uploading the fixes
2009-02-23 16:45:18 Scott James Remnant (Canonical) lvm2: milestone jaunty-alpha-5
2009-02-23 16:47:16 Scott James Remnant (Canonical) bug added attachment 'udev.inotify.patch' (udev patch to add inotify watch after processing RUN rules)
2009-02-23 20:35:14 Launchpad Janitor udev: status Triaged Fix Released
2009-02-23 22:30:07 Launchpad Janitor lvm2: status Triaged Fix Released
2009-02-23 22:33:58 Scott James Remnant (Canonical) bug added attachment 'open-readonly.patch' (Patch for LVM2 to open in readonly when only reading)
2009-04-13 16:56:32 Francisco Borges removed subscriber Francisco Borges
2009-06-28 04:08:09 Launchpad Janitor branch linked lp:ubuntu/karmic/lvm2
2010-08-23 07:20:39 Launchpad Janitor branch linked lp:~ubuntu-core-dev/ubuntu/maverick/udev/ubuntu