power g5 does not boot lucid beta

[Sergey V. Udaltsov]

I upgraded Power G5. It went fine till last apt-get dist-upgrade. Now, it is not booting.

When I am booting to single-user, I see the following:

First, a lot of messages from udevd about "SYSFS will be removed in a future"

Then

/: clean, .... files, .... blocks
init: mounted-tmp main process (805) terminated with status 127
mountall: Event failed
/boot: recovering journal
/boot: clean, .... files, .... blocks
init: ureadahear-other main process (844) terminated with status 4

That's it. After that point everything is frozen (even kbd indicators do not work).

I am using LVM on /usr /home /var if it makes any difference.

I can boot with init=/bin/sh

If you need more info, please let me know

[Sergey V. Udaltsov (sergey-udaltsov)]

kernel 2.6.32-18

[Sergey V. Udaltsov (sergey-udaltsov)]

(powerpc64)

[Sergey V. Udaltsov (sergey-udaltsov)]

sorry for the noise. the full name as reported by uname
Linux (none) 2.6.32-18-powerpc64-smp #27-Ubuntu SMP Sat Mar 27 03:27:46 UTC 2010 ppc64 GNU/Linux

[Sergey V. Udaltsov (sergey-udaltsov)]

The latest update of mountall did fix the situation.

It might be related to #527666. I do use LVM. And when booting in "graphical" mode I see that message "Waiting for /var [SM]"

[Sergey V. Udaltsov (sergey-udaltsov)]

Sorry, it did NOT fix the situation

[Sergey V. Udaltsov (sergey-udaltsov)]

After the latest fix 2.10, I see the following: it does not wait for var, it reports failed mount, asking to skip (S) or manually fix (M).

If I am going for the manual fix, I see the following:

The /dev/mapper/control file is the only file in /dev/mapper. Nothing else in that dir.

vgdisplay and lvdispay show proper VG and LVs.

[Scott James Remnant (Canonical) (canonical-scott)]

Add --debug to /etc/init/mountall.conf (the exec line) and boot without "splash" - can you capture the screen at the point that mount fails?

[Scott James Remnant (Canonical) (canonical-scott)]

Just read your last comment, sorry; given you have no device nodes in /dev/mapper, this implies an lvm issue

[Sergey V. Udaltsov (sergey-udaltsov)]

I run mountall --debug. Here is what I captured to the file

[Sergey V. Udaltsov (sergey-udaltsov)]

there is no /dev/VG1 directory (VG1 is the name of my volume group)

[Sergey V. Udaltsov (sergey-udaltsov)]

Here is udev log

[Scott James Remnant (Canonical) (canonical-scott)]

Your udev log shows no dm-X events, so no LVM devices are being created

[Sergey V. Udaltsov (sergey-udaltsov)]

Doing apt-get install --reinstall does not help. Tried both mountall and lvm2

[Sergey V. Udaltsov (sergey-udaltsov)]

I found the matter of problem!

For some reason, my VG gets disabled on every boot! When I do lvscan (after getting into manual fix mode), I see all volumes are inactive. After I do vgchange -a y, the VG gets enabled and I can continue the boot process. And this happens at every boot.

Is this an lvm issue or mountall issue?

[Sergey V. Udaltsov (sergey-udaltsov)]

There is very similar one: https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/554478

[Scott James Remnant (Canonical) (canonical-scott)]

This is an LVM issue - mountall doesn't get involved in LVM bring-up *at all*, it only mounts the resulting filesystems once they appear.

It's up to LVM to activate devices as it goes

[Steve Langasek (vorlon)]

From your udev log:

UDEV [1271194892.224159] add /devices/pci0001:00/0001:00:07.0/0001:05:0c.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb1 (block)
UDEV_LOG=3
ACTION=add
DEVPATH=/devices/pci0001:00/0001:00:07.0/0001:05:0c.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb1
SUBSYSTEM=block
DEVNAME=/dev/sdb1
DEVTYPE=partition
SEQNUM=1102
ID_ATA=1
ID_TYPE=disk
ID_BUS=ata
ID_MODEL=DW_CDW5200SJ5-N5BC_1
ID_MODEL_ENC=DW\x20CDW5200SJ5-N5BC\x201\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
ID_REVISION=010.E210
ID_SERIAL=DW_CDW5200SJ5-N5BC_1_W_-DCWNAHK011331
ID_SERIAL_SHORT=W_-DCWNAHK011331
ID_ATA_WRITE_CACHE=1
ID_ATA_WRITE_CACHE_ENABLED=1
ID_ATA_FEATURE_SET_HPA=1
ID_ATA_FEATURE_SET_HPA_ENABLED=1
ID_ATA_FEATURE_SET_PM=1
ID_ATA_FEATURE_SET_PM_ENABLED=1
ID_ATA_FEATURE_SET_SECURITY=1
ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=0
ID_ATA_FEATURE_SET_SMART=1
ID_ATA_FEATURE_SET_SMART_ENABLED=1
ID_ATA_FEATURE_SET_AAM=1
ID_ATA_FEATURE_SET_AAM_ENABLED=1
ID_ATA_FEATURE_SET_AAM_VENDOR_RECOMMENDED_VALUE=128
ID_ATA_FEATURE_SET_AAM_CURRENT_VALUE=128
ID_ATA_FEATURE_SET_PUIS=1
ID_ATA_FEATURE_SET_PUIS_ENABLED=0
ID_ATA_DOWNLOAD_MICROCODE=1
ID_ATA_SATA=1
ID_ATA_SATA_SIGNAL_RATE_GEN2=1
ID_ATA_SATA_SIGNAL_RATE_GEN1=1
ID_SCSI_COMPAT=SATA_WDC_WD2500JS-55_WD-WCANKH103113
ID_PATH=pci-0001:05:0c.0-scsi-1:0:0:0
ID_PART_TABLE_TYPE=dos
ID_FS_UUID=53eHzV-oR6r-SImQ-bO63-BxGa-liKx-P4c215
ID_FS_UUID_ENC=53eHzV-oR6r-SImQ-bO63-BxGa-liKx-P4c215
ID_FS_VERSION=LVM2\x20001
ID_FS_TYPE=LVM2_member
ID_FS_USAGE=raid
UDISKS_PRESENTATION_NOPOLICY=0
UDISKS_PARTITION=1
UDISKS_PARTITION_SCHEME=mbr
UDISKS_PARTITION_NUMBER=1
UDISKS_PARTITION_TYPE=0x83
UDISKS_PARTITION_SIZE=250056705024
UDISKS_PARTITION_SLAVE=/sys/devices/pci0001:00/0001:00:07.0/0001:05:0c.0/host1/target1:0:0/1:0:0:0/block/sdb
UDISKS_PARTITION_OFFSET=32256
UDISKS_PARTITION_ALIGNMENT_OFFSET=0
MAJOR=8
MINOR=17
DEVLINKS=/dev/block/8:17 /dev/disk/by-id/ata-DW_CDW5200SJ5-N5BC_1_W_-DCWNAHK011331-part1 /dev/disk/by-id/scsi-SATA_WDC_WD2500JS-55_WD-WCANKH103113-part1 /dev/disk/by-path/pci-0001:05:0c.0-scsi-1:0:0:0-part1

This should match the udev rule in /lib/udev/rules.d/85-lvm2.rules, triggering 'vgchange -a y' to be called as soon as /dev/sdb1 is seen. Do you have that file on your system? Can you try adding an additional RUN+="echo lvm2 udev rule called > /dev/lvm-logfile", to confirm that the rule is being triggered?

[Kees Cook (kees)]

Can you verify in your initramfs that the udev rules exist, and that the "watershed" binary is available? From you booted system, what is the output of "ls -l /etc/udev/rules.d/" (perhaps something old has snuck in and is messing with stuff)? And maybe even "ls -l /lib/udev/rules.d/" for good measure too.

[Sergey V. Udaltsov (sergey-udaltsov)]

$ ls -l /lib/udev/rules.d/85-lvm2.rules
-rw-r--r-- 1 root root 272 2010-02-18 12:59 /lib/udev/rules.d/85-lvm2.rules

$ locate watershed
/lib/udev/watershed
/usr/share/doc/watershed
/usr/share/doc/watershed/changelog.gz
/usr/share/doc/watershed/copyright
/usr/share/initramfs-tools/hooks/watershed
/var/cache/apt/archives/watershed_5_powerpc.deb
/var/lib/dpkg/info/watershed.list
/var/lib/dpkg/info/watershed.md5sums
/var/lib/dpkg/info/watershed.postinst

$ ls -l /etc/udev/rules.d/
total 80
lrwxrwxrwx 1 root root 22 2006-09-10 20:49 025_logitechmouse.rules -> ../logitechmouse.rules
-rw-r--r-- 1 root root 108 2006-12-10 14:56 10-visor.rules
-rw-r--r-- 1 root root 34568 2008-04-10 17:41 45-libmtp7.rules
-rw-r--r-- 1 root root 1704 2008-06-19 10:55 45-libnjb5.rules
-rw-r--r-- 1 root root 1451 2006-04-21 13:14 45-logitechmouse.rules
-rw-r--r-- 1 root root 104 2007-03-27 15:36 65-wacom.rules
-rw-r--r-- 1 root root 1246 2009-10-06 22:06 70-persistent-cd.rules
-rw-r--r-- 1 root root 550 2009-05-09 17:03 70-persistent-net.rules
-rw-r--r-- 1 root root 289 2007-04-10 14:11 85-evms.rules
-rw-r--r-- 1 root root 297 2006-05-17 11:35 85-kino.rules
-rw-r--r-- 1 root root 107 2009-09-07 00:40 99-btnx.rules
-rw-r--r-- 1 root root 1157 2009-09-24 04:02 README

$ ls -l /lib/udev/rules.d/
total 660
-rw-r--r-- 1 root root 29 2010-01-27 00:28 40-fuse-utils.rules
-rw-r--r-- 1 root root 583 2010-02-03 22:15 40-gnupg2.rules
-rw-r--r-- 1 root root 583 2010-01-06 01:09 40-gnupg.rules
-rw-r--r-- 1 root root 6733 2010-04-12 11:50 40-hplip.rules
-rw-r--r-- 1 root root 90 2009-12-03 12:45 40-ia64.rules
-rw-r--r-- 1 root root 410 2009-12-03 12:45 40-infiniband.rules
-rw-r--r-- 1 root root 180 2009-12-03 12:45 40-isdn.rules
-rw-r--r-- 1 root root 297 2010-02-11 01:33 40-kino.rules
-rw-r--r-- 1 root root 150222 2010-03-19 22:38 40-libgphoto2-2.rules
-rw-r--r-- 1 root root 2656 2010-01-06 08:15 40-libpisock9.rules
-rw-r--r-- 1 root root 42579 2010-02-08 20:25 40-libsane.rules
-rw-r--r-- 1 root root 2874 2009-10-25 07:09 40-lomoco.rules
-rw-r--r-- 1 root root 174 2009-12-03 12:45 40-pilot-links.rules
-rw-r--r-- 1 root root 212 2009-12-03 12:45 40-ppc.rules
-rw-r--r-- 1 root root 14829 2010-04-11 13:24 40-usb-media-players.rules
-rw-r--r-- 1 root root 334 2009-12-03 12:45 40-zaptel.rules
-rw-r--r-- 1 root root 30 2010-01-27 00:28 45-fuse.rules
-rw-r--r-- 1 root root 72677 2010-02-17 09:45 45-libmtp8.rules
-rw-r--r-- 1 root root 1674 2010-03-06 18:33 45-libnjb5.rules
-rw-r--r-- 1 root root 219 2010-04-13 15:01 50-firmware.rules
-rw-r--r-- 1 root root 4140 2010-04-13 15:01 50-udev-default.rules
-rw-r--r-- 1 root root 2643 2010-02-18 12:59 55-dm.rules
-rw-r--r-- 1 root root 445 2010-04-12 11:50 56-hpmud_support.rules
-rw-r--r-- 1 root root 558 2010-02-18 12:59 56-lvm.rules
-rw-r--r-- 1 root root 240 2010-04-13 15:01 60-cdrom_id.rules
-rw-r--r-- 1 root root 250 2010-04-13 15:01 60-floppy.rules
-rw-r--r-- 1 root root 676 2010-04-13 15:01 60-persistent-alsa.rules
-rw-r--r-- 1 root root 2005 2010-04-13 15:01 60-persistent-input.rules
-rw-r--r-- 1 root root 893 2010-04-13 15:01 60-persistent-serial.r...

[Sergey V. Udaltsov (sergey-udaltsov)]

sorry for giving the wrong url to similar bug:

https://bugs.launchpad.net/ubuntu/+bug/368551

[Sergey V. Udaltsov (sergey-udaltsov)]

> Can you try adding an additional RUN+="echo lvm2 udev rule called > /dev/lvm-logfile", to confirm that the rule is being triggered?

That file does not appear (/dev/lvm-logfile).

ps -ef shows some udev processes

[Sergey V. Udaltsov (sergey-udaltsov)]

/proc/self/mountinfo

[Sergey V. Udaltsov (sergey-udaltsov)]

Seems related to #560717.

Anyway, the root cause is found: device-mapper is a module (in ports!!!) so it is not loaded when necessary.