/dev/dm-10 is not created for LVM volume

[Arie Skliarouk]

Binary package hint: udev

After I created 10th logical volume (shared - see below), udev did not create /dev/dm-10 device node:
root@mf:~# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 236 2011-06-05 09:04 control
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-devs -> ../dm-3
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-devs2 -> ../dm-5
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-devs3 -> ../dm-6
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-devs4 -> ../dm-8
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-dspp -> ../dm-1
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-lxc -> ../dm-0
lrwxrwxrwx 1 root root 7 2011-06-12 00:52 mf-mail -> ../dm-9
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-matchbox -> ../dm-4
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-muze -> ../dm-2
lrwxrwxrwx 1 root root 8 2011-06-14 17:18 mf-shared -> ../dm-10
lrwxrwxrwx 1 root root 7 2011-06-05 09:04 mf-tests -> ../dm-7
root@mf:~# ls -l /dev/dm*
brw-rw---- 1 root disk 251, 0 2011-06-05 09:04 /dev/dm-0
brw-rw---- 1 root disk 251, 1 2011-06-05 09:04 /dev/dm-1
brw-rw---- 1 root disk 251, 2 2011-06-05 09:04 /dev/dm-2
brw-rw---- 1 root disk 251, 3 2011-06-05 09:04 /dev/dm-3
brw-rw---- 1 root disk 251, 4 2011-06-05 09:04 /dev/dm-4
brw-rw---- 1 root disk 251, 5 2011-06-05 09:04 /dev/dm-5
brw-rw---- 1 root disk 251, 6 2011-06-05 09:04 /dev/dm-6
brw-rw---- 1 root disk 251, 7 2011-06-05 09:04 /dev/dm-7
brw-rw---- 1 root disk 251, 8 2011-06-05 09:04 /dev/dm-8
brw-rw---- 1 root disk 251, 9 2011-06-12 00:52 /dev/dm-9
root@mf:~#

Ubuntu 11.04 server amd64, kernel 2.6.38-8-server.
ii udev 167-0ubuntu3 rule-based device node and kernel event manager
ii lvm2 2.02.66-4ubuntu2 The Linux Logical Volume Manager
ii libdevmapper-event1.02.1 2:1.02.48-4ubuntu2 The Linux Kernel Device Mapper userspace library
ii libdevmapper1.02.1 2:1.02.48-4ubuntu2 The Linux Kernel Device Mapper userspace library

[G. (g.-redhat-bugs)]

I think I've found why none of my lvm mounts are working on boot up.

First the vgchange in fedora-storage-init never seems to get hit. And udev seems to have a problem setting up the drivers as well.

[root@liandra systemd]# /sbin/lvm vgchange -a y --sysinit
  10 logical volume(s) in volume group "LiandraStore" now active
  semid 32768: semop failed for cookie 0xd4d83af: incorrect semaphore state
  Failed to set a proper state for notification semaphore identified by cookie value 223183791 (0xd4d83af) to initialize waiting for incoming notifications.
  /dev/mapper/LiandraStore-VirtualMachines not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-Old--Liandra not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-Backups not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-LiveCD not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-iTunes_Media not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-DVD--Working not set up by udev: Falling back to direct node creation.
  The link /dev/LiandraStore/VirtualMachines should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/Old-Liandra should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/Backups should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/LiveCD should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/iTunes_Media should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/DVD-Working should had been created by udev but it was not found. Falling back to direct link creation.

Any ideas how I debug this further?

[G. (g.-redhat-bugs)]

This may be related to by dbus not starting as well in bug: 689861. Not sure.

[Harald (harald-redhat-bugs)]

are you sure udevd is running?

[G. (g.-redhat-bugs)]

I'm assuming it is:

[root@liandra ~]# ps -ef | grep udev
root 488 1 0 Mar23 ? 00:00:00 /sbin/udevd
root 18778 18319 0 09:12 pts/1 00:00:00 grep --color=auto udev

[root@liandra ~]# lvcreate -L 10G -n Test LiandraStore
  /dev/mapper/LiandraStore-Test not set up by udev: Falling back to direct node creation.
  The link /dev/LiandraStore/Test should had been created by udev but it was not found. Falling back to direct link creation.
  Logical volume "Test" created

[root@liandra ~]# lvremove LiandraStore/Test
Do you really want to remove active logical volume Test? [y/n]: y
  Logical volume "Test" successfully removed
  semid 98304: semop failed for cookie 0xd4d613a: incorrect semaphore state
  Failed to set a proper state for notification semaphore identified by cookie value 223174970 (0xd4d613a) to initialize waiting for incoming notifications.

Any further tests I ran run to verify it is running correctly?

[Harald (harald-redhat-bugs)]

any ideas from the lvm team, what's going wrong here?

[Milan (milan-redhat-bugs)]

This usually means, that you are using udev-enabled lvm (default in Fedora) but there are missing lvm udev rules (explicitly rule with "dmsetup udevcomplete" call, 95-dm-notify.rules).

lvm will fallback to manual node removal (I think that the warning is about semaphore state is some minor bug in lvm code but it is just consequence - if udev rules missing, your system is simply misconfigured.)

Verify, that device-mapper is properly installed.

To diagnose it, please run the lvm command with -vvvv and add output here.

[G. (g.-redhat-bugs)]

Created attachment 488442
Verbose lvcreate command

[G. (g.-redhat-bugs)]

Also verified the /lib/udev/rules.d/95-dm-notify.rules is the same in F15 as it is in other releases.

[Peter (peter-redhat-bugs)]

What's the kernel version used? Is this a stock Fedora kernel?

[G. (g.-redhat-bugs)]

Yes it's stock. 2.6.38-1.fc15.x86_64

[Peter (peter-redhat-bugs)]

Well, based on the log, uevent was not generated correctly:

  " Uevent not generated! Calling udev_complete internally to avoid process lock-up."

This decrements the value of the semaphore used for synchronization. However, if the event (for some reason) *is* generated despite the fact that the kernel function responsible for sending the uevent reports a failure and then we decrement the value once again from within the udev rules based on the generated event (/lib/udev/rules.d/95-dm-notify.rules), we can end up with that:

  "semid 32768: semop failed for cookie 0xd4d83af: incorrect semaphore state..."

The other messages about "... not set up by udev. Falling back to direct node creation" is just a consequence of failed synchronisation with udev (since we decremented semaphore value prematurely).

However I'm not quite sure now why this happens. It could be a bug in kernel code as well... (but I can't reproduce that)

[Harald (harald-redhat-bugs)]

you might also try the new udev version in F15

[G. (g.-redhat-bugs)]

New version in F15 seems to have solved the problems.

[Arie Skliarouk (skliarie)]

Something weird happens:
The /dev/dm-10 is created on a freshly installed ubuntu 11.04. When I tried to create the logical volume again on the affected machine - it worked, albeit with a strange message:

  The link /dev/mf/lvmtest should had been created by udev but it was not found. Falling back to direct link creation.
  Logical volume "lvmtest" created

When I tried to remove it, I got an frightening message:

root@mf:~# lvremove /dev/mf/lvmtest
Do you really want to remove active logical volume lvmtest? [y/n]: y
  semid 524288: semop failed for cookie 0xd4d2311: incorrect semaphore state
  Failed to set a proper state for notification semaphore identified by cookie value 223159057 (0xd4d2311) to initialize waiting for incoming notifications.
  Logical volume "lvmtest" successfully removed

Googling for the keywords revealed similar problem on Fedora F15, which was solved by updated version of udev:
https://bugzilla.redhat.com/show_bug.cgi?id=690308

So looks like initial assignment of the bug to udev was correct after all.

[Dave Gilbert (ubuntu-treblig)]

Original reporter had tried to change it to lvm2 from udev as part of another report; merged as dup and changed.

[Arie Skliarouk (skliarie)]

I tried to understand the changes done to the udev-167-2 in F15 to fix the problem.
The commitdiffs can be see on the following page ("fixed udev-trigger.service"):
http://pkgs.fedoraproject.org/gitweb/?p=udev.git;a=shortlog

But looks like the bug already was fixed in the udev of ubuntu...

Dave, do you imply that the bug is really in lvm2 subsystem?

[Arie Skliarouk (skliarie)]

Until this is fixed, I wrote a shell script that looks in the /dev/mapper directory and recreates nodes that the udev missed.

Helps to me, might help to someone else.

[Dave Gilbert (ubuntu-treblig)]

Hi Arie,
  No I'm not sure if it's lvm2 or udev, just that you originally had two separate bug reports one for each and I just merged them and it looked like you'd made the 2nd one lvm2 hence why I changed this one.

Incidentally, I've seen that semaphore error when using cryptsetup luksOpen to open crypted fs; it doesn't seem to cause problems.

At the moment I seem to be getting all the entries in /dev/dm-* and /dev/mapper/* , but I've seen some errors/warnings on and off that I was attributing to problems with COW and similar lvm devices; but I could never really pin it down.

Dave

[Arie Skliarouk (skliarie)]

Here is the screenshot of the console when one can see failed attempts of udev to create the device nodes.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Arie Skliarouk (skliarie)]

The bug still exists in ubuntu 13.04.