Unable to use use LVM with DRBD block devices as PV

[Tessa]

This is a bug reported in Debian and Redhat, which I'm currently seeing on fresh installs of Ubuntu/karmic. The details of my problem generally match the Debian bug report, which I'll link here.

ProblemType: Bug
Architecture: amd64
Date: Fri Nov 27 15:12:25 2009
DistroRelease: Ubuntu 9.10
Package: drbd8-utils 2:8.3.3-0ubuntu1
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_CA.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-15.50-server
SourcePackage: drbd8
Uname: Linux 2.6.31-15-server x86_64

[Wolfgang (wolfgang-redhat-bugs)]

Description of problem:

I'm trying to set up a nested LVM configuration with DRBD, see for example
http://www.drbd.org/users-guide-emb/s-nested-lvm.html . Creating logical volumes on top of a DRBD device fails.

Version-Release number of selected component (if applicable):

lvm2-2.02.48-2.fc11.i586
kernel-2.6.30.8-64.fc11.i586

How reproducible:

stable, happens always.

Steps to Reproduce:
1. set up a DRBD device
2. create a physical volume on it:
# pvcreate /dev/drbd0
  Physical volume "/dev/drbd0" successfully created
3. create a volume group on it:
# vgcreate replicated /dev/drbd0
  Volume group "replicated" successfully created
# vgscan
  Reading all physical volumes. This may take a while...
  Found volume group "replicated" using metadata type lvm2
  Found volume group "vg0" using metadata type lvm2
  Found volume group "vg1" using metadata type lvm2
# vgchange -a y replicated
  0 logical volume(s) in volume group "replicated" now active
4. try to create a logical volum:
# lvcreate --name foo --size 32G replicated
  device-mapper: reload ioctl failed: Invalid argument
  Aborting. Failed to activate new LV to wipe the start of it.

Actual results:

LV not created.

Error messages on stdout:
  device-mapper: reload ioctl failed: Invalid argument
  Aborting. Failed to activate new LV to wipe the start of it.
Syslog messages:
# Oct 24 21:44:01 gemini kernel: device-mapper: table: 253:11: linear: dm-linear: Device lookup failed
Oct 24 21:44:01 gemini kernel: device-mapper: ioctl: error adding target to table

Expected results:

LV created, no error messages.

Additional info:

I am pretty sure that used to work in older versions (probably F9 or F10),
but unfortunately I don't have any records of this setup any more.

There are similar bug reports from other distributions, so this seems to be a generic (kernel?) issue, see for example http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=533436

[Alasdair (alasdair-redhat-bugs)]

As per the Debian bug, run the tool with -vvvv, look for the line similar to this one:
       Adding target: 0 8388608 linear 252:2 384

Take the '252:2' there as major:minor and investigate the state of that device. Report all its attributes - use 'blockdev', find it in /sys/block etc. Check if anything has it open. Check its size etc. etc.

Also test with udev disabled in case that's interfering.

[Wolfgang (wolfgang-redhat-bugs)]

Created attachment 366101
Running pvcreate, vgcreate and lvcreate with "-vvvv" + additional info

[Wolfgang (wolfgang-redhat-bugs)]

OK, I attached the requested additional information - the only thing I
cannot do right now is testing without udev - I don't know how to
disable udev (and still have a resonably "normal" Fedora 11 running).
Please advice if really needed - although I don't see where udev
would interfere.

[Note this is from another system, actually the peer; problem is the
very same though.]

Please let me know if any additional information is requqired.

Wolfgang Denk, <email address hidden>

[Milan (milan-redhat-bugs)]

Please can you also attach output (tarball) from lvmdump?
(from some system where is /dev/drbd* device present and lvm fails)

[Wolfgang (wolfgang-redhat-bugs)]

Created attachment 366106
Output of "lvmdump -a -m"

[Milan (milan-redhat-bugs)]

Thanks.

There is something strange:
- you create pvcreate /dev/drbd0 (/dev/drbd0 is major:minor 147:0)
- but in metadata PV as /dev/dm-2 (which is 253:2 and in fact it is vg0/drbd0 volume mapped to 8:3 - which is /dev/sda3)

So now we need to find where this confusion happens.

[Milan (milan-redhat-bugs)]

seems one thing still missing in lvmdump - please can you paste output of these files?

cat /proc/devices /proc/partitions

Thanks.

[Wolfgang (wolfgang-redhat-bugs)]

-> cat /proc/devices
Character devices:
  1 mem
  4 /dev/vc/0
  4 tty
  4 ttyS
  5 /dev/tty
  5 /dev/console
  5 /dev/ptmx
  7 vcs
  9 st
 10 misc
 13 input
 14 sound
 21 sg
 29 fb
 86 ch
 99 ppdev
116 alsa
128 ptm
136 pts
162 raw
180 usb
188 ttyUSB
189 usb_device
202 cpu/msr
203 cpu/cpuid
206 osst
226 drm
248 firewire
249 hidraw
250 usb_endpoint
251 usbmon
252 bsg
253 pcmcia
254 rtc

Block devices:
  1 ramdisk
  2 fd
259 blkext
  7 loop
  8 sd
  9 md
 11 sr
 65 sd
 66 sd
 67 sd
 68 sd
 69 sd
 70 sd
 71 sd
128 sd
129 sd
130 sd
131 sd
132 sd
133 sd
134 sd
135 sd
147 drbd
253 device-mapper
254 mdp
-> cat /proc/partitions
major minor #blocks name

   8 0 244198584 sda
   8 1 128488 sda1
   8 2 4192965 sda2
   8 3 239874547 sda3
   8 16 390711384 sdb
   8 17 390708801 sdb1
 253 0 33554432 dm-0
 253 1 83886080 dm-1
 253 3 390705152 dm-3
 253 2 100663296 dm-2
 147 0 100660188 drbd0

[Milan (milan-redhat-bugs)]

We should probably add some especial case for DRBD similar to MDinto lvm code.

But for the time being, you can blacklist underlying device for DRBD,
see lvm.conf filter setting .

When you create new PV on /dev/drbdX device, lvm should see only PV on /dev/drbd and not the underlying device.

(run vgscan ; pvs and you see something like this:
# pvs
  PV VG Fmt Attr PSize PFree
  /dev/drbd0 vg_test lvm2 a- 1020.00M 620.00M

and no duplicate warnings like
  Found duplicate PV TQrcpi4AAEzMGDHYY1cbwsByUkpLuEjj: using /dev/sdf not /dev/drbd0)

Then DRBD PV works as expected. (Your configuration is more complicated, because you are using LV for backend DRBD device - but it still should work with proper filter setting.)

Maybe for testing, explicitly name your PVs in lvm.conf.

For your config it can be:
filter = [ "a/drbd/", "a/sda/", "a/sdb/", "r/.*/" ]

(or similar rule). Then run vgscan to refresh device cache and try again.

[Wolfgang (wolfgang-redhat-bugs)]

I'm afraid I don't get what you mean, or it doesn't work. I don't see any "Found duplicate PV" warnings:

# diff /etc/lvm/lvm.conf.ORIG /etc/lvm/lvm.conf
53c53,55
< filter = [ "a/.*/" ]
---
> #filter = [ "a/.*/" ]
> # For DEBD setup, accept only DRBD and "SCSI" disks
> filter = [ "a/drbd/", "a/sda/", "a/sdb/", "r/.*/" ]
# vgscan
  Reading all physical volumes. This may take a while...
  Found volume group "vg1" using metadata type lvm2
  Found volume group "vg0" using metadata type lvm2
# pvs
  PV VG Fmt Attr PSize PFree
  /dev/sda3 vg0 lvm2 a- 228.75G 20.75G
  /dev/sdb1 vg1 lvm2 a- 372.61G 0
# pvcreate /dev/drbd0
  Physical volume "/dev/drbd0" successfully created
# vgcreate replicated /dev/drbd0
  Volume group "replicated" successfully created
# vgscan
  Reading all physical volumes. This may take a while...
  Found volume group "vg1" using metadata type lvm2
  Found volume group "vg0" using metadata type lvm2
  Found volume group "replicated" using metadata type lvm2
# pvs
  PV VG Fmt Attr PSize PFree
  /dev/dm-2 replicated lvm2 a- 96.00G 96.00G
  /dev/sda3 vg0 lvm2 a- 228.75G 20.75G
  /dev/sdb1 vg1 lvm2 a- 372.61G 0
# lvcreate --name Mail --size 32G replicated
  device-mapper: reload ioctl failed: Invalid argument
  Aborting. Failed to activate new LV to wipe the start of it.

Oct 27 13:10:13 nyx kernel: device-mapper: table: 253:4: linear: dm-linear: Device lookup failed
Oct 27 13:10:13 nyx kernel: device-mapper: ioctl: error adding target to table

As far as I can tell nothing has changed. Did I miss something?

[Milan (milan-redhat-bugs)]

(In reply to comment #10)
> # pvs
> PV VG Fmt Attr PSize PFree
> /dev/dm-2 replicated lvm2 a- 96.00G 96.00G

this is stil wrong, it uses directly LV instead of drbd device.

Can you try
filter = [ "a|^/dev/drbd|", "a/sda/", "a/sdb/", "r/.*/" ]

I forgot you have drbd in LV name :)

[Wolfgang (wolfgang-redhat-bugs)]

Indeed - this gets it working:

# diff /etc/lvm/lvm.conf.ORIG /etc/lvm/lvm.conf
53c53,55
< filter = [ "a/.*/" ]
---
> #filter = [ "a/.*/" ]
> # For DEBD setup, accept only DRBD and "SCSI" disks
> filter = [ "a|^/dev/drbd|", "a/sda/", "a/sdb/", "r/.*/" ]
# vgscan
  Reading all physical volumes. This may take a while...
  Found volume group "vg1" using metadata type lvm2
  Found volume group "vg0" using metadata type lvm2
# pvs
  PV VG Fmt Attr PSize PFree
  /dev/sda3 vg0 lvm2 a- 228.75G 20.75G
  /dev/sdb1 vg1 lvm2 a- 372.61G 0
# pvcreate /dev/drbd0
  Physical volume "/dev/drbd0" successfully created
# vgcreate replicated /dev/drbd0
  Volume group "replicated" successfully created
# pvs
  PV VG Fmt Attr PSize PFree
  /dev/drbd0 replicated lvm2 a- 96.00G 96.00G
  /dev/sda3 vg0 lvm2 a- 228.75G 20.75G
  /dev/sdb1 vg1 lvm2 a- 372.61G 0
# lvcreate --name Mail --size 32G replicated
  Logical volume "Mail" created

Thanks a lot!

[Milan (milan-redhat-bugs)]

Patch to automatically prefer DRBD top device similar to MD mirror images is here
https://www.redhat.com/archives/lvm-devel/2009-October/msg00229.html

[Tessa (unit3)]

The Redhat bug has a more thorough investigation of the issue, as well as a potential patch we can use. The Debian bug has a partial workaround using kpartx.

[Tessa (unit3)]

/etc/lvm/lvm.conf filter suggestion from redhat bug works well, actually. After filtering out the block device my drbd device is based on, lvcreate works as expected.

[Ante Karamatić (ivoks)]

This is not a bug in drbd, but in lvm.

[Milan (milan-redhat-bugs)]

The code is for some time in rawhide and workaround (filters) exists for older systems, so I am closing this as fixed in rawhide/F13.

[Alasdair G. Kergon (agk2)]

Dealt with upstream a while back.