dmraid and LVM are incompatible

[justin]

Binary package hint: dmraid

I have an Asus m2n-sli motherboard with nvidia raid chipset. I have setup my 2 seagate harddrives to mirror in the raid setup. I am dualbooting between windows xp and feisty. I installed successfully using the fakeraid howto. Here are some outputs of commands to get an idea of my layout.

root@ziggy:~# dmraid -s
*** Active Set
name : nvidia_ajafeffc
size : 625142400
stride : 128
type : mirror
status : ok
subsets: 0
devs : 2
spares : 0
root@ziggy:~# dmraid -l
asr : Adaptec HostRAID ASR (0,1,10)
ddf1 : SNIA DDF1 (0,1,4,5,linear)
hpt37x : Highpoint HPT37X (S,0,1,10,01)
hpt45x : Highpoint HPT45X (S,0,1,10)
isw : Intel Software RAID (0,1)
jmicron : JMicron ATARAID (S,0,1)
lsi : LSI Logic MegaRAID (0,1,10)
nvidia : NVidia RAID (S,0,1,10,5)
pdc : Promise FastTrack (S,0,1,10)
sil : Silicon Image(tm) Medley(tm) (0,1,10)
via : VIA Software RAID (S,0,1,10)
dos : DOS partitions on SW RAIDs

root@ziggy:~# fdisk -l /dev/mapper/nvidia_ajafeffc

Disk /dev/mapper/nvidia_ajafeffc: 320.0 GB, 320072932352 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

                      Device Boot Start End Blocks Id System
/dev/mapper/nvidia_ajafeffc1 * 1 5099 40957686 7 HPFS/NTFS
/dev/mapper/nvidia_ajafeffc2 5100 9969 39118275 8e Linux LVM
/dev/mapper/nvidia_ajafeffc3 9970 9982 97656+ 83 Linux
/dev/mapper/nvidia_ajafeffc4 9983 38913 232388257+ 7 HPFS/NTFS

root@ziggy:~# fdisk -l /dev/sda

Disk /dev/sda: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot Start End Blocks Id System
/dev/sda1 * 1 5099 40957686 7 HPFS/NTFS
/dev/sda2 5100 9969 39118275 8e Linux LVM
/dev/sda3 9970 9982 97656+ 83 Linux
/dev/sda4 9983 38913 232388257+ 7 HPFS/NTFS

root@ziggy:~# fdisk -l /dev/sdb

Disk /dev/sdb: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot Start End Blocks Id System
/dev/sdb1 * 1 5099 40957686 7 HPFS/NTFS
/dev/sdb2 5100 9969 39118275 8e Linux LVM
/dev/sdb3 9970 9982 97656+ 83 Linux
/dev/sdb4 9983 38913 232388257+ 7 HPFS/NTFS

root@ziggy:~# cat /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
proc /proc proc rw 0 0
/dev/volgroup/root / ext3 defaults,errors=remount-ro 0 1
/dev/volgroup/swap none swap sw 0 0
/dev/mapper/nvidia_ajafeffc3 /boot ext3 defaults 0 2
/dev/mapper/nvidia_ajafeffc4 /media/storage ntfs-3g defaults 0 0

root@ziggy:~# dmraid -ay
RAID set "nvidia_ajafeffc" already active
ERROR: opening "/dev/.static/dev/mapper/nvidia_ajafeffc"

output of revelant menu.lst from grub

title Ubuntu, kernel 2.6.20-16-generic
root (hd0,2)
kernel /vmlinuz-2.6.20-16-generic root=/dev/mapper/volgroup-root ro quiet splash
initrd /initrd.img-2.6.20-16-generic

title Ubuntu, kernel 2.6.20-16-generic (recovery mode)
root (hd0,2)
kernel /vmlinuz-2.6.20-16-generic root=/dev/mapper/volgroup-root ro single
initrd /initrd.img-2.6.20-16-generic

title Ubuntu, memtest86+
root (hd0,2)
kernel /memtest86+.bin

title Windows XP
rootnoverify (hd0,0)
chainloader +1

root@ziggy:~# lvdisplay
  --- Logical volume ---
  LV Name /dev/volgroup/root
  VG Name volgroup
  LV UUID 6NjirO-b8ap-Dgqp-9LlS-zIuZ-jB0W-UgjoAc
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 36.43 GB
  Current LE 9325
  Segments 1
  Allocation inherit
  Read ahead sectors 0
  Block device 254:0

  --- Logical volume ---
  LV Name /dev/volgroup/swap
  VG Name volgroup
  LV UUID fg5ubR-Okhe-6Ezy-sNx2-JwOd-pYpY-hPXC4Q
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 900.00 MB
  Current LE 225
  Segments 1
  Allocation inherit
  Read ahead sectors 0
  Block device 254:1

root@ziggy:~# pvdisplay
  --- Physical volume ---
  PV Name /dev/mapper/nvidia_ajafeffc2
  VG Name volgroup
  PV Size 37.30 GB / not usable 0
  Allocatable yes (but full)
  PE Size (KByte) 4096
  Total PE 9550
  Free PE 0
  Allocated PE 9550
  PV UUID eonDPR-9lMv-88T5-v92e-gOh7-eL40-obvWBz

And now for a description of the bug. After I did my install. I rebooted off the livecd and went about installing all my applications, configuring my settings ,etc. Rebooted a couple of times and everything was fine. Then I rebooted once and all of a sudden I'm booted into the same environment I was in post install and none of my files are there? I'm confused as to what could have happened. I reboot again and I boot back into my setup that I previously had. Right now it's 50/50 as to which installation I will boot to when I reboot. I am at a loss for how this is occuring. Windows is behaving properly in the mirror, and all the commands seem to point to the mirror working in linux yet it seems that there are 2 seperate installs and no mirroring taking place. Please let me know if you need any clarification or any more output from commands. I am at a loss for how this is occuring.

[Phillip Susi (psusi)]

Can you post the output of "sudo dmsetup table"?

[justin (jlintz)]

nvidia_ajafeffc: 0 625142446 mirror core 2 131072 nosync 2 8:0 0 8:16 0

nvidia_ajafeffc4: 0 464776515 linear 254:2 160360830

nvidia_ajafeffc3: 0 195313 linear 254:2 160156250

nvidia_ajafeffc2: 0 78236550 linear 254:2 81915435

nvidia_ajafeffc1: 0 81915372 linear 254:2 63

volgroup-swap: 0 1843200 linear 8:2 76390784

volgroup-root: 0 76390400 linear 8:2 384

> Can you post the output of "sudo dmsetup table"?
>
> --
> dmraid says mirroring but isn't
> https://bugs.launchpad.net/bugs/129285
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
- Justin Lintz

[Phillip Susi (psusi)]

Your lvm swap and root are not going through the dmraid mirror, instead they are directly on the primary disk. Is there a reason you are using LVM?

I think the problem is that lvm sees the lvm partition on the primary disk and takes control of it, as it is not dmraid aware. This issue needs resolved somehow between the two tools so that lvm looks at the dmraid mirror device and NOT the underlying disks. Adding lvm to the effected list and marking bug as triaged.

[justin (jlintz)]

I wanted to be able to resize my root partition to experiment with
creating xen images. Thanks for the help. Hopefully can get this
resolved/fixed.

On 8/1/07, Phillip Susi <email address hidden> wrote:
> Your lvm swap and root are not going through the dmraid mirror, instead
> they are directly on the primary disk. Is there a reason you are using
> LVM?
>
> I think the problem is that lvm sees the lvm partition on the primary
> disk and takes control of it, as it is not dmraid aware. This issue
> needs resolved somehow between the two tools so that lvm looks at the
> dmraid mirror device and NOT the underlying disks. Adding lvm to the
> effected list and marking bug as triaged.
>
>
> ** Also affects: lvm-common (Ubuntu)
> Importance: Undecided
> Status: New
>
> ** Changed in: dmraid (Ubuntu)
> Importance: Undecided => Low
> Status: New => Triaged
>
> ** Changed in: lvm-common (Ubuntu)
> Status: New => Triaged
>
> --
> dmraid says mirroring but isn't
> https://bugs.launchpad.net/bugs/129285
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
- Justin Lintz

[Phillip Susi (psusi)]

I wouldn't hold your breath. Hopefully it will be fixed by 8.04 but it won't be in 7.10. In the mean time I suggest that you drop the use of lvm; you don't need it to resize anyhow.

[Alasdair G. Kergon (agk2)]

On Mon, Jul 30, 2007 at 06:12:13PM -0000, justin wrote:
> root@ziggy:~# lvdisplay

I'm puzzled why there are no warning messages printed here.

Please can you send the contents of /etc/lvm/lvm.conf and
also the full output of 'vgscan -vvvv'?

Alasdair
--
<email address hidden>

[justin (jlintz)]

Unfortunately I have since repartitioned to not use lvm. =/

[Kees Cook (kees)]

As a work-around, you can explicity tell LVM to ignore the raw devices, too, in the "devices" section of /etc/lvm/lvm.conf:

    filter = [ "r|^/dev/sd.*|" ]

[TJF (tjfoerster)]

Hi,

i am trying (!) to install xubuntu 7.10 from live-cd. Installing dmraid is no problem. I get the right /dev/mapper/sil_... Every time the installation breaks because of the swap-partition-problem (LVM). I cannot find any way to disable this automatic LVM choice at the installation, because i don't need it. I have nowhere choosen LVM. Before formatting the partitions, I can see (overview), what will be done...

Can anybody help me, to disable this?

Thanks a lot!
Thomas

[Phillip Susi (psusi)]

As far as I know, the livecd does not even have an option to install using LVM; it only uses standard partitions. You haven't provided much information to go on here though, for instance, I don't know what you mean by "the swap-partition-problem", or what makes you think you are using LVM. Heck, the last time I tried the installer wouldn't work so you have to bootstrap the system by hand per https://wiki.ubuntu.com/FakeRaidHowto.

[TJF (tjfoerster)]

I made it this way: http://ubuntuforums.org/showthread.php?t=464758

After installing dmraid and debootstrap I can see in gparted the two single /dev/sda and /dev/sdb. Also /dev/mapper/sil_.... During this phase nothing is formatted but partitiones. Later, when the installation started, and the partitioning seems to work (see my link above). The conclusion shows that the single partitions will be formatted this way: "LVM VG sil_..." The installation breaks, with the message (similar...): "Trying to write a file sytem, type swap, failed. ... could not be mounted". After this - starting gparted - I can see that changes happened on sda and sdb... So it seems to me, that swap cannot be handled in /dev/mapper/sil_....

[Phillip Susi (psusi)]

You have to install manually using debootstrap. You can not use the normal installer, and you certainly don't do both. At any rate, this discussion has nothing to do with this bug report, so let's stop cluttering it up.

[Charles Duffy (M1) (cduffy-messageone)]

This configuration can be made to work, but there are some packages which need changes for it to be out-of-the-box:

Changes to the lvm2 package:
 - /etc/lvm/lvm.conf should have as a default preferred_names = [ "^/dev/mapper/" ]
 - /usr/share/initramfs-tools/hooks/lvm2 should list dmraid as a prerequisite (so PREREQ="udev dmraid"); depending on how dependencies are resolved, it may make sense to be conditional as follows:
   {{{
PREREQ="udev"
if [ -e /sbin/dmraid ] ; then
        PREREQ="$PREREQ dmraid"
fi
}}}
 - /usr/share/initramfs-tools/hooks/lvm2 should copy /etc/lvm/lvm.conf into the target:
   {{{
if [ -f /etc/lvm/lvm.conf ] ; then
        mkdir -p ${DESTDIR}/etc/lvm/
        cp /etc/lvm/lvm.conf ${DESTDIR}/etc/lvm/
fi
}}}

Changes to the dmraid package:
 - /usr/share/initramfs-tools/scripts/local-top/dmraid should be moved to /usr/share/initramfs-tools/scripts/init-premount/dmraid

[xavier (xavier-ml)]

ubuntu hardy amd64 + dell vostro 200 (ICHR9) work with dmraid and lvm :

1) do the change form Charles Duffy above
2) change /etc/udev/rules.d/85-lvm2.rules as

SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_FS_TYPE}=="lvm*|LVM*", \
 RUN+="watershed sh -c '[ -x /sbin/dmraid ] && /sbin/dmraid -ay; /sbin/lvm vgscan; /sbin/lvm vgchange -a y'"

yoopi ;)

[Luke Yelavich (themuso)]

Have you tested these changes? Please note that the dmraid initramfs script is only present as a catch all, in case udev doesn't correctly identify a dmraid array, which is the case for a few dmraid compatible configurations.

[Charles Duffy (M1) (cduffy-messageone)]

@Luke - I have been running with these patches (manually applied) since I first posted them in the comment left on February 21st, and manually reapplied them when upgrading to Ibex. Behavior otherwise is to mount a randomly selected member of the mirror rather than the mirror itself.

[Phillip Susi (psusi)]

The dmraid change isn't really appropriate since it only needs to be run for local boots, which is why it was in local-top. Also the script really should go away entirely since udev runs dmraid. It seems having it in both places creates a race condition to see which one activates the array first.

We already have udev rules to choose which device to point the by-uuid link to, it seems to me that lvm and dmraid both should be modified to be told by udev which device to scan, rather than enumerating over every /dev/xxxx* it finds. That way when udev decides the device belongs to dmraid, it won't ask lvm to scan it to see if it contains a pv, and therefore, lvm won't try to use it.

[Bryce Harrington (bryce)]

I'm unsubscribing the Sponsors' Team for now. Please re-subscribe when ready.

[Bryce Harrington (bryce)]

[Unsubbing ubuntu-main-sponsors for now; please resubscribe when the aforementioned issues have been resolved and a new debdiff is ready for review/upload.]

[Giuseppe Iuculano (giuseppe-iuculano)]

I think that this issue is fixed with:

* debian/dmraid-activate: Use the newly introduced -Z flag for dmraid, to
    instruct the kernel to remove partition device nodes from array member
    disks. This means that there is no chance of race conditions with udev
    and creating UUID/label symbolic links either for member disk nodes,
    or dmraid device-mapper nodes.

Charles, can you confirm this?

[Charles Duffy (M1) (cduffy-messageone)]

The patch referenced in comment 22 appears to operate as advertised, removing /dev/sd[a-z][0-9] entries corresponding with the RAID members.

I am, however, somewhat concerned -- there are still numerous references to the underlying partitions based on the identity of the underlying hardware, such as /dev/mapper/1ATA_ST3320620AS_9QF8SL{J8,RF}-part{1,2,3,4,5}, and I don't see anything in the default lvm.conf which would remove such entries from consideration.

[Phillip Susi (psusi)]

The issue seems to have been sorted out as of Karmic because lvm on top of dmraid works fine. I'm not sure exactly how, but I think udev now only instructs lvm to scan devices that blkid has identified as lvm physical volumes, and blkid gives priority to dmraid signatures first.