creation of snapshots fails unpredictably

[Reinhard Tartler]

Often, lvm snapshots are created sucessfully:

root@hades:~ # lvcreate -s -n sid_snap /dev/hades_stripe/chroot_sid -L2g -v
    Setting chunksize to 16 sectors.
    Finding volume group "hades_stripe"
    Archiving volume group "hades_stripe" metadata (seqno 200).
    Creating logical volume sid_snap
    Creating volume group backup "/etc/lvm/backup/hades_stripe" (seqno 201).
    Found volume group "hades_stripe"
    Loading hades_stripe-sid_snap
    Zeroing start of logical volume "sid_snap"
    Found volume group "hades_stripe"
    Removing hades_stripe-sid_snap
    Found volume group "hades_stripe"
    Found volume group "hades_stripe"
    Found volume group "hades_stripe"
    Loading hades_stripe-chroot_sid-real
    Loading hades_stripe-sid_snap-cow
    Loading hades_stripe-sid_snap
    Loading hades_stripe-chroot_sid
    Creating volume group backup "/etc/lvm/backup/hades_stripe" (seqno 202).
  Logical volume "sid_snap" created

Sometimes, however (approx 30% of the time), it fails like this:
root@hades:~ # lvcreate -s -n sid_snap /dev/hades_stripe/chroot_sid -L2g -v
    Setting chunksize to 16 sectors.
    Finding volume group "hades_stripe"
    Archiving volume group "hades_stripe" metadata (seqno 203).
    Creating logical volume sid_snap
    Creating volume group backup "/etc/lvm/backup/hades_stripe" (seqno 204).
    Found volume group "hades_stripe"
    Loading hades_stripe-sid_snap
    Zeroing start of logical volume "sid_snap"
    Found volume group "hades_stripe"
  LV hades_stripe/sid_snap in use: not removing
  Couldn't deactivate new snapshot.

It is there, and I need to remove it before recreating it for use:
root@hades:~ # lvremove /dev/hades_stripe/sid_snap -f
  Logical volume "sid_snap" successfully removed

then I retry it and often, it succeeds.

This behaviour completly confuses schroot, which provides chroots on lvm snapshots.

This happens on my dapper/amd64 machine. I wonder why it doesn't on my x86 laptop. Perhaps this is arch dependant?

[Marco d'Itri (md)]

On Dec 17, Andreas Sundstrom <email address hidden> wrote:

> matrix:/home/sunkan# /sbin/lvm lvcreate -s -L 1G -n var_s /dev/3ware/var
> LV 3ware/var_s in use: not removing
Find why it's in use then.

--
ciao,
Marco

[Andreas Sundstrom (sunkan)]

Marco d'Itri wrote:
> On Dec 17, Andreas Sundstrom <email address hidden> wrote:
>
>
>>matrix:/home/sunkan# /sbin/lvm lvcreate -s -L 1G -n var_s /dev/3ware/var
>> LV 3ware/var_s in use: not removing
>
> Find why it's in use then.
>

It's not that simple unfortunately.. It's not created when I run that
command. Everything is 100% reproducible for me here.. I've tried it
numerous times before submitting the bug.

Any ideas on how to diagnose the problem?

[Andreas Sundstrom (sunkan)]

Marco d'Itri wrote:
> On Dec 17, Andreas Sundstrom <email address hidden> wrote:
>
>
>>matrix:/home/sunkan# /sbin/lvm lvcreate -s -L 1G -n var_s /dev/3ware/var
>> LV 3ware/var_s in use: not removing
>
> Find why it's in use then.
>

After a quick test it seems that reverting this change back to how it
was in 0.071 fixes it.

matrix:/# diff -u /etc/udev/udev.rules~ /etc/udev/udev.rules
--- /etc/udev/udev.rules~ 2005-12-17 02:13:53.000000000 +0100
+++ /etc/udev/udev.rules 2005-12-17 11:08:03.000000000 +0100
@@ -93,6 +93,6 @@
 SUBSYSTEM=="aoe", KERNEL=="interfaces", NAME="etherd/%k"

 # device mapper creates its own device nodes, so ignore these
-KERNEL=="dm-[0-9]*", NAME=""
+KERNEL=="dm-[0-9]*", OPTIONS+="ignore_device"
 KERNEL=="device-mapper", NAME="mapper/control"

Do you know the background to why this has changed?
I can't notice any other change in behaviour except that snapshotting
fails for some reason I don't quite understand.

[Filippo Giunchedi (filippo)]

On Mon, Dec 19, 2005 at 07:31:58PM +0000, Alasdair G Kergon wrote:
> On Mon, Dec 19, 2005 at 04:33:39PM +0100, Filippo Giunchedi wrote:
> > Package: lvm2
> > Version: 2.02.01-1
> > Severity: normal
>
> > LV chroots/test1 in use: not deactivating
> > Couldn't deactivate new snapshot.
>
> > but sometimes I can successfully create snapshots..
>
> What version of the udev package?
> Try downgrading it.
> There's been another report suggesting that package may now be
> accessing devices owned by lvm2 that it should not be interfering
> with.

apparently http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=343671#msg20 has the
solution, works here. I'm not sure either why ignore_device has been removed

filippo
--
Filippo Giunchedi
PGP key: 0x6B79D401
random quote follows:

Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it.
-- Brian W. Kernighan

[Marco d'Itri (md)]

On Dec 19, Filippo Giunchedi <email address hidden> wrote:

> apparently http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=343671#msg20 has the
> solution, works here. I'm not sure either why ignore_device has been removed
Because another Debian maintainer requested it with a good rationale,
so I am not going to revert this change until somebody will explain how
NAME="" breaks lvm and why this would be an udev bug.

--
ciao,
Marco

[Alasdair G. Kergon (agk2)]

On Mon, Dec 19, 2005 at 10:15:33PM +0100, Marco d'Itri wrote:
> On Dec 19, Filippo Giunchedi <email address hidden> wrote:
> > apparently http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=343671#msg20 has the
> > solution, works here. I'm not sure either why ignore_device has been removed
> Because another Debian maintainer requested it with a good rationale,
> so I am not going to revert this change until somebody will explain how
> NAME="" breaks lvm and why this would be an udev bug.

Maybe someone can strace udev or look at the source and see what the
difference between those two settings is?

If people want to use udev at the same time as lvm2/device-mapper they have
to configure it to ignore device-mapper devices completely. Its
asynchronous design just doesn't fit alongside code that is creating,
using and destroying short-lived devices.

Alasdair
--
<email address hidden>

[Bastian Blank (waldi)]

On Mon, Dec 19, 2005 at 10:16:37PM +0000, Alasdair G Kergon wrote:
> If people want to use udev at the same time as lvm2/device-mapper they have
> to configure it to ignore device-mapper devices completely.

Why? The kernel don't wait for the completion of events and the devices
are exported as dm-*.

> Its
> asynchronous design just doesn't fit alongside code that is creating,
> using and destroying short-lived devices.

EPARSE

Bastian

--
Is truth not truth for all?
  -- Natira, "For the World is Hollow and I have Touched
     the Sky", stardate 5476.4.

[Alasdair G. Kergon (agk2)]

On Tue, Dec 20, 2005 at 11:27:22AM +0100, Bastian Blank wrote:
> Why? The kernel don't wait for the completion of events and the devices
> are exported as dm-*.

Exactly - it's an asynchronous design prone to races - udev does not
offer an option for the kernel to wait for the completion. So by the
time the relatively-slow udev gets around to processing the event the
device could already have been deleted or have changed name or
anything! Device-mapper facilitates rapid changes like that by
design. The asynchronous udev design does not cope satisfactorily
with temporary devices which might only exist for a fraction of a
second - so LVM2 has to avoid it.

But enough speculation - please can someone provides a trace or source
code showing what udev is or isn't doing that might be interfering
with LVM2? For example, if udev ever opens the devices that could
race against LVM2 deleting them.

Alasdair
--
<email address hidden>

[Marco d'Itri (md)]

On Dec 20, Alasdair G Kergon <email address hidden> wrote:

> But enough speculation - please can someone provides a trace or source
> code showing what udev is or isn't doing that might be interfering
> with LVM2? For example, if udev ever opens the devices that could
> race against LVM2 deleting them.
This is the configuration which allegedly does not work, it means that
udev will not create or delete any device name matching dm-[0-9]*.

KERNEL=="dm-[0-9]*", NAME=""

I wonder if *some other package* is shipping rules which affect these
devices.

--
ciao,
Marco

[Marco d'Itri (md)]

reassign 343671 lvm2
merge 343671 344040
thanks

On Dec 20, Marco d'Itri <email address hidden> wrote:

> This is the configuration which allegedly does not work, it means that
> udev will not create or delete any device name matching dm-[0-9]*.
>
> KERNEL=="dm-[0-9]*", NAME=""
>
> I wonder if *some other package* is shipping rules which affect these
> devices.
I am reassigning this bug to lvm2, if somebody can find an actual
problem in my package then feel free to reassign it back.

--
ciao,
Marco

[Andreas Sundstrom (sunkan)]

Marco d'Itri wrote:
> reassign 343671 lvm2
> merge 343671 344040
> thanks
>
> On Dec 20, Marco d'Itri <email address hidden> wrote:
>
>
>>This is the configuration which allegedly does not work, it means that
>>udev will not create or delete any device name matching dm-[0-9]*.
>>
>>KERNEL=="dm-[0-9]*", NAME=""
>>
>>I wonder if *some other package* is shipping rules which affect these
>>devices.
>
> I am reassigning this bug to lvm2, if somebody can find an actual
> problem in my package then feel free to reassign it back.
>

Ok, I have done some stracing on the udevd process and the short result
is this:

-- udev-076-6-only-dm-org.log 2005-12-27 10:26:00.000000000 +0100
+++ udev-076-6-only-dm-ignore-device.log 2005-12-27 10:26:13.000000000 +0100
@@ -2,4 +2,24 @@
 unlink("/dev/.udev/failed/block@dm-7") = -1 ENOENT (No such file or directory)
 symlink("/sys/block/dm-7", "/dev/.udev/queue/block@dm-7") = 0
 unlink("/dev/.udev/failed/block@dm-7") = -1 ENOENT (No such file or directory)
+rename("/dev/.udev/queue/block@dm-7", "/dev/.udev/failed/block@dm-7") = 0
+recv(4, "remove@/block/dm-7\0ACTION=remove"..., 1536, 0) = 98
+unlink("/dev/.udev/failed/block@dm-7") = 0
+symlink("/sys/block/dm-7", "/dev/.udev/queue/block@dm-7") = 0
+unlink("/dev/.udev/failed/block@dm-7") = -1 ENOENT (No such file or directory)
 unlink("/dev/.udev/queue/block@dm-7") = 0
+recv(4, "add@/block/dm-7\0ACTION=add\0DEVPA"..., 1536, 0) = 92
+unlink("/dev/.udev/failed/block@dm-7") = -1 ENOENT (No such file or directory)
+symlink("/sys/block/dm-7", "/dev/.udev/queue/block@dm-7") = 0
+recv(4, "add@/block/dm-8\0ACTION=add\0DEVPA"..., 1536, 0) = 92
+unlink("/dev/.udev/failed/block@dm-8") = -1 ENOENT (No such file or directory)
+symlink("/sys/block/dm-8", "/dev/.udev/queue/block@dm-8") = 0
+recv(4, "add@/block/dm-9\0ACTION=add\0DEVPA"..., 1536, 0) = 92
+unlink("/dev/.udev/failed/block@dm-9") = -1 ENOENT (No such file or directory)
+symlink("/sys/block/dm-9", "/dev/.udev/queue/block@dm-9") = 0
+unlink("/dev/.udev/failed/block@dm-7") = -1 ENOENT (No such file or directory)
+rename("/dev/.udev/queue/block@dm-7", "/dev/.udev/failed/block@dm-7") = 0
+unlink("/dev/.udev/failed/block@dm-8") = -1 ENOENT (No such file or directory)
+rename("/dev/.udev/queue/block@dm-8", "/dev/.udev/failed/block@dm-8") = 0
+unlink("/dev/.udev/failed/block@dm-9") = -1 ENOENT (No such file or directory)
+rename("/dev/.udev/queue/block@dm-9", "/dev/.udev/failed/block@dm-9") = 0
sunkan@ans-addon:/tmp$

It seems to me that when NAME="" is used udev unlinks the dm-7 device
from /dev/.udev/queue/ but when ignore_device is used it is first
moved to /dev/.udev/failed/ and then unlinked when udev recieves an
remove action.

I don't understand this enough to tell if it's udev or lvm that
actually needs fixing. Maybe someonelse understands it?

Let me know if I can gather more information for you. I have
attached the "raw" strace logs too, maybe there's more info there that
I have missed.

/Andreas

[Reinhard Tartler (siretart)]

Often, lvm snapshots are created sucessfully:

root@hades:~ # lvcreate -s -n sid_snap /dev/hades_stripe/chroot_sid -L2g -v
    Setting chunksize to 16 sectors.
    Finding volume group "hades_stripe"
    Archiving volume group "hades_stripe" metadata (seqno 200).
    Creating logical volume sid_snap
    Creating volume group backup "/etc/lvm/backup/hades_stripe" (seqno 201).
    Found volume group "hades_stripe"
    Loading hades_stripe-sid_snap
    Zeroing start of logical volume "sid_snap"
    Found volume group "hades_stripe"
    Removing hades_stripe-sid_snap
    Found volume group "hades_stripe"
    Found volume group "hades_stripe"
    Found volume group "hades_stripe"
    Loading hades_stripe-chroot_sid-real
    Loading hades_stripe-sid_snap-cow
    Loading hades_stripe-sid_snap
    Loading hades_stripe-chroot_sid
    Creating volume group backup "/etc/lvm/backup/hades_stripe" (seqno 202).
  Logical volume "sid_snap" created

Sometimes, however (approx 30% of the time), it fails like this:
root@hades:~ # lvcreate -s -n sid_snap /dev/hades_stripe/chroot_sid -L2g -v
    Setting chunksize to 16 sectors.
    Finding volume group "hades_stripe"
    Archiving volume group "hades_stripe" metadata (seqno 203).
    Creating logical volume sid_snap
    Creating volume group backup "/etc/lvm/backup/hades_stripe" (seqno 204).
    Found volume group "hades_stripe"
    Loading hades_stripe-sid_snap
    Zeroing start of logical volume "sid_snap"
    Found volume group "hades_stripe"
  LV hades_stripe/sid_snap in use: not removing
  Couldn't deactivate new snapshot.

It is there, and I need to remove it before recreating it for use:
root@hades:~ # lvremove /dev/hades_stripe/sid_snap -f
  Logical volume "sid_snap" successfully removed

then I retry it and often, it succeeds.

This behaviour completly confuses schroot, which provides chroots on lvm snapshots.

This happens on my dapper/amd64 machine. I wonder why it doesn't on my x86 laptop. Perhaps this is arch dependant?

[Reinhard Tartler (siretart)]

subscribing our lvm guru, fabbione

[Reinhard Tartler (siretart)]

I added a watch on debian #343671, because it sounds quite similar. I tried the patch mentioned there, but it seems that debian uses a completly different way of configuring udev. *sigh*

[Reinhard Tartler (siretart)]

in rules.d/20-names.rules, I changed the following line:

KERNEL=="dm-[0-9]*", NAME="dm/%n"

to the following:
KERNEL=="dm-[0-9]*", NAME="dm/%n",OPTIONS+="ignore_device"

and voila, the problem has vanished.so this really seems to be debian #343671

[Scott James Remnant (Canonical) (canonical-scott)]

Try just

KERNEL=="dm-[0-9]*", OPTIONS+="ignore_device"

No reason to NAME a device you're about to ignore.

[Reinhard Tartler (siretart)]

I tried this rules, and it worked for me [tm].

I even updated my initramfs with 'update-initramfs -uv', and my system still boots. :)

[bigredradio (dhuffman)]

This works for me, however, I would like to better understand why. This problem is critical for me and I need to know that if I recommend to users to update this, it will not break something else. I am researching it now, but any information as to how you came up with this solution would be helpful. Thanks in advance.

[Reinhard Tartler (siretart)]

Maybe we should set the milestone to 6.06 as well?

[Fabio Massimo Di Nitto (fabbione)]

hmm this is a very interesting one. Anyway I can reproduce it reliably on i386 too and the change to the udev rule works around it.

I am going to do one more test before asking for inclusion.

Fabio

[Fabio Massimo Di Nitto (fabbione)]

Scott, I did retest the workaround, including a / on LVM and it does its job.

I am fully aware that it is not the right solution but in such short time from release i don't think we can have a proper fix.

Can you please upload a udev with the OPTIONS+="ignore_device" please?

Thanks
Fabio

[Fabio Massimo Di Nitto (fabbione)]

Speaking with one of the upstream guys:

<fabbione> pjc: do you mind to give me an opinion on https://launchpad.net/distros/ubuntu/+source/lvm2/+bug/38409/ ?
<pjc> what's happening here is that LVM creates the snapshot LV,
<pjc> temporarily activates it to zero the start (so the kernel knows its a new snapshot)
<pjc> then deactivates it (which fails because udev is using it)
<pjc> then wants to reactivates the whole LV/snapshot stack.
<pjc> ...the last bit desn't happen because the deactivate failed.
<fabbione> ooook
<fabbione> so my next natural question is why udev and lvm need to do the same device handling, overlapping and racing?
<pjc> well, they're doing different things,
* fabbione scratches his head...
<fabbione> i can understand what LVM is doing and why
<pjc> hmm, maybe they're not.
<fabbione> 2 things i can't really explains:
<pjc> I suppose LVM maybe ought to wait for udev to create the device, ... maybe
<fabbione> 1) why lvm takes care of creating devices when there is udev
<fabbione> 2) if lvm really needs to manage it's own temporary devices, why can't it do outside the /dev tree?
<pjc> I suppose 1) is historical, there wasn't always udev
<pjc> 2) is a good point.
<fabbione> 1) i agree
<fabbione> 2) i know :P
<fabbione> i wonder how much it would break to do 2)
<pjc> the thing about 2) is, how to do stop udev interfering? If devmapper creates a block device, how to we stop udev inerfering
<pjc> it might be easier to just flag it somehow.
<pjc> as a temp device
<fabbione> if we move temporary devices to /whatevertmpdir, the final device will be created by udev on kernel/devmapper input
<fabbione> so there should be cooperation and not interference at that point
<pjc> agk won't like that I guess, it means breaking the LVM/libdevmapper API /again/
<fabbione> i don't the API but if internal temporary devices are *really* internal and temporary and not exposed to the client, then i don't see how that would break the API
<fabbione> i don't *know the API ^^
<fabbione> just a wild guess
<fabbione> you know the code.. i don't
<pjc> LVM tells libdevmapper to create the device, then LVM uses it. then it tells libdevmapper to deactivate the device
<pjc> so (and I'm guessing a little here) there would need to be a flag saying it's temporary
<fabbione> does lvm also tells libdevmapper the path to the device?
<pjc> I'm not /that/ familiar with the lvm/devmapper interface, it's a bit of a black art.
<pjc> I'm not sure.
<fabbione> yeah i can imagine
<fabbione> oh sorry i forgot to ask you.. do you mind if i copy/paste to the bug for reference?
<fabbione> otherwise i will write a short summary
<pjc> go ahead. agk is really the guy to tackle about this. but he's terrifyingly busy
<fabbione> i am not surprised about that either
<fabbione> i think i can work on it enough to look at the API at least
<fabbione> perhaps it's much easier than it looks like
<pjc> it isn't
<fabbione> it's still worth a shot :)
<fabbione> thanks a lot dude

[Scott James Remnant (Canonical) (canonical-scott)]

New udev uploaded.

[Fabio Massimo Di Nitto (fabbione)]

Scott thanks for uploading the workaround. I want to keep this bug open for reference until we have a real fix.

Fabio

[Scott James Remnant (Canonical) (canonical-scott)]

I understood that was the real fix?

Is there a different fix in the works?

[Fabio Massimo Di Nitto (fabbione)]

I am talking with LVM upstream (as in the previous discussion pasted in the bug) to see if this is the fix or there is a better fix that should be done in LVM.

The point is that lvm is creating temporary devices in /dev/mapper in order to trigger some kernel stuff. These temporary devices are clashing with udev managing /dev.

Now this is yet another "lvm is at fault, no udev is at fault, no lvm..... etc" kind of bug. As I discussed with Patrick (pjc) it would make sense to slam temporary devices out of /dev and allow udev to have full control over /dev as it is supposed. On the other side changing lvm do that is not trivial and upstream might not want to do it.

The discussion is going on. From my perspective what we have now is neither a workaround or the bug fix. It makes things working.

Fabio

[Scott James Remnant (Canonical) (canonical-scott)]

Ah, excellent. Sounds very fair to me, I agree that udev should own /dev and those temporary devices should be created elsewhere -- or at least in a nominated sub-directory of /dev that udev doesn't manage.

[Reinhard Tartler (siretart)]

anyway, what we are currently doing now fixes the symptom, and I havn't seen sideeffects from this yet. I'm glad that this fix is now in, and I agree that we should watch upstream investigating this issue, and focus on other bugs right now.

Thanks for your time working on this, guys!

[Matt Zimmerman (mdz)]

This change alleviates the problem; reducing severity to normal.

udev (079-0ubuntu28) dapper; urgency=low

  * Ignore device-mapper devices and let LVM take care of creating them.

 -- Scott James Remnant <email address hidden> Fri, 28 Apr 2006 15:42:37 +0100

[Reinhard Tartler (siretart)]

the change which alleviated the problem was reverted, which causes this bug to reappear. for feisty we look for the actual bug, so raising severity again

[Reinhard Tartler (siretart)]

fabionne asked me to not set this to critical, since the proper fix was about to land.

[Reinhard Tartler (siretart)]

Fabio: Any updates on this?

[Reinhard Tartler (siretart)]

As of today, I'm no longer able to create snapshots at all:

root@hades:~ # lvcreate -s -n dapper32-snap -L 2G /dev/hades_stripe/chroot_dapper32
  LV hades_stripe/dapper32-snap in use: not deactivating
  Couldn't deactivate new snapshot.

I'm not sure if this is the same bug, or some other race in the udev scripts.

[Kees Cook (kees)]

I opened bug 84672 for the "in use: not deactivating" problem. It seemed like a separate issue from this bug.

[Reinhard Tartler (siretart)]

targetting for beta. fabbione told me ages ago that a fix was in a pipe, the workaround for edgy has been removed and the impact is quite severe.

[Tollef Fog Heen (tfheen)]

Moving milestone forward

[Scott James Remnant (Canonical) (canonical-scott)]

Packages for testing are available here:

   http://people.ubuntu.com/~scott/packages

Please build all of the sources, and install the binaries from them. This should hopefully correct the problem.

[Scott James Remnant (Canonical) (canonical-scott)]

We believe that this problem has been corrected by a series of uploads today. Please update to ensure you have the following package versions:

    dmsetup, libdevmapper1.02 - 1.02.08-1ubuntu6
    lvm-common - 1.5.20ubuntu12
    lvm2 - 2.02.06-2ubuntu9
    mdadm - 2.5.6-7ubuntu5 (not applicable unless you're also using mdadm)
    udev, volumeid, libvolume-id0 - 108-0ubuntu1

The problem was caused by a number of ordering issues and race conditions relating to when lvm and mdadm were called, and how those interacted to ensure the devices were created and their contents examined.

In particular, due to a typo, both udev and devicemapper were attempting to create the /dev/mapper device node. This has been corrected so that only udev does this, and devicemapper waits, as was originally intended.

Note that this event-based sequence is substantially different from Debian, so any bugs filed there will not be relevant to helping solve problems in Ubuntu.

This should now work correctly. If it does not, I would ask that you do not re-open this bug, and instead file a new bug on lvm2 for your exact problem, even if someone else has already filed one, with verbose details about your setup and how you cause the error.