[SRU] lvremove often fails on first attempt when removing snapshot
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| lvm2 (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
| Raring |
Won't Fix
|
High
|
Dimitri John Ledkov | ||
| Saucy |
Fix Released
|
High
|
Unassigned | ||
Bug Description
[SRU Justification]
[Impact]
* Removing LVM snapshots is unreliable on Raring on some hardware.
* Tools that abstract LVM actions will likely fail if they expect these actions
to function correctly and reliably. Specifically, OpenStack Cinder requires
reliability here.
[Test Case]
The following test case seems to trigger the bug on two classes of hardware I've used, but not on a virtual machine:
sudo apt-get install lvm2
# A free block device
BDEV="sdb"
# Create original LVM volume.
pvcreate /dev/$BDEV
vgcreate testing-vg /dev/$BDEV
lvcreate -L 1024M -n testing test
# Run a snapshot create/delete cycle and observe frequent failures.
lvcreate -L 1024M --name testing-snapshot --snapshot /dev/testing-
lvremove -f /dev/testing-
Example failure:
$ lvcreate -L 1024M --name testing-snapshot --snapshot /dev/testing-
$ lvremove -f /dev/testing-
Logical volume "testing-snapshot" created
Unable to deactivate open testing-
Failed to resume testing.
[Solution]
Recent updates in Debian+Saucy to the LVM and dm udev rules seem to alleviate the issue and I no longer see it when applying those changes from lvm2 2.02.98-6ubuntu1 (Patch attached)
-------
# lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu Saucy Salamander (development branch)
Release: 13.10
Codename: saucy
# uname -r
3.11.0-4-generic
# dpkg -l | grep lvm2
ii lvm2 2.02.98-1ubuntu5 amd64 Linux Logical Volume Manager
Experiencing this on saucy across a number of boxes, all of similar hardware. Unable to reproduce in a virtual machine using saucy cloud images.
# BDEV="sdb"
# apt-get install -y lvm2
# umount /mnt
# pvcreate /dev/$BDEV
# vgcreate testing-vg /dev/$BDEV
# lvcreate -L 1024M -n testing testing-vg
# lvcreate -L 1024M --name testing-snapshot --snapshot /dev/testing-
# lvremove -f /dev/testing-
Unable to deactivate open testing-
Failed to resume testing.
# lvremove -f /dev/testing-
Logical volume "testing-snapshot" successfully removed
Comparing output of 'dmsetup info' before and after failed attempts:
- The failed attempt removes the -cow device (testing-
- Pre failure, the Open Count of testing-
- After successful lvremove of the snapshot, the -real device persists. On unaffected systems, the post-snapshot removal dm table is the same as it was pre-creation of snapshot. In this case, a single device (testing-
# dmsetup info
Name: testing-
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 2
Event number: 0
Major, minor: 252, 2
Number of targets: 1
UUID: LVM-cYTNHi8q1D1
Name: testing-
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 252, 1
Number of targets: 1
UUID: LVM-cYTNHi8q1D1
Name: testing-
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 252, 3
Number of targets: 1
UUID: LVM-cYTNHi8q1D1
Name: testing--vg-testing
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 252, 0
Number of targets: 1
UUID: LVM-cYTNHi8q1D1
# lvremove -f /dev/testing-
Unable to deactivate open testing-
Failed to resume testing.
# dmsetup info
Name: testing-
State: ACTIVE
Read Ahead: 0
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 252, 2
Number of targets: 1
UUID: LVM-cYTNHi8q1D1
Name: testing-
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 252, 1
Number of targets: 1
UUID: LVM-cYTNHi8q1D1
Name: testing--vg-testing
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 252, 0
Number of targets: 1
UUID: LVM-cYTNHi8q1D1
# lvremove -f /dev/testing-
Logical volume "testing-snapshot" successfully removed
# dmsetup info
Name: testing-
State: ACTIVE
Read Ahead: 0
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 252, 2
Number of targets: 1
UUID: LVM-cYTNHi8q1D1
Name: testing--vg-testing
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 252, 0
Number of targets: 1
UUID: LVM-cYTNHi8q1D1
Output of 'udevadm monitor' with comments for contexts (also attached):
# Original LV creation
KERNEL[3890.419526] add /devices/
KERNEL[3890.419572] add /devices/
KERNEL[3890.419903] change /devices/
UDEV [3890.420177] add /devices/
UDEV [3890.420594] add /devices/
UDEV [3890.439210] change /devices/
KERNEL[3890.440430] change /devices/
UDEV [3890.458613] change /devices/
KERNEL[3890.493110] change /devices/
UDEV [3890.553035] change /devices/
# Snapshot creation
KERNEL[3915.481512] add /devices/
KERNEL[3915.481542] add /devices/
KERNEL[3915.481814] change /devices/
UDEV [3915.482069] add /devices/
UDEV [3915.482468] add /devices/
UDEV [3915.503721] change /devices/
KERNEL[3915.504157] change /devices/
KERNEL[3915.548206] add /devices/
KERNEL[3915.548288] add /devices/
UDEV [3915.548503] add /devices/
KERNEL[3915.548787] add /devices/
KERNEL[3915.548882] add /devices/
UDEV [3915.549093] add /devices/
KERNEL[3915.549173] change /devices/
UDEV [3915.549240] add /devices/
UDEV [3915.549710] add /devices/
UDEV [3915.553716] change /devices/
KERNEL[3915.563685] change /devices/
KERNEL[3915.563823] change /devices/
KERNEL[3915.563915] change /devices/
UDEV [3915.567734] change /devices/
UDEV [3915.586136] change /devices/
UDEV [3915.603960] change /devices/
UDEV [3915.611925] change /devices/
KERNEL[3915.625968] change /devices/
UDEV [3915.672978] change /devices/
# Failed snapshot delete
KERNEL[3937.161806] change /devices/
KERNEL[3937.161839] change /devices/
KERNEL[3937.161929] change /devices/
KERNEL[3937.162154] remove /devices/
KERNEL[3937.162305] remove /devices/
KERNEL[3937.162375] remove /devices/
UDEV [3937.162842] remove /devices/
UDEV [3937.169283] change /devices/
UDEV [3937.172103] remove /devices/
UDEV [3937.172217] remove /devices/
KERNEL[3937.176945] change /devices/
KERNEL[3937.177255] change /devices/
UDEV [3937.220478] change /devices/
UDEV [3937.221039] change /devices/
UDEV [3937.222593] change /devices/
UDEV [3937.297968] change /devices/
# Successful snapshot delete
KERNEL[3953.366547] remove /devices/
KERNEL[3953.366713] remove /devices/
KERNEL[3953.366805] remove /devices/
UDEV [3953.367431] remove /devices/
UDEV [3953.369726] remove /devices/
UDEV [3953.369849] remove /devices/
KERNEL[3953.439220] change /devices/
UDEV [3953.510027] change /devices/
| Adam Gandelman (gandelman-a) wrote | #1 |
| Adam Gandelman (gandelman-a) wrote | #2 |
| Brian Murray (brian-murray) wrote | #3 |
| Adam Gandelman (gandelman-a) wrote | #4 |
| Adam Gandelman (gandelman-a) wrote | #5 |
| Changed in lvm2 (Ubuntu): | |
| importance: | Undecided → High |
| Adam Gandelman (gandelman-a) wrote | #6 |
| Adam Gandelman (gandelman-a) wrote | #7 |
| Changed in lvm2 (Ubuntu Saucy): | |
| status: | New → Fix Released |
| Changed in lvm2 (Ubuntu Raring): | |
| importance: | Undecided → High |
| summary: |
- lvremove always fails on first attempt when removing snapshot + [SRU] lvremove often fails on first attempt when removing snapshot |
| Dimitri John Ledkov (xnox) wrote | #9 |
| Changed in lvm2 (Ubuntu Raring): | |
| assignee: | nobody → Dmitrijs Ledkovs (xnox) |
| status: | New → Confirmed |
| Adam Gandelman (gandelman-a) wrote | #10 |
| Sebastian Unger (sebunger44) wrote | #11 |
| Dimitri John Ledkov (xnox) wrote Re: [Bug 1223576] Re: [SRU] lvremove often fails on first attempt when removing snapshot | #12 |
| Rolf Leggewie (r0lf) wrote | #13 |
| Changed in lvm2 (Ubuntu Raring): | |
| status: | Confirmed → Won't Fix |
Using the same hardware this seems to not affect 12.04, but on 13.04 the failure occurs %50 of the time. Every other create + delete cycle completes without error.