| 2014-01-17 15:41:13 |
Dirk Mueller |
bug |
|
|
added bug |
| 2014-01-17 15:47:42 |
OpenStack Infra |
cinder: status |
New |
In Progress |
|
| 2014-01-17 15:47:42 |
OpenStack Infra |
cinder: assignee |
|
Dirk Mueller (dmllr) |
|
| 2014-01-17 15:50:08 |
Dirk Mueller |
description |
On my system, I can get cinder-volume to hang on issuing lvm related commands after deleting snapshots (that are non-thin provisioned LVM snapshots with clear_volume set to zero).
the issue is that lvm locks up due to trying to access suspended device mapper entries, and at some point cinder-volume does a lvm related command and hangs on that.
I can use this script to trigger the issue:
=== cut hang.sh ===
enable_fix=1
#enable_fix=0
vg=cinder-volumes
v=testvol.$$
lvcreate --name $v $vg -L 1g
sleep 2
lvcreate --name snap-$v --snapshot $vg/$v -L 1g
vgp=/dev/mapper/${vg/-/--}-snap--${v/-/--}
sleep 2
( sleep 10 < $vgp-cow ) &
test "$enable_fix" -eq "1" && lvchange -y -an $vg/snap-$v
lvremove -f $vg/snap-$v
sleep 1
lvremove -f $vg/$v
=== cut hang.sh ===
vg needs to be set to a lvm VG that exists and can take a few gig of space. whenever enable_fix is set to 0, lvremove -f ends with :
Unable to deactivate open cinder--volumes-snap--testvol.27700-cow (252:5)
Failed to resume snap-testvol.27700.
libdevmapper exiting with 1 device(s) still suspended.
this is because the sleep command before keeps a fd open on the -cow. apparently in real-life this is either udev or the dd command still having the fd open for some reason I have not yet understood.
The deactivation before removing seems to help. |
On my system, I can get cinder-volume to hang on issuing lvm related commands after deleting snapshots (that are non-thin provisioned LVM snapshots with clear_volume set to zero).
the issue is that lvm locks up due to trying to access suspended device mapper entries, and at some point cinder-volume does a lvm related command and hangs on that.
I can use this script to trigger the issue:
=== cut hang.sh ===
enable_fix=1
#enable_fix=0
vg=cinder-volumes
v=testvol.$$
lvcreate --name $v $vg -L 1g
sleep 2
lvcreate --name snap-$v --snapshot $vg/$v -L 1g
vgp=/dev/mapper/${vg/-/--}-snap--${v/-/--}
sleep 2
( sleep 10 < $vgp-cow ) &
test "$enable_fix" -eq "1" && lvchange -y -an $vg/snap-$v
lvremove -f $vg/snap-$v
sleep 1
lvremove -f $vg/$v
=== cut hang.sh ===
vg needs to be set to a lvm VG that exists and can take a few gig of space. whenever enable_fix is set to 0, lvremove -f ends with :
Unable to deactivate open cinder--volumes-snap--testvol.27700-cow (252:5)
Failed to resume snap-testvol.27700.
libdevmapper exiting with 1 device(s) still suspended.
this is because the sleep command before keeps a fd open on the -cow. The script then also never finishes and any other lvm command hangs as well.
apparently in real-life this is either udev or the dd command still having the fd open for some reason I have not yet understood.
The deactivation before removing seems to help. |
|
| 2014-01-17 15:54:22 |
Ralf Haferkamp |
bug |
|
|
added subscriber Ralf Haferkamp |
| 2014-01-17 15:56:02 |
Dirk Mueller |
description |
On my system, I can get cinder-volume to hang on issuing lvm related commands after deleting snapshots (that are non-thin provisioned LVM snapshots with clear_volume set to zero).
the issue is that lvm locks up due to trying to access suspended device mapper entries, and at some point cinder-volume does a lvm related command and hangs on that.
I can use this script to trigger the issue:
=== cut hang.sh ===
enable_fix=1
#enable_fix=0
vg=cinder-volumes
v=testvol.$$
lvcreate --name $v $vg -L 1g
sleep 2
lvcreate --name snap-$v --snapshot $vg/$v -L 1g
vgp=/dev/mapper/${vg/-/--}-snap--${v/-/--}
sleep 2
( sleep 10 < $vgp-cow ) &
test "$enable_fix" -eq "1" && lvchange -y -an $vg/snap-$v
lvremove -f $vg/snap-$v
sleep 1
lvremove -f $vg/$v
=== cut hang.sh ===
vg needs to be set to a lvm VG that exists and can take a few gig of space. whenever enable_fix is set to 0, lvremove -f ends with :
Unable to deactivate open cinder--volumes-snap--testvol.27700-cow (252:5)
Failed to resume snap-testvol.27700.
libdevmapper exiting with 1 device(s) still suspended.
this is because the sleep command before keeps a fd open on the -cow. The script then also never finishes and any other lvm command hangs as well.
apparently in real-life this is either udev or the dd command still having the fd open for some reason I have not yet understood.
The deactivation before removing seems to help. |
On lvm2 2.02.98, I can get cinder-volume to hang on issuing lvm related commands after deleting snapshots (that are non-thin provisioned LVM snapshots with clear_volume set to zero).
the issue is that lvm locks up due to trying to access suspended device mapper entries, and at some point cinder-volume does a lvm related command and hangs on that.
I can use this script to trigger the issue:
=== cut hang.sh ===
enable_fix=1
#enable_fix=0
vg=cinder-volumes
v=testvol.$$
lvcreate --name $v $vg -L 1g
sleep 2
lvcreate --name snap-$v --snapshot $vg/$v -L 1g
vgp=/dev/mapper/${vg/-/--}-snap--${v/-/--}
sleep 2
( sleep 10 < $vgp-cow ) &
test "$enable_fix" -eq "1" && lvchange -y -an $vg/snap-$v
lvremove -f $vg/snap-$v
sleep 1
lvremove -f $vg/$v
=== cut hang.sh ===
vg needs to be set to a lvm VG that exists and can take a few gig of space. whenever enable_fix is set to 0, lvremove -f ends with :
Unable to deactivate open cinder--volumes-snap--testvol.27700-cow (252:5)
Failed to resume snap-testvol.27700.
libdevmapper exiting with 1 device(s) still suspended.
this is because the sleep command before keeps a fd open on the -cow. The script then also never finishes and any other lvm command hangs as well.
apparently in real-life this is either udev or the dd command still having the fd open for some reason I have not yet understood.
The deactivation before removing seems to help. |
|
| 2014-03-01 09:57:16 |
lirenke |
bug |
|
|
added subscriber lirenke |
| 2014-03-02 21:20:45 |
OpenStack Infra |
cinder: status |
In Progress |
Fix Committed |
|
| 2014-03-03 05:59:21 |
lirenke |
bug watch added |
|
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=659762 |
|
| 2014-03-05 17:16:04 |
Thierry Carrez |
cinder: status |
Fix Committed |
Fix Released |
|
| 2014-03-05 17:16:04 |
Thierry Carrez |
cinder: milestone |
|
icehouse-3 |
|
| 2014-04-17 09:33:01 |
Thierry Carrez |
cinder: milestone |
icehouse-3 |
2014.1 |
|
| 2014-05-21 17:00:01 |
Mike Perez |
tags |
|
drivers lvm |
|
| 2014-06-12 21:03:14 |
Dirk Mueller |
description |
On lvm2 2.02.98, I can get cinder-volume to hang on issuing lvm related commands after deleting snapshots (that are non-thin provisioned LVM snapshots with clear_volume set to zero).
the issue is that lvm locks up due to trying to access suspended device mapper entries, and at some point cinder-volume does a lvm related command and hangs on that.
I can use this script to trigger the issue:
=== cut hang.sh ===
enable_fix=1
#enable_fix=0
vg=cinder-volumes
v=testvol.$$
lvcreate --name $v $vg -L 1g
sleep 2
lvcreate --name snap-$v --snapshot $vg/$v -L 1g
vgp=/dev/mapper/${vg/-/--}-snap--${v/-/--}
sleep 2
( sleep 10 < $vgp-cow ) &
test "$enable_fix" -eq "1" && lvchange -y -an $vg/snap-$v
lvremove -f $vg/snap-$v
sleep 1
lvremove -f $vg/$v
=== cut hang.sh ===
vg needs to be set to a lvm VG that exists and can take a few gig of space. whenever enable_fix is set to 0, lvremove -f ends with :
Unable to deactivate open cinder--volumes-snap--testvol.27700-cow (252:5)
Failed to resume snap-testvol.27700.
libdevmapper exiting with 1 device(s) still suspended.
this is because the sleep command before keeps a fd open on the -cow. The script then also never finishes and any other lvm command hangs as well.
apparently in real-life this is either udev or the dd command still having the fd open for some reason I have not yet understood.
The deactivation before removing seems to help. |
On any lvm2 version without lvmetad running, I can get cinder-volume to hang on issuing lvm related commands after deleting snapshots (that are non-thin provisioned LVM snapshots with clear_volume set to zero).
the issue is that lvm locks up due to trying to access suspended device mapper entries, and at some point cinder-volume does a lvm related command and hangs on that. a setting of ignore_suspended_devices = 1 in lvm.conf helps with that, as lvremove hangs on scanning the device state (which it
needs to do because it doesn't have current information available via lvmetad).
I can use this script to trigger the issue:
=== cut hang.sh ===
enable_fix=1
#enable_fix=0
vg=cinder-volumes
v=testvol.$$
lvcreate --name $v $vg -L 1g
sleep 2
lvcreate --name snap-$v --snapshot $vg/$v -L 1g
vgp=/dev/mapper/${vg/-/--}-snap--${v/-/--}
sleep 2
( sleep 10 < $vgp-cow ) &
test "$enable_fix" -eq "1" && lvchange -y -an $vg/snap-$v
lvremove -f $vg/snap-$v
sleep 1
lvremove -f $vg/$v
=== cut hang.sh ===
vg needs to be set to a lvm VG that exists and can take a few gig of space. whenever enable_fix is set to 0, lvremove -f ends with :
Unable to deactivate open cinder--volumes-snap--testvol.27700-cow (252:5)
Failed to resume snap-testvol.27700.
libdevmapper exiting with 1 device(s) still suspended.
this is because the sleep command before keeps a fd open on the -cow. The script then also never finishes and any other lvm command hangs as well.
apparently in real-life this is either udev or the dd command still having the fd open for some reason I have not yet understood.
The deactivation before removing seems to help. |
|
| 2014-07-29 22:59:59 |
OpenStack Infra |
tags |
drivers lvm |
drivers in-stable-icehouse lvm |
|
| 2014-08-07 01:19:36 |
Chuck Short |
nominated for series |
|
cinder/icehouse |
|
| 2014-08-07 01:19:36 |
Chuck Short |
bug task added |
|
cinder/icehouse |
|
| 2014-08-07 01:19:42 |
Chuck Short |
cinder/icehouse: milestone |
|
2014.1.2 |
|
| 2014-08-07 01:19:45 |
Chuck Short |
cinder/icehouse: status |
New |
Fix Committed |
|
| 2014-08-07 19:10:12 |
Chuck Short |
cinder/icehouse: status |
Fix Committed |
Fix Released |
|
| 2015-10-30 20:56:19 |
Matt Riedemann |
bug task added |
|
nova |
|
| 2015-10-30 20:56:33 |
Matt Riedemann |
tags |
drivers in-stable-icehouse lvm |
drivers in-stable-icehouse lvm volumes |
|
| 2015-10-30 21:18:59 |
Matt Riedemann |
bug task deleted |
nova |
|
|
