lvremove hangs in D state while running OSTF test

[Dmitry Teselkin]

Upstream bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1523491

---
2015-12-05 03:38:20 FAILURE Create volume and attach it to instance (fuel_health.tests.smoke.test_create_volume.VolumesTest.test_volume_create) Time limit exceeded while waiting for volume becoming 'available' to finish. Please refer to OpenStack logs for more details. File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/smoke/test_create_volume.py", line 87, in test_volume_create
    volume, 'available')
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 183, in verify
    " Please refer to OpenStack logs for more details.")
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
Step 2 failed: Time limit exceeded while waiting for volume becoming 'available' to finish. Please refer to OpenStack logs for more details.
---

On a target system we have
---
root@node-3:~# ps aux| grep lv
root 8336 0.0 0.0 10432 624 pts/6 S+ 14:56 0:00 grep --color=auto lv
root 31813 0.0 0.0 62096 2032 ? S 14:16 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvremove --config activation { retry_deactivation = 1} -f cinder/volume-f4aebd4f-d1f4-4bbd-9cfd-01d58ac53ac0
root 31817 0.0 0.2 31464 7752 ? S 14:16 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvremove --config activation { retry_deactivation = 1} -f cinder/volume-f4aebd4f-d1f4-4bbd-9cfd-01d58ac53ac0
root 31818 0.0 0.5 46564 15644 ? D<L 14:16 0:00 /sbin/lvremove --config activation { retry_deactivation = 1} -f cinder/volume-f4aebd4f-d1f4-4bbd-9cfd-01d58ac53ac0
root 32015 0.0 0.0 62096 2028 ? S 14:16 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder
root 32016 0.0 0.2 31464 7756 ? S 14:16 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder
root 32019 0.0 0.0 36148 2048 ? S 14:16 0:00 /sbin/vgs --noheadings --unit=g -o name,size,free,lv_count,uuid --separator : --nosuffix cinder
root 32064 0.0 0.0 62096 2028 ? S 14:16 0:00 sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-91d06179-130c-476a-a1df-bdacd3671a1c cinder -L 1g
root 32065 0.0 0.2 31464 7756 ? S 14:16 0:00 /usr/bin/python /usr/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-91d06179-130c-476a-a1df-bdacd3671a1c cinder -L 1g
root 32066 0.0 0.0 36148 2048 ? S 14:16 0:00 /sbin/lvcreate -n volume-91d06179-130c-476a-a1df-bdacd3671a1c cinder -L 1g
---

---
root@node-3:/var/log# dmesg | grep lvremove
[ 2640.624092] INFO: task lvremove:31818 blocked for more than 120 seconds.
[ 2640.629039] lvremove D ffff8800bce13180 0 31818 31817 0x00000000
[ 2760.628068] INFO: task lvremove:31818 blocked for more than 120 seconds.
[ 2760.631812] lvremove D ffff8800bce13180 0 31818 31817 0x00000000
[ 2880.628034] INFO: task lvremove:31818 blocked for more than 120 seconds.
[ 2880.631224] lvremove D ffff8800bce13180 0 31818 31817 0x00000000
[ 3000.628092] INFO: task lvremove:31818 blocked for more than 120 seconds.
[ 3000.635711] lvremove D ffff8800bce13180 0 31818 31817 0x00000000
[ 3120.632086] INFO: task lvremove:31818 blocked for more than 120 seconds.
[ 3120.638963] lvremove D ffff8800bce13180 0 31818 31817 0x00000000
[ 3240.636050] INFO: task lvremove:31818 blocked for more than 120 seconds.
[ 3240.641051] lvremove D ffff8800bce13180 0 31818 31817 0x00000000
[ 3360.640087] INFO: task lvremove:31818 blocked for more than 120 seconds.
[ 3360.646907] lvremove D ffff8800bce13180 0 31818 31817 0x00000000
[ 3480.644098] INFO: task lvremove:31818 blocked for more than 120 seconds.
[ 3480.650870] lvremove D ffff8800bce13180 0 31818 31817 0x00000000
[ 3720.648066] INFO: task lvremove:31818 blocked for more than 120 seconds.
[ 3720.652524] lvremove D ffff8800bce13180 0 31818 31817 0x00000000
[ 3840.652062] INFO: task lvremove:31818 blocked for more than 120 seconds.
[ 3840.655915] lvremove D ffff8800bce13180 0 31818 31817 0x00000000
---

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/14432

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14432
Submitter: Pawel Brzozowski <email address hidden>
Branch: master

Commit: 88973bf335b1ae0efe6ca418dc2e24df6246f099
Author: Artem Silenkov <email address hidden>
Date: Sat Dec 5 15:04:43 2015

lvm2 backported from ubuntu xenial

We have lvremove stuck trying to delete volume
We could try to verify against last stable version

* packages/trusty/lvm2
* sources are http://packages.ubuntu.com/xenial/lvm2

Change-Id: I50f6013f28dc5f98787ce9941a35a7d0cc5cf9ac
Related-Bug: #1523107

[Artem Silenkov (asilenkov)]

This package is ont intended to be merged.
Just for tests do not merge pls.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 8.0
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/14433

[Artem Silenkov (asilenkov)]

Root cause is this kernel:
https://launchpad.net/ubuntu/+source/linux/3.13.0-72.115
from proposed-updates.

[Artem Silenkov (asilenkov)]

Lowering down priority as we have workaround for this.

Issue was not fixed we need to investigate further. Just probability of this error was lowered but not eliminated completely.

[Pavel Boldin (pboldin)]

The problem is incorrect backporting of [1] to the Ubuntu Trusty kernel [2].

While original patch moves `dm_get_live_table` to the appropriate section mutex lock the backported code just adds another call to it under the mutex leaving the original call under spinlock in place.

[1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=2a708cff93f1845b9239bc7d6310aef54e716c6a
[2] http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git/commit/?id=340f3e9970d6251a549d0c8cf90179b190f15def

[Roman Podoliaka (rpodolyaka)]

Per comment from the upstream bug report:

We are aware of this issue. The correct backport is in Ubuntu-3.13.0-73.116 (currently in -proposed)

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1523491/comments/2

[Artem Silenkov (asilenkov)]

There is nothing to do from packaging team side.

[Roman Podoliaka (rpodolyaka)]

As stated in #8, this is not going to make into -updates and is already fixed in -proposed. \

Marking as Invalid.