On Ubuntu 14.04 something really wrong happens: According the dmesg the lvm relates tasks are blocked in the io scheduler. [ 1440.468099] INFO: task lvm:21729 blocked for more than 120 seconds. [ 1440.469298] Not tainted 3.13.0-24-generic #47-Ubuntu [ 1440.470334] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1440.471736] lvm D ffff88021fd94440 0 21729 21726 0x00000000 [ 1440.471741] ffff8801c259dae0 0000000000000002 ffff8800ae3ac7d0 ffff8801c259dfd8 [ 1440.471743] 0000000000014440 0000000000014440 ffff8800ae3ac7d0 ffff88021fd94cd8 [ 1440.471745] 0000000000000000 ffff880212174f00 0000000000000000 ffff8800ae3ac7d0 [ 1440.471748] Call Trace: [ 1440.471755] [<ffffffff8171a20d>] io_schedule+0x9d/0x140 [ 1440.471760] [<ffffffff811f6d74>] do_blockdev_direct_IO+0x1ce4/0x2910 [ 1440.471762] [<ffffffff811f2640>] ? bdev_inode_switch_bdi+0x120/0x160 [ 1440.471765] [<ffffffff811f1be0>] ? I_BDEV+0x10/0x10 [ 1440.471767] [<ffffffff811f79f5>] __blockdev_direct_IO+0x55/0x60 [ 1440.471768] [<ffffffff811f1be0>] ? I_BDEV+0x10/0x10 [ 1440.471770] [<ffffffff811f22d6>] blkdev_direct_IO+0x56/0x60 [ 1440.471772] [<ffffffff811f1be0>] ? I_BDEV+0x10/0x10 [ 1440.471777] [<ffffffff81150b5b>] generic_file_aio_read+0x69b/0x700 [ 1440.471781] [<ffffffff811c8e18>] ? path_openat+0x158/0x620 [ 1440.471783] [<ffffffff811f275b>] blkdev_aio_read+0x4b/0x70 [ 1440.471786] [<ffffffff811b8d1a>] do_sync_read+0x5a/0x90 [ 1440.471789] [<ffffffff811b93b5>] vfs_read+0x95/0x160 [ 1440.471790] [<ffffffff811b9ec9>] SyS_read+0x49/0xa0 [ 1440.471794] [<ffffffff817266bf>] tracesys+0xe1/0xe6 [ 1440.471796] INFO: task lvremove:21759 blocked for more than 120 seconds. [ 1440.473042] Not tainted 3.13.0-24-generic #47-Ubuntu [ 1440.474019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1440.475436] lvremove D ffff88021fc14440 0 21759 21544 0x00000000 [ 1440.475440] ffff8800ae1c3ae0 0000000000000002 ffff8801dad12fe0 ffff8800ae1c3fd8 [ 1440.475442] 0000000000014440 0000000000014440 ffff8801dad12fe0 ffff88021fc14cd8 [ 1440.475444] 0000000000000000 ffff8801daeaf200 0000000000000000 ffff8801dad12fe0 [ 1440.475446] Call Trace: [ 1440.475449] [<ffffffff8171a20d>] io_schedule+0x9d/0x140 [ 1440.475451] [<ffffffff811f6d74>] do_blockdev_direct_IO+0x1ce4/0x2910 [ 1440.475454] [<ffffffff811f1be0>] ? I_BDEV+0x10/0x10 [ 1440.475456] [<ffffffff811f79f5>] __blockdev_direct_IO+0x55/0x60 [ 1440.475457] [<ffffffff811f1be0>] ? I_BDEV+0x10/0x10 [ 1440.475459] [<ffffffff811f22d6>] blkdev_direct_IO+0x56/0x60 [ 1440.475461] [<ffffffff811f1be0>] ? I_BDEV+0x10/0x10 [ 1440.475463] [<ffffffff81150b5b>] generic_file_aio_read+0x69b/0x700 [ 1440.475466] [<ffffffff811c8e18>] ? path_openat+0x158/0x620 [ 1440.475468] [<ffffffff811f275b>] blkdev_aio_read+0x4b/0x70 [ 1440.475470] [<ffffffff811b8d1a>] do_sync_read+0x5a/0x90 [ 1440.475472] [<ffffffff811b93b5>] vfs_read+0x95/0x160 [ 1440.475474] [<ffffffff811b9ec9>] SyS_read+0x49/0xa0 [ 1440.475476] [<ffffffff817266bf>] tracesys+0xe1/0xe6
It does not seams to be cinder bug, please consider adding kernel or lvm or udev to the affected components.
I was not able to reproduce this issue on Fedora 20.
I also noted after an lvremove all lvs is active on Ubuntu 14.04, it is unusual.
On Ubuntu 14.04 something really wrong happens: kernel/ hung_task_ timeout_ secs" disables this message. 20d>] io_schedule+ 0x9d/0x140 d74>] do_blockdev_ direct_ IO+0x1ce4/ 0x2910 640>] ? bdev_inode_ switch_ bdi+0x120/ 0x160 be0>] ? I_BDEV+0x10/0x10 9f5>] __blockdev_ direct_ IO+0x55/ 0x60 be0>] ? I_BDEV+0x10/0x10 2d6>] blkdev_ direct_ IO+0x56/ 0x60 be0>] ? I_BDEV+0x10/0x10 b5b>] generic_ file_aio_ read+0x69b/ 0x700 e18>] ? path_openat+ 0x158/0x620 75b>] blkdev_ aio_read+ 0x4b/0x70 d1a>] do_sync_ read+0x5a/ 0x90 3b5>] vfs_read+0x95/0x160 ec9>] SyS_read+0x49/0xa0 6bf>] tracesys+0xe1/0xe6 kernel/ hung_task_ timeout_ secs" disables this message. 20d>] io_schedule+ 0x9d/0x140 d74>] do_blockdev_ direct_ IO+0x1ce4/ 0x2910 be0>] ? I_BDEV+0x10/0x10 9f5>] __blockdev_ direct_ IO+0x55/ 0x60 be0>] ? I_BDEV+0x10/0x10 2d6>] blkdev_ direct_ IO+0x56/ 0x60 be0>] ? I_BDEV+0x10/0x10 b5b>] generic_ file_aio_ read+0x69b/ 0x700 e18>] ? path_openat+ 0x158/0x620 75b>] blkdev_ aio_read+ 0x4b/0x70 d1a>] do_sync_ read+0x5a/ 0x90 3b5>] vfs_read+0x95/0x160 ec9>] SyS_read+0x49/0xa0 6bf>] tracesys+0xe1/0xe6
According the dmesg the lvm relates tasks are blocked in the io scheduler.
[ 1440.468099] INFO: task lvm:21729 blocked for more than 120 seconds.
[ 1440.469298] Not tainted 3.13.0-24-generic #47-Ubuntu
[ 1440.470334] "echo 0 > /proc/sys/
[ 1440.471736] lvm D ffff88021fd94440 0 21729 21726 0x00000000
[ 1440.471741] ffff8801c259dae0 0000000000000002 ffff8800ae3ac7d0 ffff8801c259dfd8
[ 1440.471743] 0000000000014440 0000000000014440 ffff8800ae3ac7d0 ffff88021fd94cd8
[ 1440.471745] 0000000000000000 ffff880212174f00 0000000000000000 ffff8800ae3ac7d0
[ 1440.471748] Call Trace:
[ 1440.471755] [<ffffffff8171a
[ 1440.471760] [<ffffffff811f6
[ 1440.471762] [<ffffffff811f2
[ 1440.471765] [<ffffffff811f1
[ 1440.471767] [<ffffffff811f7
[ 1440.471768] [<ffffffff811f1
[ 1440.471770] [<ffffffff811f2
[ 1440.471772] [<ffffffff811f1
[ 1440.471777] [<ffffffff81150
[ 1440.471781] [<ffffffff811c8
[ 1440.471783] [<ffffffff811f2
[ 1440.471786] [<ffffffff811b8
[ 1440.471789] [<ffffffff811b9
[ 1440.471790] [<ffffffff811b9
[ 1440.471794] [<ffffffff81726
[ 1440.471796] INFO: task lvremove:21759 blocked for more than 120 seconds.
[ 1440.473042] Not tainted 3.13.0-24-generic #47-Ubuntu
[ 1440.474019] "echo 0 > /proc/sys/
[ 1440.475436] lvremove D ffff88021fc14440 0 21759 21544 0x00000000
[ 1440.475440] ffff8800ae1c3ae0 0000000000000002 ffff8801dad12fe0 ffff8800ae1c3fd8
[ 1440.475442] 0000000000014440 0000000000014440 ffff8801dad12fe0 ffff88021fc14cd8
[ 1440.475444] 0000000000000000 ffff8801daeaf200 0000000000000000 ffff8801dad12fe0
[ 1440.475446] Call Trace:
[ 1440.475449] [<ffffffff8171a
[ 1440.475451] [<ffffffff811f6
[ 1440.475454] [<ffffffff811f1
[ 1440.475456] [<ffffffff811f7
[ 1440.475457] [<ffffffff811f1
[ 1440.475459] [<ffffffff811f2
[ 1440.475461] [<ffffffff811f1
[ 1440.475463] [<ffffffff81150
[ 1440.475466] [<ffffffff811c8
[ 1440.475468] [<ffffffff811f2
[ 1440.475470] [<ffffffff811b8
[ 1440.475472] [<ffffffff811b9
[ 1440.475474] [<ffffffff811b9
[ 1440.475476] [<ffffffff81726
It does not seams to be cinder bug, please consider adding kernel or lvm or udev to the affected components.
I was not able to reproduce this issue on Fedora 20.
I also noted after an lvremove all lvs is active on Ubuntu 14.04,
it is unusual.