Comment 4 for bug 1197648

Revision history for this message
Rongze Zhu (zrzhit) wrote : Re: Unable to clone a volume

This is dmesg:

[ 5754.760185] INFO: task kworker/1:1:3801 blocked for more than 120 seconds.
[ 5754.761229] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5754.764108] kworker/1:1 D 0000000000000001 0 3801 2 0x00000000
[ 5754.764114] ffff8800386dbaa0 0000000000000046 ffff8800386dbad0 ffffffff8152c397
[ 5754.764119] ffff8800386dbfd8 ffff8800386dbfd8 ffff8800386dbfd8 00000000000139c0
[ 5754.764159] ffff88003864ae00 ffff8800385dc500 ffff8800386dba70 ffff88007f634258
[ 5754.764163] Call Trace:
[ 5754.764295] [<ffffffff8152c397>] ? dispatch_io+0x117/0x130
[ 5754.764329] [<ffffffff8169d8b9>] schedule+0x29/0x70
[ 5754.764335] [<ffffffff8169d98f>] io_schedule+0x8f/0xd0
[ 5754.764338] [<ffffffff8152c43d>] sync_io+0x8d/0x140
[ 5754.764342] [<ffffffff8152c43d>] ? sync_io+0x8d/0x140
[ 5754.764404] [<ffffffff8117268a>] ? discard_slab+0x3a/0x50
[ 5754.764409] [<ffffffff8152c628>] dm_io+0x138/0x140
[ 5754.764412] [<ffffffff8152c628>] ? dm_io+0x138/0x140
[ 5754.764415] [<ffffffff8152be30>] ? dp_init+0xe0/0xe0
[ 5754.764418] [<ffffffff8152bcf0>] ? bvec_next_page+0x10/0x10
[ 5754.764445] [<ffffffffa04df551>] chunk_io+0x111/0x120 [dm_snapshot]
[ 5754.764450] [<ffffffffa04df551>] ? chunk_io+0x111/0x120 [dm_snapshot]
[ 5754.764454] [<ffffffffa04de160>] ? pending_complete+0x310/0x310 [dm_snapshot]
[ 5754.764458] [<ffffffffa04df5f6>] area_io+0x26/0x30 [dm_snapshot]
[ 5754.764462] [<ffffffffa04df77d>] persistent_commit_exception+0xdd/0x160 [dm_snapshot]
[ 5754.764465] [<ffffffffa04de160>] ? pending_complete+0x310/0x310 [dm_snapshot]
[ 5754.764469] [<ffffffffa04de193>] copy_callback+0x33/0x50 [dm_snapshot]
[ 5754.764472] [<ffffffff8152d054>] run_complete_job+0x74/0xd0
[ 5754.764475] [<ffffffff8152cce5>] process_jobs+0x75/0x110
[ 5754.764478] [<ffffffff8152cfe0>] ? dispatch_job+0x80/0x80
[ 5754.764481] [<ffffffff8152cd80>] ? process_jobs+0x110/0x110
[ 5754.764484] [<ffffffff8152cdb3>] do_work+0x33/0x70
[ 5754.764496] [<ffffffff81071bba>] process_one_work+0x11a/0x480
[ 5754.764500] [<ffffffff81072bd5>] worker_thread+0x165/0x370
[ 5754.764503] [<ffffffff81072a70>] ? manage_workers.isra.29+0x130/0x130
[ 5754.764506] [<ffffffff81077b73>] kthread+0x93/0xa0
[ 5754.764515] [<ffffffff816a8324>] kernel_thread_helper+0x4/0x10
[ 5754.764519] [<ffffffff81077ae0>] ? flush_kthread_worker+0xb0/0xb0
[ 5754.764522] [<ffffffff816a8320>] ? gs_change+0x13/0x13
[ 5754.764563] INFO: task vgs:5495 blocked for more than 120 seconds.
[ 5754.765987] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5754.767635] vgs D ffffffff8180cbe0 0 5495 5494 0x00000000
[ 5754.767640] ffff88006d02faa8 0000000000000086 0000000000000000 ffff88007b30f300
[ 5754.767645] ffff88006d02ffd8 ffff88006d02ffd8 ffff88006d02ffd8 00000000000139c0
[ 5754.767648] ffff88007c05c500 ffff880038621700 ffff88006d02fa78 ffff88007f654258
[ 5754.767652] Call Trace:
[ 5754.767659] [<ffffffff8169d8b9>] schedule+0x29/0x70
[ 5754.767662] [<ffffffff8169d98f>] io_schedule+0x8f/0xd0
[ 5754.767741] [<ffffffff811c1234>] dio_await_completion+0x54/0xd0
[ 5754.767751] [<ffffffff811c36f4>] do_blockdev_direct_IO+0x964/0xdd0
[ 5754.767754] [<ffffffff811bfed0>] ? lookup_bdev+0xa0/0xa0
[ 5754.767785] [<ffffffff811c3bb5>] __blockdev_direct_IO+0x55/0x60
[ 5754.767790] [<ffffffff811bfed0>] ? lookup_bdev+0xa0/0xa0
[ 5754.767795] [<ffffffff811bf1f7>] blkdev_direct_IO+0x57/0x60
[ 5754.767797] [<ffffffff811bfed0>] ? lookup_bdev+0xa0/0xa0
[ 5754.767844] [<ffffffff81129813>] generic_file_aio_read+0x243/0x280
[ 5754.767852] [<ffffffff81196e6e>] ? path_openat+0x10e/0x440
[ 5754.767858] [<ffffffff8118734a>] do_sync_read+0xda/0x120
[ 5754.767885] [<ffffffff812b9e03>] ? security_file_permission+0x93/0xb0
[ 5754.767889] [<ffffffff811877d1>] ? rw_verify_area+0x61/0xf0
[ 5754.767892] [<ffffffff81187cb0>] vfs_read+0xb0/0x180
[ 5754.767895] [<ffffffff81187dca>] sys_read+0x4a/0x90
[ 5754.767900] [<ffffffff816a7029>] system_call_fastpath+0x16/0x1b
[ 5874.627608] INFO: task vgs:5495 blocked for more than 120 seconds.
[ 5874.628551] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5874.629901] vgs D ffffffff8180cbe0 0 5495 5494 0x00000000
[ 5874.629905] ffff88006d02faa8 0000000000000086 0000000000000000 ffff88007b30f300
[ 5874.629908] ffff88006d02ffd8 ffff88006d02ffd8 ffff88006d02ffd8 00000000000139c0
[ 5874.629933] ffff88007c05c500 ffff880038621700 ffff88006d02fa78 ffff88007f654258
[ 5874.629935] Call Trace:
[ 5874.629942] [<ffffffff8169d8b9>] schedule+0x29/0x70
[ 5874.629945] [<ffffffff8169d98f>] io_schedule+0x8f/0xd0
[ 5874.629950] [<ffffffff811c1234>] dio_await_completion+0x54/0xd0
[ 5874.629952] [<ffffffff811c36f4>] do_blockdev_direct_IO+0x964/0xdd0
[ 5874.629954] [<ffffffff811bfed0>] ? lookup_bdev+0xa0/0xa0
[ 5874.629956] [<ffffffff811c3bb5>] __blockdev_direct_IO+0x55/0x60
[ 5874.629958] [<ffffffff811bfed0>] ? lookup_bdev+0xa0/0xa0
[ 5874.629961] [<ffffffff811bf1f7>] blkdev_direct_IO+0x57/0x60
[ 5874.629962] [<ffffffff811bfed0>] ? lookup_bdev+0xa0/0xa0
[ 5874.629966] [<ffffffff81129813>] generic_file_aio_read+0x243/0x280
[ 5874.629969] [<ffffffff81196e6e>] ? path_openat+0x10e/0x440
[ 5874.629972] [<ffffffff8118734a>] do_sync_read+0xda/0x120
[ 5874.629976] [<ffffffff812b9e03>] ? security_file_permission+0x93/0xb0
[ 5874.629978] [<ffffffff811877d1>] ? rw_verify_area+0x61/0xf0
[ 5874.629980] [<ffffffff81187cb0>] vfs_read+0xb0/0x180
[ 5874.629982] [<ffffffff81187dca>] sys_read+0x4a/0x90
[ 5874.629985] [<ffffffff816a7029>] system_call_fastpath+0x16/0x1b
[ 5874.629987] INFO: task lvs:5513 blocked for more than 120 seconds.
[ 5874.631678] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5874.633143] lvs D ffffffff8180cbe0 0 5513 5512 0x00000000
[ 5874.633148] ffff88000d8d9aa8 0000000000000082 0000000000000000 ffff88007b30f580
[ 5874.633152] ffff88000d8d9fd8 ffff88000d8d9fd8 ffff88000d8d9fd8 00000000000139c0
[ 5874.633154] ffff88007c05c500 ffff88006583ae00 ffff88000d8d9a78 ffff88007f654258
[ 5874.633156] Call Trace:
[ 5874.633164] [<ffffffff8169d8b9>] schedule+0x29/0x70
[ 5874.633167] [<ffffffff8169d98f>] io_schedule+0x8f/0xd0
[ 5874.633170] [<ffffffff811c1234>] dio_await_completion+0x54/0xd0
[ 5874.633172] [<ffffffff811c36f4>] do_blockdev_direct_IO+0x964/0xdd0
[ 5874.633175] [<ffffffff811bfed0>] ? lookup_bdev+0xa0/0xa0
[ 5874.633177] [<ffffffff811c3bb5>] __blockdev_direct_IO+0x55/0x60
[ 5874.633179] [<ffffffff811bfed0>] ? lookup_bdev+0xa0/0xa0
[ 5874.633182] [<ffffffff811bf1f7>] blkdev_direct_IO+0x57/0x60
[ 5874.633183] [<ffffffff811bfed0>] ? lookup_bdev+0xa0/0xa0
[ 5874.633187] [<ffffffff81129813>] generic_file_aio_read+0x243/0x280
[ 5874.633191] [<ffffffff81196e6e>] ? path_openat+0x10e/0x440
[ 5874.633195] [<ffffffff8118734a>] do_sync_read+0xda/0x120
[ 5874.633198] [<ffffffff812b9e03>] ? security_file_permission+0x93/0xb0
[ 5874.633201] [<ffffffff811877d1>] ? rw_verify_area+0x61/0xf0
[ 5874.633203] [<ffffffff81187cb0>] vfs_read+0xb0/0x180
[ 5874.633205] [<ffffffff81187dca>] sys_read+0x4a/0x90
[ 5874.633209] [<ffffffff816a7029>] system_call_fastpath+0x16/0x1b
[ 5989.039990] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
[ 5989.231592] quiet_error: 262279 callbacks suppressed
[ 5989.231597] Buffer I/O error on device dm-8, logical block 261122
[ 5989.231658] Buffer I/O error on device dm-8, logical block 261123
[ 5989.231659] lost page write due to I/O error on dm-8
[ 5989.231677] Buffer I/O error on device dm-8, logical block 261124
[ 5989.231677] lost page write due to I/O error on dm-8
[ 5989.231684] Buffer I/O error on device dm-8, logical block 261125
[ 5989.231685] lost page write due to I/O error on dm-8
[ 5989.231691] Buffer I/O error on device dm-8, logical block 261126
[ 5989.231691] lost page write due to I/O error on dm-8
[ 5989.231696] Buffer I/O error on device dm-8, logical block 261127
[ 5989.231696] lost page write due to I/O error on dm-8
[ 5989.231702] Buffer I/O error on device dm-8, logical block 261128
[ 5989.231702] lost page write due to I/O error on dm-8
[ 5989.231707] Buffer I/O error on device dm-8, logical block 261129
[ 5989.231707] lost page write due to I/O error on dm-8
[ 5989.231712] Buffer I/O error on device dm-8, logical block 261130
[ 5989.231751] lost page write due to I/O error on dm-8
[ 5989.231752] Buffer I/O error on device dm-8, logical block 203264
[ 5989.231753] lost page write due to I/O error on dm-8
[ 5989.246128] lost page write due to I/O error on dm-8
zhu@ubuntu:/opt/stack/cinder$