When this happens I get the following in my kern.log:
Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.188153] INFO: task kworker/0:1:19 blocked for more than 120 seconds. Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.190751] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.193638] kworker/0:1 D 0000000000000000 0 19 2 0x00000000 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.193641] ffff88007b8ebab0 0000000000000046 00000000357fca28 ffffffff81c17400 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.193644] ffff88007b8ebfd8 ffff88007b8ebfd8 ffff88007b8ebfd8 00000000000136c0 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.193646] ffff88007aaf5b80 ffff88007b8d44a0 ffff88007b8eba80 ffff88007fc13f80 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.193648] Call Trace: Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.197632] [<ffffffff816508cf>] schedule+0x3f/0x60 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220403] [<ffffffff8165097f>] io_schedule+0x8f/0xd0 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220437] [<ffffffff814f66bd>] sync_io+0x8d/0x140 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220457] [<ffffffff81117067>] ? mempool_free+0x57/0xb0 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220466] [<ffffffff814f68a8>] dm_io+0x138/0x140 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220473] [<ffffffff814f60b0>] ? dp_init+0xe0/0xe0 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220480] [<ffffffff814f5f70>] ? bvec_next_page+0x10/0x10 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220495] [<ffffffffa0309601>] chunk_io+0x111/0x120 [dm_snapshot] Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220507] [<ffffffffa0308180>] ? pending_complete+0x310/0x310 [dm_snapshot] Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220519] [<ffffffffa0309867>] persistent_commit_exception+0x117/0x160 [dm_snapshot] Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220530] [<ffffffffa0308180>] ? pending_complete+0x310/0x310 [dm_snapshot] Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220540] [<ffffffffa03081b3>] copy_callback+0x33/0x50 [dm_snapshot] Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220549] [<ffffffff814f72c4>] run_complete_job+0x74/0xd0 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220557] [<ffffffff814f6f55>] process_jobs+0x75/0x110 Sep 11 19:09:40 server-3bc2ef49-4871-4dcc-a7c4-7de14141e6ec kernel: [ 1680.220564] [<ffffffff814f7250>] ? dispatch_job+0x80/0x80
When this happens I get the following in my kern.log:
Sep 11 19:09:40 server- 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.188153] INFO: task kworker/0:1:19 blocked for more than 120 seconds. 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.190751] "echo 0 > /proc/sys/ kernel/ hung_task_ timeout_ secs" disables this message. 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.193638] kworker/0:1 D 0000000000000000 0 19 2 0x00000000 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.193641] ffff88007b8ebab0 0000000000000046 00000000357fca28 ffffffff81c17400 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.193644] ffff88007b8ebfd8 ffff88007b8ebfd8 ffff88007b8ebfd8 00000000000136c0 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.193646] ffff88007aaf5b80 ffff88007b8d44a0 ffff88007b8eba80 ffff88007fc13f80 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.193648] Call Trace: 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.197632] [<ffffffff81650 8cf>] schedule+0x3f/0x60 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220403] [<ffffffff81650 97f>] io_schedule+ 0x8f/0xd0 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220437] [<ffffffff814f6 6bd>] sync_io+0x8d/0x140 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220457] [<ffffffff81117 067>] ? mempool_ free+0x57/ 0xb0 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220466] [<ffffffff814f6 8a8>] dm_io+0x138/0x140 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220473] [<ffffffff814f6 0b0>] ? dp_init+0xe0/0xe0 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220480] [<ffffffff814f5 f70>] ? bvec_next_ page+0x10/ 0x10 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220495] [<ffffffffa0309 601>] chunk_io+ 0x111/0x120 [dm_snapshot] 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220507] [<ffffffffa0308 180>] ? pending_ complete+ 0x310/0x310 [dm_snapshot] 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220519] [<ffffffffa0309 867>] persistent_ commit_ exception+ 0x117/0x160 [dm_snapshot] 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220530] [<ffffffffa0308 180>] ? pending_ complete+ 0x310/0x310 [dm_snapshot] 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220540] [<ffffffffa0308 1b3>] copy_callback+ 0x33/0x50 [dm_snapshot] 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220549] [<ffffffff814f7 2c4>] run_complete_ job+0x74/ 0xd0 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220557] [<ffffffff814f6 f55>] process_ jobs+0x75/ 0x110 3bc2ef49- 4871-4dcc- a7c4-7de14141e6 ec kernel: [ 1680.220564] [<ffffffff814f7 250>] ? dispatch_ job+0x80/ 0x80
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-
Sep 11 19:09:40 server-