| 2016-11-28 03:38:13 |
john |
bug |
|
|
added bug |
| 2016-11-28 03:42:50 |
john |
attachment added |
|
hung_kernel_messages.txt https://bugs.launchpad.net/ubuntu/+source/linux-lts-xenial/+bug/1645187/+attachment/4783929/+files/hung_kernel_messages.txt |
|
| 2016-11-28 03:44:12 |
john |
description |
Server's load become high with tasks in D state, no choice but to reboot the system.
Could be related to the following ? :
https://bugzilla.kernel.org/show_bug.cgi?id=119841
https://www.redhat.com/archives/dm-devel/2016-June/msg00399.html
https://patchwork.kernel.org/patch/9223697/
https://xen.crc.id.au/bugs/view.php?id=75
ii xen-hypervisor-4.4-amd64 4.4.2-0ubuntu0.14.04.7 amd64 Xen Hypervisor on AMD64
ii linux-image-extra-4.4.0-47-generic 4.4.0-47.68~14.04.1 amd64 Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii linux-image-4.4.0-47-generic 4.4.0-47.68~14.04.1 amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
kernel messages:
----------------
Nov 28 07:03:31 server1 kernel: [890070.994700] INFO: task blkback.3.xvda2:5756 blocked for more than 120 seconds.
Nov 28 07:03:31 server1 kernel: [890070.994758] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu
Nov 28 07:03:31 server1 kernel: [890070.994806] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 07:03:31 server1 kernel: [890070.994884] blkback.3.xvda2 D ffff8800b7ff3928 0 5756 2 0x00000000
Nov 28 07:03:31 server1 kernel: [890070.994890] ffff8800b7ff3928 ffffffff81e13500 ffff8800b6a4be80 ffff8800b7ff4000
Nov 28 07:03:31 server1 kernel: [890070.994895] ffff88013a83cc18 ffff88013a83cc00 ffffffff00000000 fffffffe00000001
Nov 28 07:03:31 server1 kernel: [890070.994898] ffff8800b7ff3940 ffffffff817fafc5 ffff8800b6a4be80 ffff8800b7ff39c0
Nov 28 07:03:31 server1 kernel: [890070.994902] Call Trace:
Nov 28 07:03:31 server1 kernel: [890070.994912] [<ffffffff817fafc5>] schedule+0x35/0x80
Nov 28 07:03:31 server1 kernel: [890070.994917] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320
Nov 28 07:03:31 server1 kernel: [890070.994923] [<ffffffff81689577>] ? push+0x47/0x50
Nov 28 07:03:31 server1 kernel: [890070.994927] [<ffffffff81689f07>] ? dm_kcopyd_copy+0x147/0x1f0
Nov 28 07:03:31 server1 kernel: [890070.994931] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20
Nov 28 07:03:31 server1 kernel: [890070.994933] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40
Nov 28 07:03:31 server1 kernel: [890070.994939] [<ffffffffc0314dae>] __origin_write+0x6e/0x210 [dm_snapshot]
Nov 28 07:03:31 server1 kernel: [890070.994944] [<ffffffff81185625>] ? mempool_alloc_slab+0x15/0x20
Nov 28 07:03:31 server1 kernel: [890070.994946] [<ffffffff8118574f>] ? mempool_alloc+0x5f/0x150
Nov 28 07:03:31 server1 kernel: [890070.994949] [<ffffffffc0314fb7>] do_origin.isra.14+0x67/0x90 [dm_snapshot]
Nov 28 07:03:31 server1 kernel: [890070.994952] [<ffffffffc0315042>] origin_map+0x62/0x80 [dm_snapshot]
Nov 28 07:03:31 server1 kernel: [890070.994955] [<ffffffff8167f2da>] __map_bio+0x3a/0x110
Nov 28 07:03:31 server1 kernel: [890070.994957] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0
Nov 28 07:03:31 server1 kernel: [890070.994960] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0
Nov 28 07:03:31 server1 kernel: [890070.994964] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0
Nov 28 07:03:31 server1 kernel: [890070.994968] [<ffffffff813aa357>] submit_bio+0x77/0x150
Nov 28 07:03:31 server1 kernel: [890070.994971] [<ffffffff813a1c71>] ? bio_alloc_bioset+0x181/0x2a0
Nov 28 07:03:31 server1 kernel: [890070.994977] [<ffffffffc02c6a1d>] dispatch_rw_block_io+0x4fd/0x9b0 [xen_blkback]
Nov 28 07:03:31 server1 kernel: [890070.994981] [<ffffffff8101c244>] ? xen_load_sp0+0x84/0x180
Nov 28 07:03:31 server1 kernel: [890070.994985] [<ffffffffc02c70c5>] __do_block_io_op+0x1f5/0x650 [xen_blkback]
Nov 28 07:03:31 server1 kernel: [890070.994990] [<ffffffff810e5e18>] ? del_timer_sync+0x48/0x50
Nov 28 07:03:31 server1 kernel: [890070.994993] [<ffffffff817fd8ab>] ? schedule_timeout+0x16b/0x2d0
Nov 28 07:03:31 server1 kernel: [890070.994997] [<ffffffffc02c7880>] xen_blkif_schedule+0xd0/0x820 [xen_blkback]
Nov 28 07:03:31 server1 kernel: [890070.995002] [<ffffffff810a4e1a>] ? finish_task_switch+0x7a/0x290
Nov 28 07:03:31 server1 kernel: [890070.995004] [<ffffffff817fa969>] ? __schedule+0x359/0x980
Nov 28 07:03:31 server1 kernel: [890070.995010] [<ffffffff810bde70>] ? prepare_to_wait_event+0xf0/0xf0
Nov 28 07:03:31 server1 kernel: [890070.995014] [<ffffffffc02c77b0>] ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
Nov 28 07:03:31 server1 kernel: [890070.995018] [<ffffffff8109ba29>] kthread+0xc9/0xe0
Nov 28 07:03:31 server1 kernel: [890070.995021] [<ffffffff8109b960>] ? kthread_park+0x60/0x60
Nov 28 07:03:31 server1 kernel: [890070.995025] [<ffffffff817febcf>] ret_from_fork+0x3f/0x70
Nov 28 07:03:31 server1 kernel: [890070.995027] [<ffffffff8109b960>] ? kthread_park+0x60/0x60
Nov 28 07:03:31 server1 kernel: [890070.995033] INFO: task kworker/u4:1:8922 blocked for more than 120 seconds.
Nov 28 07:03:31 server1 kernel: [890070.995106] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu
Nov 28 07:03:31 server1 kernel: [890070.995148] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 07:03:31 server1 kernel: [890070.995200] kworker/u4:1 D ffff8800a7fbb628 0 8922 2 0x00000000
Nov 28 07:03:31 server1 kernel: [890070.995208] Workqueue: writeback wb_workfn (flush-252:12)
Nov 28 07:03:31 server1 kernel: [890070.995210] ffff8800a7fbb628 ffff880139ad3200 ffff8800b7dfcb00 ffff8800a7fbc000
Nov 28 07:03:31 server1 kernel: [890070.995212] ffff88013a83cc18 ffff88013a83cc00 ffffffff00000000 fffffffe00000001
Nov 28 07:03:31 server1 kernel: [890070.995215] ffff8800a7fbb640 ffffffff817fafc5 ffff8800b7dfcb00 ffff8800a7fbb6c0
Nov 28 07:03:31 server1 kernel: [890070.995217] Call Trace:
Nov 28 07:03:31 server1 kernel: [890070.995220] [<ffffffff817fafc5>] schedule+0x35/0x80
Nov 28 07:03:31 server1 kernel: [890070.995223] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320
Nov 28 07:03:31 server1 kernel: [890070.995229] [<ffffffff811fad2a>] ? __slab_alloc+0x4d/0x5c
Nov 28 07:03:31 server1 kernel: [890070.995232] [<ffffffff811dc3eb>] ? kmem_cache_alloc+0x1bb/0x200
Nov 28 07:03:31 server1 kernel: [890070.995236] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20
Nov 28 07:03:31 server1 kernel: [890070.995239] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40
Nov 28 07:03:31 server1 kernel: [890070.995244] [<ffffffffc0315d12>] snapshot_map+0x62/0x390 [dm_snapshot]
Nov 28 07:03:31 server1 kernel: [890070.995272] [<ffffffff8167f2da>] __map_bio+0x3a/0x110
Nov 28 07:03:31 server1 kernel: [890070.995275] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0
Nov 28 07:03:31 server1 kernel: [890070.995278] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0
Nov 28 07:03:31 server1 kernel: [890070.995282] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0
Nov 28 07:03:31 server1 kernel: [890070.995284] [<ffffffff813aa357>] submit_bio+0x77/0x150
Nov 28 07:03:31 server1 kernel: [890070.995286] [<ffffffff813a1c71>] ? bio_alloc_bioset+0x181/0x2a0
Nov 28 07:03:31 server1 kernel: [890070.995289] [<ffffffff812347ef>] submit_bh_wbc+0x12f/0x160
Nov 28 07:03:31 server1 kernel: [890070.995292] [<ffffffff81236615>] __block_write_full_page.constprop.39+0x125/0x360
Nov 28 07:03:31 server1 kernel: [890070.995293] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20
Nov 28 07:03:31 server1 kernel: [890070.995295] [<ffffffff8123692e>] block_write_full_page+0xde/0x100
Nov 28 07:03:31 server1 kernel: [890070.995298] [<ffffffff81237378>] blkdev_writepage+0x18/0x20
Nov 28 07:03:31 server1 kernel: [890070.995300] [<ffffffff8118d253>] __writepage+0x13/0x40
Nov 28 07:03:31 server1 kernel: [890070.995302] [<ffffffff8118e5c1>] write_cache_pages+0x241/0x4c0
Nov 28 07:03:31 server1 kernel: [890070.995304] [<ffffffff8118d240>] ? wb_update_dirty_ratelimit+0x1c0/0x1c0
Nov 28 07:03:31 server1 kernel: [890070.995307] [<ffffffff8118e883>] generic_writepages+0x43/0x60
Nov 28 07:03:31 server1 kernel: [890070.995310] [<ffffffff8118d3ff>] ? __wb_calc_thresh+0x2f/0x120
Nov 28 07:03:31 server1 kernel: [890070.995313] [<ffffffff8118f46e>] do_writepages+0x1e/0x30
Nov 28 07:03:31 server1 kernel: [890070.995316] [<ffffffff8122b215>] __writeback_single_inode+0x45/0x340
Nov 28 07:03:31 server1 kernel: [890070.995319] [<ffffffff8122ba4b>] writeback_sb_inodes+0x26b/0x5c0
Nov 28 07:03:31 server1 kernel: [890070.995322] [<ffffffff8122be26>] __writeback_inodes_wb+0x86/0xc0
Nov 28 07:03:31 server1 kernel: [890070.995325] [<ffffffff8122c0b2>] wb_writeback+0x252/0x2e0
Nov 28 07:03:31 server1 kernel: [890070.995328] [<ffffffff8122c818>] wb_workfn+0x238/0x3d0
Nov 28 07:03:31 server1 kernel: [890070.995332] [<ffffffff810c4e11>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Nov 28 07:03:31 server1 kernel: [890070.995337] [<ffffffff81095b40>] process_one_work+0x150/0x3f0
Nov 28 07:03:31 server1 kernel: [890070.995341] [<ffffffff810962ba>] worker_thread+0x11a/0x470
Nov 28 07:03:31 server1 kernel: [890070.995345] [<ffffffff817fa969>] ? __schedule+0x359/0x980
Nov 28 07:03:31 server1 kernel: [890070.995347] [<ffffffff810961a0>] ? rescuer_thread+0x310/0x310
Nov 28 07:03:31 server1 kernel: [890070.995349] [<ffffffff8109ba29>] kthread+0xc9/0xe0
Nov 28 07:03:31 server1 kernel: [890070.995351] [<ffffffff8109b960>] ? kthread_park+0x60/0x60
Nov 28 07:03:31 server1 kernel: [890070.995354] [<ffffffff817febcf>] ret_from_fork+0x3f/0x70
Nov 28 07:03:31 server1 kernel: [890070.995356] [<ffffffff8109b960>] ? kthread_park+0x60/0x60
Nov 28 07:03:31 server1 kernel: [890070.995361] INFO: task fsck.ext4:11082 blocked for more than 120 seconds.
Nov 28 07:03:31 server1 kernel: [890070.995433] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu
Nov 28 07:03:31 server1 kernel: [890070.995475] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 07:03:31 server1 kernel: [890070.995519] fsck.ext4 D ffff8800a7caf7d8 0 11082 11081 0x00000000
Nov 28 07:03:31 server1 kernel: [890070.995522] ffff8800a7caf7d8 ffff88013b000000 ffff8800b8463200 ffff8800a7cb0000
Nov 28 07:03:31 server1 kernel: [890070.995525] ffff88013a83cc18 ffff88013a83cc00 ffffffff00000000 fffffffe00000001
Nov 28 07:03:31 server1 kernel: [890070.995527] ffff8800a7caf7f0 ffffffff817fafc5 ffff8800b8463200 ffff8800a7caf870
Nov 28 07:03:31 server1 kernel: [890070.995529] Call Trace:
Nov 28 07:03:31 server1 kernel: [890070.995532] [<ffffffff817fafc5>] schedule+0x35/0x80
Nov 28 07:03:31 server1 kernel: [890070.995535] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320
Nov 28 07:03:31 server1 kernel: [890070.995537] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20
Nov 28 07:03:31 server1 kernel: [890070.995539] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40
Nov 28 07:03:31 server1 kernel: [890070.995543] [<ffffffffc0315d12>] snapshot_map+0x62/0x390 [dm_snapshot]
Nov 28 07:03:31 server1 kernel: [890070.995545] [<ffffffff8167f2da>] __map_bio+0x3a/0x110
Nov 28 07:03:31 server1 kernel: [890070.995547] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0
Nov 28 07:03:31 server1 kernel: [890070.995550] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0
Nov 28 07:03:31 server1 kernel: [890070.995553] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0
Nov 28 07:03:31 server1 kernel: [890070.995555] [<ffffffff813aa357>] submit_bio+0x77/0x150
Nov 28 07:03:31 server1 kernel: [890070.995558] [<ffffffff81191245>] ? release_pages+0xc5/0x260
Nov 28 07:03:31 server1 kernel: [890070.995560] [<ffffffff810c4e11>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Nov 28 07:03:31 server1 kernel: [890070.995563] [<ffffffff8123d3c9>] do_mpage_readpage+0x2d9/0x6d0
Nov 28 07:03:31 server1 kernel: [890070.995565] [<ffffffff81191ade>] ? lru_cache_add+0xe/0x10
Nov 28 07:03:31 server1 kernel: [890070.995567] [<ffffffff8123d8c3>] mpage_readpages+0x103/0x150
Nov 28 07:03:31 server1 kernel: [890070.995569] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20
Nov 28 07:03:31 server1 kernel: [890070.995571] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20
Nov 28 07:03:31 server1 kernel: [890070.995573] [<ffffffff8123733d>] blkdev_readpages+0x1d/0x20
Nov 28 07:03:31 server1 kernel: [890070.995575] [<ffffffff8118fb94>] __do_page_cache_readahead+0x174/0x200
Nov 28 07:03:31 server1 kernel: [890070.995577] [<ffffffff8118fd55>] ondemand_readahead+0x135/0x260
Nov 28 07:03:31 server1 kernel: [890070.995579] [<ffffffff8168033a>] ? dm_any_congested+0x4a/0x50
Nov 28 07:03:31 server1 kernel: [890070.995581] [<ffffffff8118feec>] page_cache_async_readahead+0x6c/0x70
Nov 28 07:03:31 server1 kernel: [890070.995584] [<ffffffff811846a0>] generic_file_read_iter+0x390/0x5c0
Nov 28 07:03:31 server1 kernel: [890070.995586] [<ffffffff812375e7>] blkdev_read_iter+0x37/0x40
Nov 28 07:03:31 server1 kernel: [890070.995589] [<ffffffff811fdc55>] new_sync_read+0x85/0xb0
Nov 28 07:03:31 server1 kernel: [890070.995591] [<ffffffff811fdca7>] __vfs_read+0x27/0x40
Nov 28 07:03:31 server1 kernel: [890070.995593] [<ffffffff811fe24f>] vfs_read+0x7f/0x130
Nov 28 07:03:31 server1 kernel: [890070.995595] [<ffffffff811ff026>] SyS_read+0x46/0xa0
Nov 28 07:03:31 server1 kernel: [890070.995597] [<ffffffff811fdf27>] ? SyS_lseek+0x87/0xb0
Nov 28 07:03:31 server1 kernel: [890070.995599] [<ffffffff817fe836>] entry_SYSCALL_64_fastpath+0x16/0x75
Nov 28 07:04:29 server4 kernel: [504268.435113] INFO: task blkback.44.xvda:8031 blocked for more than 120 seconds.
Nov 28 07:04:29 server4 kernel: [504268.435173] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu
Nov 28 07:04:29 server4 kernel: [504268.435233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 07:04:29 server4 kernel: [504268.435295] blkback.44.xvda D ffff8801709b7928 0 8031 2 0x00000000
Nov 28 07:04:29 server4 kernel: [504268.435334] ffff8801709b7928 ffff88017ccf8dc0 ffff880119c30000 ffff8801709b8000
Nov 28 07:04:29 server4 kernel: [504268.435339] ffff8800047c0c18 ffff8800047c0c00 ffffffff00000000 fffffffe00000001
Nov 28 07:04:29 server4 kernel: [504268.435343] ffff8801709b7940 ffffffff817fafc5 ffff880119c30000 ffff8801709b79c0
Nov 28 07:04:29 server4 kernel: [504268.435347] Call Trace:
Nov 28 07:04:29 server4 kernel: [504268.435358] [<ffffffff817fafc5>] schedule+0x35/0x80
Nov 28 07:04:29 server4 kernel: [504268.435362] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320
Nov 28 07:04:29 server4 kernel: [504268.435367] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20
Nov 28 07:04:29 server4 kernel: [504268.435369] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40
Nov 28 07:04:29 server4 kernel: [504268.435377] [<ffffffffc0397dae>] __origin_write+0x6e/0x210 [dm_snapshot]
Nov 28 07:04:29 server4 kernel: [504268.435382] [<ffffffff81185625>] ? mempool_alloc_slab+0x15/0x20
Nov 28 07:04:29 server4 kernel: [504268.435385] [<ffffffff8118574f>] ? mempool_alloc+0x5f/0x150
Nov 28 07:04:29 server4 kernel: [504268.435388] [<ffffffffc0397fb7>] do_origin.isra.14+0x67/0x90 [dm_snapshot]
Nov 28 07:04:29 server4 kernel: [504268.435391] [<ffffffffc0398042>] origin_map+0x62/0x80 [dm_snapshot]
Nov 28 07:04:29 server4 kernel: [504268.435395] [<ffffffff8167f2da>] __map_bio+0x3a/0x110
Nov 28 07:04:29 server4 kernel: [504268.435398] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0
Nov 28 07:04:29 server4 kernel: [504268.435401] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0
Nov 28 07:04:29 server4 kernel: [504268.435406] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0
Nov 28 07:04:29 server4 kernel: [504268.435409] [<ffffffff813aa357>] submit_bio+0x77/0x150
Nov 28 07:04:29 server4 kernel: [504268.435411] [<ffffffff813a1c71>] ? bio_alloc_bioset+0x181/0x2a0
Nov 28 07:04:29 server4 kernel: [504268.435417] [<ffffffffc03a8a1d>] dispatch_rw_block_io+0x4fd/0x9b0 [xen_blkback]
Nov 28 07:04:29 server4 kernel: [504268.435420] [<ffffffff8101c244>] ? xen_load_sp0+0x84/0x180
Nov 28 07:04:29 server4 kernel: [504268.435423] [<ffffffffc03a90c5>] __do_block_io_op+0x1f5/0x650 [xen_blkback]
Nov 28 07:04:29 server4 kernel: [504268.435427] [<ffffffff810e5e18>] ? del_timer_sync+0x48/0x50
Nov 28 07:04:29 server4 kernel: [504268.435429] [<ffffffff817fd8ab>] ? schedule_timeout+0x16b/0x2d0
Nov 28 07:04:29 server4 kernel: [504268.435432] [<ffffffffc03a9880>] xen_blkif_schedule+0xd0/0x820 [xen_blkback]
Nov 28 07:04:29 server4 kernel: [504268.435436] [<ffffffff810a4e1a>] ? finish_task_switch+0x7a/0x290
Nov 28 07:04:29 server4 kernel: [504268.435438] [<ffffffff817fa969>] ? __schedule+0x359/0x980
Nov 28 07:04:29 server4 kernel: [504268.435443] [<ffffffff810bde70>] ? prepare_to_wait_event+0xf0/0xf0
Nov 28 07:04:29 server4 kernel: [504268.435446] [<ffffffffc03a97b0>] ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
Nov 28 07:04:29 server4 kernel: [504268.435449] [<ffffffff8109ba29>] kthread+0xc9/0xe0
Nov 28 07:04:29 server4 kernel: [504268.435451] [<ffffffff8109b960>] ? kthread_park+0x60/0x60
Nov 28 07:04:29 server4 kernel: [504268.435455] [<ffffffff817febcf>] ret_from_fork+0x3f/0x70
Nov 28 07:04:29 server4 kernel: [504268.435457] [<ffffffff8109b960>] ? kthread_park+0x60/0x60
Nov 28 07:04:29 server4 kernel: [504268.435473] INFO: task kworker/u4:2:10156 blocked for more than 120 seconds.
Nov 28 07:04:29 server4 kernel: [504268.435523] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu
Nov 28 07:04:29 server4 kernel: [504268.435568] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 07:04:29 server4 kernel: [504268.435617] kworker/u4:2 D ffff880160633628 0 10156 2 0x00000000
Nov 28 07:04:29 server4 kernel: [504268.435625] Workqueue: writeback wb_workfn (flush-252:36)
Nov 28 07:04:29 server4 kernel: [504268.435627] ffff880160633628 ffff880007366040 ffff880076470dc0 ffff880160634000
Nov 28 07:04:29 server4 kernel: [504268.435630] ffff8800047c0c18 ffff8800047c0c00 ffffffff00000000 fffffffe00000001
Nov 28 07:04:29 server4 kernel: [504268.435632] ffff880160633640 ffffffff817fafc5 ffff880076470dc0 ffff8801606336c0
Nov 28 07:04:29 server4 kernel: [504268.435635] Call Trace:
Nov 28 07:04:29 server4 kernel: [504268.435637] [<ffffffff817fafc5>] schedule+0x35/0x80
Nov 28 07:04:29 server4 kernel: [504268.435640] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320
Nov 28 07:04:29 server4 kernel: [504268.435642] [<ffffffff813a1a5c>] ? bvec_alloc+0x5c/0xf0
Nov 28 07:04:29 server4 kernel: [504268.435645] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20
Nov 28 07:04:29 server4 kernel: [504268.435647] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40
Nov 28 07:04:29 server4 kernel: [504268.435650] [<ffffffffc0398d12>] snapshot_map+0x62/0x390 [dm_snapshot]
Nov 28 07:04:29 server4 kernel: [504268.435653] [<ffffffff8167f2da>] __map_bio+0x3a/0x110
Nov 28 07:04:29 server4 kernel: [504268.435656] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0
Nov 28 07:04:29 server4 kernel: [504268.435658] [<ffffffff810c4e11>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Nov 28 07:04:29 server4 kernel: [504268.435661] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0
Nov 28 07:04:29 server4 kernel: [504268.435664] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0
Nov 28 07:04:29 server4 kernel: [504268.435667] [<ffffffff813aa357>] submit_bio+0x77/0x150
Nov 28 07:04:29 server4 kernel: [504268.435669] [<ffffffff813a1c71>] ? bio_alloc_bioset+0x181/0x2a0
Nov 28 07:04:29 server4 kernel: [504268.435673] [<ffffffff812347ef>] submit_bh_wbc+0x12f/0x160
Nov 28 07:04:29 server4 kernel: [504268.435675] [<ffffffff81236615>] __block_write_full_page.constprop.39+0x125/0x360
Nov 28 07:04:29 server4 kernel: [504268.435677] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20
Nov 28 07:04:29 server4 kernel: [504268.435679] [<ffffffff8123692e>] block_write_full_page+0xde/0x100
Nov 28 07:04:29 server4 kernel: [504268.435682] [<ffffffff81237378>] blkdev_writepage+0x18/0x20
Nov 28 07:04:29 server4 kernel: [504268.435684] [<ffffffff8118d253>] __writepage+0x13/0x40
Nov 28 07:04:29 server4 kernel: [504268.435687] [<ffffffff8118e5c1>] write_cache_pages+0x241/0x4c0
Nov 28 07:04:29 server4 kernel: [504268.435689] [<ffffffff8118d240>] ? wb_update_dirty_ratelimit+0x1c0/0x1c0
Nov 28 07:04:29 server4 kernel: [504268.435692] [<ffffffff817fe23a>] ? _raw_spin_unlock_irqrestore+0x1a/0x20
Nov 28 07:04:29 server4 kernel: [504268.435701] [<ffffffffc0126226>] ? _base_get_chain_buffer_tracker+0x86/0xd0 [mpt3sas]
Nov 28 07:04:29 server4 kernel: [504268.435703] [<ffffffff8118e883>] generic_writepages+0x43/0x60
Nov 28 07:04:29 server4 kernel: [504268.435706] [<ffffffff8118f46e>] do_writepages+0x1e/0x30
Nov 28 07:04:29 server4 kernel: [504268.435708] [<ffffffff8122b215>] __writeback_single_inode+0x45/0x340
Nov 28 07:04:29 server4 kernel: [504268.435710] [<ffffffff8122ba4b>] writeback_sb_inodes+0x26b/0x5c0
Nov 28 07:04:29 server4 kernel: [504268.435713] [<ffffffff8122be26>] __writeback_inodes_wb+0x86/0xc0
Nov 28 07:04:29 server4 kernel: [504268.435715] [<ffffffff8122c0b2>] wb_writeback+0x252/0x2e0
Nov 28 07:04:29 server4 kernel: [504268.435717] [<ffffffff8122c8a2>] wb_workfn+0x2c2/0x3d0
Nov 28 07:04:29 server4 kernel: [504268.435719] [<ffffffff810c4e11>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Nov 28 07:04:29 server4 kernel: [504268.435724] [<ffffffff81095b40>] process_one_work+0x150/0x3f0
Nov 28 07:04:29 server4 kernel: [504268.435727] [<ffffffff810962ba>] worker_thread+0x11a/0x470
Nov 28 07:04:29 server4 kernel: [504268.435729] [<ffffffff817fa969>] ? __schedule+0x359/0x980
Nov 28 07:04:29 server4 kernel: [504268.435732] [<ffffffff810961a0>] ? rescuer_thread+0x310/0x310
Nov 28 07:04:29 server4 kernel: [504268.435734] [<ffffffff8109ba29>] kthread+0xc9/0xe0
Nov 28 07:04:29 server4 kernel: [504268.435736] [<ffffffff8109b960>] ? kthread_park+0x60/0x60
Nov 28 07:04:29 server4 kernel: [504268.435739] [<ffffffff817febcf>] ret_from_fork+0x3f/0x70
Nov 28 07:04:29 server4 kernel: [504268.435741] [<ffffffff8109b960>] ? kthread_park+0x60/0x60
Nov 28 07:04:29 server4 kernel: [504268.435750] INFO: task fsck.ext4:17037 blocked for more than 120 seconds.
Nov 28 07:04:29 server4 kernel: [504268.435797] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu
Nov 28 07:04:29 server4 kernel: [504268.435842] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 07:04:29 server4 kernel: [504268.435890] fsck.ext4 D ffff88016b6a7888 0 17037 17035 0x00000000
Nov 28 07:04:29 server4 kernel: [504268.435893] ffff88016b6a7888 ffff880004569b80 ffff880010cb1b80 ffff88016b6a8000
Nov 28 07:04:29 server4 kernel: [504268.435895] ffff8800047c0c18 ffff8800047c0c00 ffffffff00000000 fffffffe00000001
Nov 28 07:04:29 server4 kernel: [504268.435898] ffff88016b6a78a0 ffffffff817fafc5 ffff880010cb1b80 ffff88016b6a7918
Nov 28 07:04:29 server4 kernel: [504268.435901] Call Trace:
Nov 28 07:04:29 server4 kernel: [504268.435903] [<ffffffff817fafc5>] schedule+0x35/0x80
Nov 28 07:04:29 server4 kernel: [504268.435905] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320
Nov 28 07:04:29 server4 kernel: [504268.435908] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20
Nov 28 07:04:29 server4 kernel: [504268.435910] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40
Nov 28 07:04:29 server4 kernel: [504268.435913] [<ffffffffc0398d12>] snapshot_map+0x62/0x390 [dm_snapshot]
Nov 28 07:04:29 server4 kernel: [504268.435916] [<ffffffff8167f2da>] __map_bio+0x3a/0x110
Nov 28 07:04:29 server4 kernel: [504268.435919] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0
Nov 28 07:04:29 server4 kernel: [504268.435921] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0
Nov 28 07:04:29 server4 kernel: [504268.435924] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0
Nov 28 07:04:29 server4 kernel: [504268.435927] [<ffffffff813aa357>] submit_bio+0x77/0x150
Nov 28 07:04:29 server4 kernel: [504268.435930] [<ffffffff8123cf6a>] mpage_bio_submit+0x2a/0x40
Nov 28 07:04:29 server4 kernel: [504268.435932] [<ffffffff8123d8f4>] mpage_readpages+0x134/0x150
Nov 28 07:04:29 server4 kernel: [504268.435934] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20
Nov 28 07:04:29 server4 kernel: [504268.435936] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20
Nov 28 07:04:29 server4 kernel: [504268.435938] [<ffffffff8123733d>] blkdev_readpages+0x1d/0x20
Nov 28 07:04:29 server4 kernel: [504268.435940] [<ffffffff8118fb94>] __do_page_cache_readahead+0x174/0x200
Nov 28 07:04:29 server4 kernel: [504268.435943] [<ffffffff8118fd55>] ondemand_readahead+0x135/0x260
Nov 28 07:04:29 server4 kernel: [504268.435945] [<ffffffff8168033a>] ? dm_any_congested+0x4a/0x50
Nov 28 07:04:29 server4 kernel: [504268.435948] [<ffffffff8118feec>] page_cache_async_readahead+0x6c/0x70
Nov 28 07:04:29 server4 kernel: [504268.435951] [<ffffffff811846a0>] generic_file_read_iter+0x390/0x5c0
Nov 28 07:04:29 server4 kernel: [504268.435953] [<ffffffff812375e7>] blkdev_read_iter+0x37/0x40
Nov 28 07:04:29 server4 kernel: [504268.435957] [<ffffffff811fdc55>] new_sync_read+0x85/0xb0
Nov 28 07:04:29 server4 kernel: [504268.435959] [<ffffffff811fdca7>] __vfs_read+0x27/0x40
Nov 28 07:04:29 server4 kernel: [504268.435961] [<ffffffff811fe24f>] vfs_read+0x7f/0x130
Nov 28 07:04:29 server4 kernel: [504268.435964] [<ffffffff811ff026>] SyS_read+0x46/0xa0
Nov 28 07:04:29 server4 kernel: [504268.435966] [<ffffffff811fdf27>] ? SyS_lseek+0x87/0xb0
Nov 28 07:04:29 server4 kernel: [504268.435969] [<ffffffff817fe836>] entry_SYSCALL_64_fastpath+0x16/0x75 |
Server's load become high with tasks in D state, no choice but to reboot the system.
Could be related to the following ? :
https://bugzilla.kernel.org/show_bug.cgi?id=119841
https://www.redhat.com/archives/dm-devel/2016-June/msg00399.html
https://patchwork.kernel.org/patch/9223697/
https://xen.crc.id.au/bugs/view.php?id=75
ii xen-hypervisor-4.4-amd64 4.4.2-0ubuntu0.14.04.7 amd64 Xen Hypervisor on AMD64
ii linux-image-extra-4.4.0-47-generic 4.4.0-47.68~14.04.1 amd64 Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii linux-image-4.4.0-47-generic 4.4.0-47.68~14.04.1 amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
kernel messages:
----------------
[890070.994700] INFO: task blkback.3.xvda2:5756 blocked for more than 120 seconds.
[890070.994758] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu
[890070.994806] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[890070.994884] blkback.3.xvda2 D ffff8800b7ff3928 0 5756 2 0x00000000
[890070.994890] ffff8800b7ff3928 ffffffff81e13500 ffff8800b6a4be80 ffff8800b7ff4000
[890070.994895] ffff88013a83cc18 ffff88013a83cc00 ffffffff00000000 fffffffe00000001
[890070.994898] ffff8800b7ff3940 ffffffff817fafc5 ffff8800b6a4be80 ffff8800b7ff39c0
[890070.994902] Call Trace:
[890070.994912] [<ffffffff817fafc5>] schedule+0x35/0x80
[890070.994917] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320
[890070.994923] [<ffffffff81689577>] ? push+0x47/0x50
[890070.994927] [<ffffffff81689f07>] ? dm_kcopyd_copy+0x147/0x1f0
[890070.994931] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20
[890070.994933] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40
[890070.994939] [<ffffffffc0314dae>] __origin_write+0x6e/0x210 [dm_snapshot]
[890070.994944] [<ffffffff81185625>] ? mempool_alloc_slab+0x15/0x20
[890070.994946] [<ffffffff8118574f>] ? mempool_alloc+0x5f/0x150
[890070.994949] [<ffffffffc0314fb7>] do_origin.isra.14+0x67/0x90 [dm_snapshot]
[890070.994952] [<ffffffffc0315042>] origin_map+0x62/0x80 [dm_snapshot]
[890070.994955] [<ffffffff8167f2da>] __map_bio+0x3a/0x110
[890070.994957] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0
[890070.994960] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0
[890070.994964] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0
[890070.994968] [<ffffffff813aa357>] submit_bio+0x77/0x150
[890070.994971] [<ffffffff813a1c71>] ? bio_alloc_bioset+0x181/0x2a0
[890070.994977] [<ffffffffc02c6a1d>] dispatch_rw_block_io+0x4fd/0x9b0 [xen_blkback]
[890070.994981] [<ffffffff8101c244>] ? xen_load_sp0+0x84/0x180
[890070.994985] [<ffffffffc02c70c5>] __do_block_io_op+0x1f5/0x650 [xen_blkback]
[890070.994990] [<ffffffff810e5e18>] ? del_timer_sync+0x48/0x50
[890070.994993] [<ffffffff817fd8ab>] ? schedule_timeout+0x16b/0x2d0
[890070.994997] [<ffffffffc02c7880>] xen_blkif_schedule+0xd0/0x820 [xen_blkback]
[890070.995002] [<ffffffff810a4e1a>] ? finish_task_switch+0x7a/0x290
[890070.995004] [<ffffffff817fa969>] ? __schedule+0x359/0x980
[890070.995010] [<ffffffff810bde70>] ? prepare_to_wait_event+0xf0/0xf0
[890070.995014] [<ffffffffc02c77b0>] ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
[890070.995018] [<ffffffff8109ba29>] kthread+0xc9/0xe0
[890070.995021] [<ffffffff8109b960>] ? kthread_park+0x60/0x60
[890070.995025] [<ffffffff817febcf>] ret_from_fork+0x3f/0x70
[890070.995027] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 |
|
| 2016-11-29 01:58:32 |
john |
bug watch added |
|
http://bugzilla.kernel.org/show_bug.cgi?id=119841 |
|
| 2016-11-29 01:58:32 |
john |
attachment added |
|
fullstack.txt.gz https://bugs.launchpad.net/ubuntu/+source/linux-lts-xenial/+bug/1645187/+attachment/4784464/+files/fullstack.txt.gz |
|
| 2016-11-29 13:36:12 |
Stefan Bader |
bug |
|
|
added subscriber Stefan Bader |
| 2016-11-29 17:51:14 |
Joseph Salisbury |
affects |
linux-lts-xenial (Ubuntu) |
linux (Ubuntu) |
|
| 2016-11-29 17:51:14 |
Joseph Salisbury |
linux (Ubuntu): importance |
Undecided |
Medium |
|
| 2016-11-29 17:51:14 |
Joseph Salisbury |
linux (Ubuntu): status |
New |
Triaged |
|
| 2016-11-29 17:51:42 |
Joseph Salisbury |
linux (Ubuntu): importance |
Medium |
High |
|
| 2016-11-29 17:52:05 |
Joseph Salisbury |
nominated for series |
|
Ubuntu Xenial |
|
| 2016-11-29 17:52:05 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Xenial) |
|
| 2016-11-29 17:52:13 |
Joseph Salisbury |
linux (Ubuntu Xenial): status |
New |
Triaged |
|
| 2016-11-29 17:52:16 |
Joseph Salisbury |
linux (Ubuntu Xenial): importance |
Undecided |
High |
|
| 2016-11-29 17:52:23 |
Joseph Salisbury |
tags |
|
kernel-da-key xenial |
|
| 2017-03-31 03:59:14 |
john |
cve linked |
|
2017-7184 |
|
