In Ubuntu17.04 as Kvm guest : While trigger kdump console hung having call traces

[bugproxy]

In ubuntu17.04 as KVM guest on ubuntu KVM Host and trying kdump on guest while kdump process console got hung having call traces

Reproducible Step:

1- Install Ubuntu17.04 as kvm guest on ubuntu kvm host
2- configure kdump
3- trigger kdump

Expected Result :

Kdump should capture

Actual Result :

Kdump console hung having continuous call traces

LOG:

[ 0.488534] Freeing unused kernel memory: 4416K (c000000008e80000 - c0000000092d0000)
[ 0.488725] This architecture does not have kernel memory protection.
Loading, please wait...
starting version 232
[ 0.501616] random: udevadm: uninitialized urandom read (16 bytes read)
[ 0.501830] random: udevadm: uninitialized urandom read (16 bytes read)
[ 0.501981] random: udevadm: uninitialized urandom read (16 bytes read)
[ 0.502162] random: udevadm: uninitialized urandom read (16 bytes read)
[ 0.502254] random: udevadm: uninitialized urandom read (16 bytes read)
[ 0.502433] random: udevadm: uninitialized urandom read (16 bytes read)
[ 0.503188] random: udevadm: uninitialized urandom read (16 bytes read)
[ 0.503270] random: udevadm: uninitialized urandom read (16 bytes read)
[ 0.503978] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[ 0.504218] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[ 242.663388] INFO: task systemd-udevd:151 blocked for more than 120 seconds.
[ 242.663514] Not tainted 4.9.0-15-generic #16-Ubuntu
[ 242.663553] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 242.663755] systemd-udevd D 0 151 145 0x00040002
[ 242.663795] Call Trace:
[ 242.663868] [c00000001fe4ecd0] [c00000000801c3a0] __switch_to+0x2e0/0x4c0
[ 242.663964] [c00000001fe4ed30] [c000000008b19398] __schedule+0x2f8/0x990
[ 242.664076] [c00000001fe4ee10] [c000000008b19a78] schedule+0x48/0xc0
[ 242.664179] [c00000001fe4ee40] [c000000008b1de54] schedule_timeout+0x274/0x470
[ 242.664334] [c00000001fe4ef30] [c000000008b19010] io_schedule_timeout+0xd0/0x160
[ 242.664502] [c00000001fe4ef80] [c000000008b1a720] bit_wait_io+0x30/0x90
[ 242.664618] [c00000001fe4efb0] [c000000008b1a168] __wait_on_bit+0xf8/0x170
[ 242.664754] [c00000001fe4f000] [c00000000824b238] wait_on_page_bit+0x98/0xb0
[ 242.664847] [c00000001fe4f060] [c00000000824d60c] do_read_cache_page+0x21c/0x4e0
[ 242.665008] [c00000001fe4f0d0] [c00000000859be78] read_dev_sector+0xb8/0x140
[ 242.665126] [c00000001fe4f100] [c0000000085a5d88] read_lba.isra.0+0x148/0x250
[ 242.665259] [c00000001fe4f170] [c0000000085a652c] efi_partition+0x12c/0x830
[ 242.665363] [c00000001fe4f2e0] [c00000000859e768] check_partition+0x158/0x2d0
[ 242.665469] [c00000001fe4f360] [c00000000859c760] rescan_partitions+0xe0/0x390
[ 242.665552] [c00000001fe4f430] [c000000008371828] __blkdev_get+0x358/0x490
[ 242.665669] [c00000001fe4f4a0] [c000000008372b50] blkdev_get+0x1a0/0x4a0
[ 242.665784] [c00000001fe4f550] [c000000008599538] device_add_disk+0x4a8/0x500
[ 242.665894] [c00000001fe4f600] [d000000000511cc8] virtblk_probe+0x560/0x928 [virtio_blk]
[ 242.665983] [c00000001fe4f6c0] [c000000008687700] virtio_dev_probe+0x1d0/0x350
[ 242.666050] [c00000001fe4f700] [c000000008716f30] driver_probe_device+0x240/0x540
[ 242.666116] [c00000001fe4f790] [c00000000871738c] __driver_attach+0x15c/0x160
[ 242.666174] [c00000001fe4f810] [c0000000087138ec] bus_for_each_dev+0x8c/0xf0
[ 242.666232] [c00000001fe4f860] [c0000000087162e4] driver_attach+0x34/0x50
[ 242.666289] [c00000001fe4f880] [c000000008715a78] bus_add_driver+0x238/0x380
[ 242.666345] [c00000001fe4f910] [c00000000871829c] driver_register+0x9c/0x180
[ 242.666403] [c00000001fe4f980] [c000000008686abc] register_virtio_driver+0x4c/0x60
[ 242.666470] [c00000001fe4f9a0] [d000000000512114] init+0x84/0xd4 [virtio_blk]
[ 242.666527] [c00000001fe4fa10] [c00000000800dde8] do_one_initcall+0x68/0x1d0
[ 242.666584] [c00000001fe4fad0] [c000000008b28e00] do_init_module+0x90/0x244
[ 242.666643] [c00000001fe4fb60] [c000000008184794] load_module+0x1614/0x17a0
[ 242.666701] [c00000001fe4fd30] [c000000008184c60] SyS_finit_module+0xf0/0x170
[ 242.666759] [c00000001fe4fe30] [c00000000800bd84] system_call+0x38/0xe0
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------

[ 363.701100] md: raid6 personality registered for level 6
[ 363.701169] md: raid5 personality registered for level 5
[ 363.701200] md: raid4 personality registered for level 4
[ 363.713493] md: raid10 personality registered for level 10
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
[ 484.327414] INFO: task systemd-udevd:151 blocked for more than 120 seconds.
[ 484.327505] Not tainted 4.9.0-15-generic #16-Ubuntu
[ 484.327557] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 484.327648] systemd-udevd D 0 151 145 0x00040002
[ 484.327704] Call Trace:
[ 484.327741] [c00000001fe4ecd0] [c00000000801c3a0] __switch_to+0x2e0/0x4c0
[ 484.327824] [c00000001fe4ed30] [c000000008b19398] __schedule+0x2f8/0x990
[ 484.327907] [c00000001fe4ee10] [c000000008b19a78] schedule+0x48/0xc0
[ 484.328015] [c00000001fe4ee40] [c000000008b1de54] schedule_timeout+0x274/0x470
[ 484.328121] [c00000001fe4ef30] [c000000008b19010] io_schedule_timeout+0xd0/0x160
[ 484.328222] [c00000001fe4ef80] [c000000008b1a720] bit_wait_io+0x30/0x90
[ 484.328304] [c00000001fe4efb0] [c000000008b1a168] __wait_on_bit+0xf8/0x170
[ 484.328386] [c00000001fe4f000] [c00000000824b238] wait_on_page_bit+0x98/0xb0
[ 484.328470] [c00000001fe4f060] [c00000000824d60c] do_read_cache_page+0x21c/0x4e0
[ 484.328566] [c00000001fe4f0d0] [c00000000859be78] read_dev_sector+0xb8/0x140
[ 484.328648] [c00000001fe4f100] [c0000000085a5d88] read_lba.isra.0+0x148/0x250
[ 484.328731] [c00000001fe4f170] [c0000000085a652c] efi_partition+0x12c/0x830
[ 484.328812] [c00000001fe4f2e0] [c00000000859e768] check_partition+0x158/0x2d0
[ 484.328919] [c00000001fe4f360] [c00000000859c760] rescan_partitions+0xe0/0x390
[ 484.329017] [c00000001fe4f430] [c000000008371828] __blkdev_get+0x358/0x490
[ 484.329099] [c00000001fe4f4a0] [c000000008372b50] blkdev_get+0x1a0/0x4a0
[ 484.329181] [c00000001fe4f550] [c000000008599538] device_add_disk+0x4a8/0x500
[ 484.329266] [c00000001fe4f600] [d000000000511cc8] virtblk_probe+0x560/0x928 [virtio_blk]
[ 484.329361] [c00000001fe4f6c0] [c000000008687700] virtio_dev_probe+0x1d0/0x350
[ 484.329458] [c00000001fe4f700] [c000000008716f30] driver_probe_device+0x240/0x540
[ 484.329551] [c00000001fe4f790] [c00000000871738c] __driver_attach+0x15c/0x160
[ 484.329634] [c00000001fe4f810] [c0000000087138ec] bus_for_each_dev+0x8c/0xf0
[ 484.329715] [c00000001fe4f860] [c0000000087162e4] driver_attach+0x34/0x50
[ 484.329796] [c00000001fe4f880] [c000000008715a78] bus_add_driver+0x238/0x380
[ 484.329876] [c00000001fe4f910] [c00000000871829c] driver_register+0x9c/0x180
[ 484.329957] [c00000001fe4f980] [c000000008686abc] register_virtio_driver+0x4c/0x60
[ 484.330052] [c00000001fe4f9a0] [d000000000512114] init+0x84/0xd4 [virtio_blk]
[ 484.330135] [c00000001fe4fa10] [c00000000800dde8] do_one_initcall+0x68/0x1d0
[ 484.330215] [c00000001fe4fad0] [c000000008b28e00] do_init_module+0x90/0x244
[ 484.330298] [c00000001fe4fb60] [c000000008184794] load_module+0x1614/0x17a0
[ 484.330380] [c00000001fe4fd30] [c000000008184c60] SyS_finit_module+0xf0/0x170
[ 484.330463] [c00000001fe4fe30] [c00000000800bd84] system_call+0x38/0xe0

root@ubuntu:~# uname -a
Linux ubuntu 4.9.0-15-generic #16-Ubuntu SMP Fri Jan 20 15:28:49 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
root@ubuntu:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-4.9.0-15-generic root=UUID=567520d7-270f-462e-9f0c-882c73bce0bc ro crashkernel=512M
root@ubuntu:~#
root@ubuntu:~# echo c > /proc/sysrq-trigger

Regards
Praveen

> Resolution: Pass "noirqdistrib" to kdump kernel.
>
> Thanks

wiki page at:

https://wiki.ubuntu.com/ppc64el/Recommendations#Kdump_not_generating_crash_dump_file

mirror this bug to Canonical for them to ensure "noirqdistrib" is passed in powerpc arch case..

[Michael Hohnbaum (hohnbaum)]

Leann,

This looks like an update for the Kernel team.

                     Michael

On 02/14/2017 04:09 AM, Launchpad Bug Tracker wrote:
> bugproxy (bugproxy) has assigned this bug to you for Ubuntu:
>
> In ubuntu17.04 as KVM guest on ubuntu KVM Host and trying kdump on guest
> while kdump process console got hung having call traces
>
>
> Reproducible Step:
>
> 1- Install Ubuntu17.04 as kvm guest on ubuntu kvm host
> 2- configure kdump
> 3- trigger kdump
>
> Expected Result :
>
> Kdump should capture
>
> Actual Result :
>
> Kdump console hung having continuous call traces
>
> LOG:
>
> [ 0.488534] Freeing unused kernel memory: 4416K (c000000008e80000 - c0000000092d0000)
> [ 0.488725] This architecture does not have kernel memory protection.
> Loading, please wait...
> starting version 232
> [ 0.501616] random: udevadm: uninitialized urandom read (16 bytes read)
> [ 0.501830] random: udevadm: uninitialized urandom read (16 bytes read)
> [ 0.501981] random: udevadm: uninitialized urandom read (16 bytes read)
> [ 0.502162] random: udevadm: uninitialized urandom read (16 bytes read)
> [ 0.502254] random: udevadm: uninitialized urandom read (16 bytes read)
> [ 0.502433] random: udevadm: uninitialized urandom read (16 bytes read)
> [ 0.503188] random: udevadm: uninitialized urandom read (16 bytes read)
> [ 0.503270] random: udevadm: uninitialized urandom read (16 bytes read)
> [ 0.503978] random: systemd-udevd: uninitialized urandom read (16 bytes read)
> [ 0.504218] random: systemd-udevd: uninitialized urandom read (16 bytes read)
> [ 242.663388] INFO: task systemd-udevd:151 blocked for more than 120 seconds.
> [ 242.663514] Not tainted 4.9.0-15-generic #16-Ubuntu
> [ 242.663553] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 242.663755] systemd-udevd D 0 151 145 0x00040002
> [ 242.663795] Call Trace:
> [ 242.663868] [c00000001fe4ecd0] [c00000000801c3a0] __switch_to+0x2e0/0x4c0
> [ 242.663964] [c00000001fe4ed30] [c000000008b19398] __schedule+0x2f8/0x990
> [ 242.664076] [c00000001fe4ee10] [c000000008b19a78] schedule+0x48/0xc0
> [ 242.664179] [c00000001fe4ee40] [c000000008b1de54] schedule_timeout+0x274/0x470
> [ 242.664334] [c00000001fe4ef30] [c000000008b19010] io_schedule_timeout+0xd0/0x160
> [ 242.664502] [c00000001fe4ef80] [c000000008b1a720] bit_wait_io+0x30/0x90
> [ 242.664618] [c00000001fe4efb0] [c000000008b1a168] __wait_on_bit+0xf8/0x170
> [ 242.664754] [c00000001fe4f000] [c00000000824b238] wait_on_page_bit+0x98/0xb0
> [ 242.664847] [c00000001fe4f060] [c00000000824d60c] do_read_cache_page+0x21c/0x4e0
> [ 242.665008] [c00000001fe4f0d0] [c00000000859be78] read_dev_sector+0xb8/0x140
> [ 242.665126] [c00000001fe4f100] [c0000000085a5d88] read_lba.isra.0+0x148/0x250
> [ 242.665259] [c00000001fe4f170] [c0000000085a652c] efi_partition+0x12c/0x830
> [ 242.665363] [c00000001fe4f2e0] [c00000000859e768] check_partition+0x158/0x2d0
> [ 242.665469] [c00000001fe4f360] [c00000000859c760] rescan_partitions+0xe0/0x390
> [ 242.665552] [c00000001fe4f430] [c000000008371828] __blkdev_get+0x358/0x490
> [ 242.665669] [c00000001fe4f4a0] [c000000008372b50] blk...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-05-22 05:24 EDT-------
passing noirqdistrib in kdump kernel seems me issue is fixed and this info is added in (16.04)
https://wiki.ubuntu.com/ppc64el/Recommendations#Kdump_not_generating_crash_dump_file

Does the same kind of documentation available for 17.04 ?

[Christian Ehrhardt  (paelzer)]

I remember the kernel Team talking about that recently, adding a kernel task for them to be notified, to fill in what might be already known.

[Joseph Salisbury (jsalisbury)]

bug 1655280 may be related.

[Thadeu Lima de Souza Cascardo (cascardo)]

If you add the following line to /etc/default/kdump-tools:

KDUMP_CMDLINE_APPEND="noirqdistrib irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service ata_piix.prefer_ms_hyperv=0"

and run:

kdump-config unload ; kdump-config load

does it fix it?

Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-09-01 11:38 EDT-------
(In reply to comment #25)
> bug 1655280 may be related.
>
> If you add the following line to /etc/default/kdump-tools:
>
> KDUMP_CMDLINE_APPEND="noirqdistrib irqpoll nr_cpus=1 nousb
> systemd.unit=kdump-tools.service ata_piix.prefer_ms_hyperv=0"
>
> and run:
>
> kdump-config unload ; kdump-config load
>
> does it fix it?
>

Hello Cascardo, passing "noirqdistrib" helps..

Thanks
Hari

[Manoj Iyer (manjo)]

IBM, is the proposed work around of setting "noirqdistrib" acceptable ? If so could we close this bug?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-10-09 01:19 EDT-------
The workaround has been tested, and it is OK to close this bug.

[Andrew Cloke (andrew-cloke)]

Following comment #8, closing bug.