ISST-LTE:KVM:Ubuntu1804:BostonLC:boslcp3: cpu hotplug on boslcp3g4 guest dumping call traces continuously.

Bug #1759722 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Fix Released
Medium
Canonical Kernel Team
linux (Ubuntu)
Fix Released
Undecided
Ubuntu on IBM Power Systems Bug Triage

Bug Description

Problem Description:
===================
After performing HOTPLUG CPU on guest boslcp3g4 , saw dumping messages on the guest console continously

Steps to re-create:
==================

> boslcp3g4 is a guest of Boston LC system "boslcp3". Both of these systems are installed with ubuntu1804 kernel.

root@boslcp3:/home# uname -a
Linux boslcp3 4.15.0-12-generic #13+leo20180320 SMP Tue Mar 20 13:10:42 CDT 2018 ppc64le ppc64le ppc64le GNU/Linux
root@boslcp3:/home# uname -r
4.15.0-12-generic

root@boslcp3g4:/kte/tools/setup.d# uname -a
Linux boslcp3g4 4.15.0-12-generic #13+leo20180320 SMP Tue Mar 20 13:10:42 CDT 2018 ppc64le ppc64le ppc64le GNU/Linux
root@boslcp3g4:/kte/tools/setup.d# uname -r
4.15.0-12-generic

> When I perform hotplug of the cpu , it was successful but throwing these call traces in the Guest console continuously-

root@boslcp3:~# virsh setvcpus boslcp3g4 64 --live

root@boslcp3g4:~# [22273.371233] Using 'aes-generic' as fallback implementation.
[22273.371377] Using 'ctr(aes-generic)' as fallback implementation.
[22273.450867] Using 'aes-generic' as fallback implementation.
[22273.450967] Using 'ctr(aes-generic)' as fallback implementation.
[22300.779703] Using 'aes-generic' as fallback implementation.
[22300.779810] Using 'ctr(aes-generic)' as fallback implementation.
[22300.789192] Using 'aes-generic' as fallback implementation.
[22300.789327] Using 'ctr(aes-generic)' as fallback implementation.
[22327.714322] Using 'aes-generic' as fallback implementation.
[22327.714423] Using 'ctr(aes-generic)' as fallback implementation.
[22327.722548] Using 'aes-generic' as fallback implementation.
[22327.722654] Using 'ctr(aes-generic)' as fallback implementation.
[22354.401415] Using 'aes-generic' as fallback implementation.
[22354.401546] Using 'ctr(aes-generic)' as fallback implementation.
[22354.422227] Using 'aes-generic' as fallback implementation.
[22354.422354] Using 'ctr(aes-generic)' as fallback implementation.
[22368.150663] WARNING: CPU: 34 PID: 40114 at /home/user/ubuntu-bionic/block/blk-mq.c:1211 __blk_mq_run_hw_queue+0xf0/0x140
[22368.153463] Modules linked in: sctp esp4 xfrm4_mode_transport ccm xfrm_user xfrm_algo minix nls_iso8859_1 xfs rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss parport_pc nfs_acl ppdev lockd grace sunrpc lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_blk virtio_scsi virtio_net
[22368.154166] CPU: 34 PID: 40114 Comm: kworker/34:0H Not tainted 4.15.0-12-generic #13+leo20180320
[22368.154241] Workqueue: kblockd blk_mq_run_work_fn
[22368.154283] NIP: c00000000068e0f0 LR: c000000000131538 CTR: c00000000068e140
[22368.154344] REGS: c000000145a2f9d0 TRAP: 0700 Not tainted (4.15.0-12-generic)
[22368.154402] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 42002424 XER: 00000000
[22368.154480] CFAR: c00000000068e070 SOFTE: 1
[22368.154480] GPR00: c000000000131538 c000000145a2fc50 c0000000016eaf00 c0000001346f7800
[22368.154480] GPR04: c0000001346f7880 0000000000000000 0000000000000000 c000000001721ed8
[22368.154480] GPR08: 0000000000000000 0000000000000001 0000000000000000 c000000145985d30
[22368.154480] GPR12: c00000000068e140 c00000000fa97600 c00000000013a368 c0000001760c35c0
[22368.154480] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[22368.154480] GPR20: c00000017ee78180 0000000000000000 0000000000000000 fffffffffffffef7
[22368.154480] GPR24: 0000000000000402 0000000000000000 c00000017ee78500 0000000000000000
[22368.154480] GPR28: c0000001346f7880 c00000017ee7c900 c00000017ee78180 c0000001346f7800
[22368.155001] NIP [c00000000068e0f0] __blk_mq_run_hw_queue+0xf0/0x140
[22368.155065] LR [c000000000131538] process_one_work+0x298/0x5a0
[22368.155115] Call Trace:
[22368.155139] [c000000145a2fc50] [c000000145a2fce0] 0xc000000145a2fce0 (unreliable)
[22368.155200] [c000000145a2fc90] [c000000000131538] process_one_work+0x298/0x5a0
[22368.155259] [c000000145a2fd20] [c0000000001318d8] worker_thread+0x98/0x630
[22368.155308] [c000000145a2fdc0] [c00000000013a508] kthread+0x1a8/0x1b0
[22368.155358] [c000000145a2fe30] [c00000000000b528] ret_from_kernel_thread+0x5c/0xb4
[22368.155415] Instruction dump:
[22368.155446] 7fa4eb78 7fc3f378 4bb10611 60000000 eba10028 ebc10030 38210040 e8010010
[22368.155506] ebe1fff8 7c0803a6 4e800020 60420000 <0fe00000> 78290464 8129000c 552902ef
[22368.155566] ---[ end trace 70e3c0daa532c721 ]---
[22368.204754] WARNING: CPU: 34 PID: 40224 at /home/user/ubuntu-bionic/block/blk-mq.c:1211 __blk_mq_run_hw_queue+0xf0/0x140
[22368.204856] Modules linked in: sctp esp4 xfrm4_mode_transport ccm xfrm_user xfrm_algo minix nls_iso8859_1 xfs rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss parport_pc nfs_acl ppdev lockd grace sunrpc lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_blk virtio_scsi virtio_net
[22368.205275] CPU: 34 PID: 40224 Comm: kworker/34:1H Tainted: G W 4.15.0-12-generic #13+leo20180320
[22368.205352] Workqueue: kblockd blk_mq_run_work_fn
[22368.205392] NIP: c00000000068e0f0 LR: c000000000131538 CTR: c00000000068e140
[22368.205449] REGS: c00000017ac8b9d0 TRAP: 0700 Tainted: G W (4.15.0-12-generic)
[22368.205513] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44002284 XER: 00000000
[22368.205588] CFAR: c00000000068e070 SOFTE: 1
[22368.205588] GPR00: c000000000131538 c00000017ac8bc50 c0000000016eaf00 c0000001346f7800
[22368.205588] GPR04: c0000001346f7880 0000000000000000 0000000044008282 c000000001721ed8
[22368.205588] GPR08: 0000000000000000 0000000000000001 0000000000000000 00000000ffffffff
[22368.205588] GPR12: c00000000068e140 c00000000fa97600 c00000000013a368 c000000179360000
[22368.205588] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[22368.205588] GPR20: c00000017ee78180 0000000000000000 0000000000000000 fffffffffffffef7
[22368.205588] GPR24: 0000000000000402 0000000000000000 c00000017ee78500 0000000000000000
[22368.205588] GPR28: c0000001346f7880 c00000017ee7c900 c00000017ee78180 c0000001346f7800
[22368.206068] NIP [c00000000068e0f0] __blk_mq_run_hw_queue+0xf0/0x140
[22368.206118] LR [c000000000131538] process_one_work+0x298/0x5a0
[22368.206165] Call Trace:
[22368.206263] [c00000017ac8bc90] [c000000000131538] process_one_work+0x298/0x5a0
[22368.206322] [c00000017ac8bd20] [c0000000001318d8] worker_thread+0x98/0x630
[22368.206373] [c00000017ac8bdc0] [c00000000013a508] kthread+0x1a8/0x1b0
[22368.206424] [c00000017ac8be30] [c00000000000b528] ret_from_kernel_thread+0x5c/0xb4
[22368.206480] Instruction dump:
[22368.206511] 7fa4eb78 7fc3f378 4bb10611 60000000 eba10028 ebc10030 38210040 e8010010
[22368.206588] ebe1fff8 7c0803a6 4e800020 60420000 <0fe00000> 78290464 8129000c 552902ef
[22368.206662] ---[ end trace 70e3c0daa532c722 ]---
[22368.340886] WARNING: workqueue cpumask: online intersect > possible intersect
[22368.378115] WARNING: CPU: 36 PID: 40286 at /home/user/ubuntu-bionic/block/blk-mq.c:1211 __blk_mq_run_hw_queue+0xf0/0x140
[22368.378224] Modules linked in: sctp esp4 xfrm4_mode_transport ccm xfrm_user xfrm_algo minix nls_iso8859_1 xfs rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss parport_pc nfs_acl ppdev lockd grace sunrpc lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_blk virtio_scsi virtio_net
[22368.378684] CPU: 36 PID: 40286 Comm: kworker/36:0H Tainted: G W 4.15.0-12-generic #13+leo20180320
[22368.378762] Workqueue: kblockd blk_mq_run_work_fn
[22368.378801] NIP: c00000000068e0f0 LR: c000000000131538 CTR: c00000000068e140
[22368.378859] REGS: c00000017ace79d0 TRAP: 0700 Tainted: G W (4.15.0-12-generic)
[22368.378942] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 42002424 XER: 00000000
[22368.379015] CFAR: c00000000068e070 SOFTE: 1
[22368.379015] GPR00: c000000000131538 c00000017ace7c50 c0000000016eaf00 c000000142f0fc00
[22368.379015] GPR04: c000000142f0fc80 0000000000000000 0000000000000000 c000000001721ed8
[22368.379015] GPR08: 0000000000000000 0000000000000001 0000000000000000 c0000001788e2930
[22368.379015] GPR12: c00000000068e140 c00000000fa98c00 c00000000013a368 c0000001760c2400
[22368.379015] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[22368.379015] GPR20: c00000017eef8180 0000000000000000 0000000000000000 fffffffffffffef7
[22368.379015] GPR24: 0000000000000402 0000000000000000 c00000017eef8500 0000000000000000
[22368.379015] GPR28: c000000142f0fc80 c00000017eefc900 c00000017eef8180 c000000142f0fc00
[22368.379497] NIP [c00000000068e0f0] __blk_mq_run_hw_queue+0xf0/0x140
[22368.379547] LR [c000000000131538] process_one_work+0x298/0x5a0
[22368.379594] Call Trace:
[22368.379617] [c00000017ace7c50] [c00000017ace7ce0] 0xc00000017ace7ce0 (unreliable)
[22368.379678] [c00000017ace7c90] [c000000000131538] process_one_work+0x298/0x5a0
[22368.379739] [c00000017ace7d20] [c0000000001318d8] worker_thread+0x98/0x630
[22368.379790] [c00000017ace7dc0] [c00000000013a508] kthread+0x1a8/0x1b0
[22368.379842] [c00000017ace7e30] [c00000000000b528] ret_from_kernel_thread+0x5c/0xb4
[22368.379907] Instruction dump:
[22368.379938] 7fa4eb78 7fc3f378 4bb10611 60000000 eba10028 ebc10030 38210040 e8010010
[22368.380005] ebe1fff8 7c0803a6 4e800020 60420000 <0fe00000> 78290464 8129000c 552902ef
[22368.380070] ---[ end trace 70e3c0daa532c723 ]---
[22368.488613] WARNING: CPU: 39 PID: 40552 at /home/user/ubuntu-bionic/block/blk-mq.c:1211 __blk_mq_run_hw_queue+0xf0/0x140
[22368.488724] Modules linked in: sctp esp4 xfrm4_mode_transport ccm xfrm_user xfrm_algo minix nls_iso8859_1 xfs rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss parport_pc nfs_acl ppdev lockd grace sunrpc lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_blk virtio_scsi virtio_net
[22368.489289] CPU: 39 PID: 40552 Comm: kworker/39:0H Tainted: G W 4.15.0-12-generic #13+leo20180320
[22368.489367] Workqueue: kblockd blk_mq_run_work_fn
[22368.489409] NIP: c00000000068e0f0 LR: c000000000131538 CTR: c00000000068e140
[22368.489467] REGS: c00000017285b9d0 TRAP: 0700 Tainted: G W (4.15.0-12-generic)
[22368.489532] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 42002424 XER: 00000000
[22368.489605] CFAR: c00000000068e070 SOFTE: 1
[22368.489605] GPR00: c000000000131538 c00000017285bc50 c0000000016eaf00 c0000001346f7800
[22368.489605] GPR04: c0000001346f7880 0000000000000000 0000000000000000 c000000001721ed8
[22368.489605] GPR08: 0000000000000000 0000000000000001 0000000000000000 c000000142ca9c30
[22368.489605] GPR12: c00000000068e140 c00000000fa9ad00 c00000000013a368 c0000001760c9e40
[22368.489605] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[22368.489605] GPR20: c00000017efb8180 0000000000000000 0000000000000000 fffffffffffffef7
[22368.489605] GPR24: 0000000000000402 0000000000000000 c00000017efb8500 0000000000000000
[22368.489605] GPR28: c0000001346f7880 c00000017efbc900 c00000017efb8180 c0000001346f7800
[22368.490093] NIP [c00000000068e0f0] __blk_mq_run_hw_queue+0xf0/0x140
[22368.490144] LR [c000000000131538] process_one_work+0x298/0x5a0
[22368.490191] Call Trace:
[22368.490213] [c00000017285bc50] [c00000017285bce0] 0xc00000017285bce0 (unreliable)
[22368.490272] [c00000017285bc90] [c000000000131538] process_one_work+0x298/0x5a0
[22368.490330] [c00000017285bd20] [c0000000001318d8] worker_thread+0x98/0x630
[22368.490379] [c00000017285bdc0] [c00000000013a508] kthread+0x1a8/0x1b0
[22368.490431] [c00000017285be30] [c00000000000b528] ret_from_kernel_thread+0x5c/0xb4
[22368.490487] Instruction dump:
[22368.490517] 7fa4eb78 7fc3f378 4bb10611 60000000 eba10028 ebc10030 38210040 e8010010
[22368.490576] ebe1fff8 7c0803a6 4e800020 60420000 <0fe00000> 78290464 8129000c 552902ef
[22368.490636] ---[ end trace 70e3c0daa532c724 ]---
[22368.522251] WARNING: CPU: 39 PID: 41050 at /home/user/ubuntu-bionic/block/blk-mq.c:1211 __blk_mq_run_hw_queue+0xf0/0x140
[22368.522353] Modules linked in: sctp esp4 xfrm4_mode_transport ccm xfrm_user xfrm_algo minix nls_iso8859_1 xfs rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss parport_pc nfs_acl ppdev lockd grace sunrpc lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_blk virtio_scsi virtio_net
[22368.522772] CPU: 39 PID: 41050 Comm: kworker/39:1H Tainted: G W 4.15.0-12-generic #13+leo20180320
[22368.522851] Workqueue: kblockd blk_mq_run_work_fn
[22368.522890] NIP: c00000000068e0f0 LR: c000000000131538 CTR: c00000000068e140
[22368.522962] REGS: c0000000e95e39d0 TRAP: 0700 Tainted: G W (4.15.0-12-generic)
[22368.523029] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44002884 XER: 00000000
[22368.523103] CFAR: c00000000068e070 SOFTE: 1
[22368.523103] GPR00: c000000000131538 c0000000e95e3c50 c0000000016eaf00 c0000001346f7800
[22368.523103] GPR04: c0000001346f7880 0000000000000000 0000000044008882 c000000001721ed8
[22368.523103] GPR08: 0000000000000000 0000000000000001 0000000000000000 00000000ffffffff
[22368.523103] GPR12: c00000000068e140 c00000000fa9ad00 c00000000013a368 c000000261facac0
[22368.523103] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[22368.523103] GPR20: c00000017efb8180 0000000000000000 0000000000000000 fffffffffffffef7
[22368.523103] GPR24: 0000000000000402 0000000000000000 c00000017efb8500 0000000000000000
[22368.523103] GPR28: c0000001346f7880 c00000017efbc900 c00000017efb8180 c0000001346f7800
[22368.523584] NIP [c00000000068e0f0] __blk_mq_run_hw_queue+0xf0/0x140
[22368.523635] LR [c000000000131538] process_one_work+0x298/0x5a0
[22368.523682] Call Trace:
[22368.523704] [c0000000e95e3c90] [c000000000131538] process_one_work+0x298/0x5a0
[22368.523761] [c0000000e95e3d20] [c0000000001318d8] worker_thread+0x98/0x630
[22368.523811] [c0000000e95e3dc0] [c00000000013a508] kthread+0x1a8/0x1b0
[22368.523862] [c0000000e95e3e30] [c00000000000b528] ret_from_kernel_thread+0x5c/0xb4
[22368.523919] Instruction dump:
[22368.523948] 7fa4eb78 7fc3f378 4bb10611 60000000 eba10028 ebc10030 38210040 e8010010
[22368.524009] ebe1fff8 7c0803a6 4e800020 60420000 <0fe00000> 78290464 8129000c 552902ef
[22368.524068] ---[ end trace 70e3c0daa532c725 ]---
[22368.525105] WARNING: CPU: 38 PID: 40474 at /home/user/ubuntu-bionic/block/blk-mq.c:1211 __blk_mq_run_hw_queue+0xf0/0x140
[22368.525198] Modules linked in: sctp esp4 xfrm4_mode_transport ccm xfrm_user xfrm_algo minix nls_iso8859_1 xfs rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss parport_pc nfs_acl ppdev lockd grace sunrpc lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_blk virtio_scsi virtio_net
[22368.525604] CPU: 38 PID: 40474 Comm: kworker/38:0H Tainted: G W 4.15.0-12-generic #13+leo20180320
[22368.525692] Workqueue: kblockd blk_mq_run_work_fn
[22368.525736] NIP: c00000000068e0f0 LR: c000000000131538 CTR: c00000000068e140
[22368.525800] REGS: c00000017282f9d0 TRAP: 0700 Tainted: G W (4.15.0-12-generic)
[22368.525869] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 42002424 XER: 00000000
[22368.525944] CFAR: c00000000068e070 SOFTE: 1
[22368.525944] GPR00: c000000000131538 c00000017282fc50 c0000000016eaf00 c000000142f0fc00
[22368.525944] GPR04: c000000142f0fc80 0000000000000000 0000000000000000 c000000001721ed8
[22368.525944] GPR08: 0000000000000000 0000000000000001 0000000000000000 c000000142ca5730
[22368.525944] GPR12: c00000000068e140 c00000000fa9a200 c00000000013a368 c0000001760c4040
[22368.525944] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[22368.525944] GPR20: c00000017ef78180 0000000000000000 0000000000000000 fffffffffffffef7
[22368.525944] GPR24: 0000000000000402 0000000000000000 c00000017ef78500 0000000000000000
[22368.525944] GPR28: c000000142f0fc80 c00000017ef7c900 c00000017ef78180 c000000142f0fc00
[22368.526428] NIP [c00000000068e0f0] __blk_mq_run_hw_queue+0xf0/0x140
[22368.526477] LR [c000000000131538] process_one_work+0x298/0x5a0
[22368.526524] Call Trace:
[22368.526546] [c00000017282fc50] [c00000017282fce0] 0xc00000017282fce0 (unreliable)
[22368.526604] [c00000017282fc90] [c000000000131538] process_one_work+0x298/0x5a0
[22368.526662] [c00000017282fd20] [c0000000001318d8] worker_thread+0x98/0x630
[22368.526711] [c00000017282fdc0] [c00000000013a508] kthread+0x1a8/0x1b0
[22368.526761] [c00000017282fe30] [c00000000000b528] ret_from_kernel_thread+0x5c/0xb4
[22368.526818] Instruction dump:
[22368.526849] 7fa4eb78 7fc3f378 4bb10611 60000000 eba10028 ebc10030 38210040 e8010010
[22368.526908] ebe1fff8 7c0803a6 4e800020 60420000 <0fe00000> 78290464 8129000c 552902ef
[22368.526984] ---[ end trace 70e3c0daa532c726 ]---
[22368.829929] WARNING: CPU: 32 PID: 39931 at /home/user/ubuntu-bionic/block/blk-mq.c:1211 __blk_mq_run_hw_queue+0xf0/0x140
[22368.830040] Modules linked in: sctp esp4 xfrm4_mode_transport ccm xfrm_user xfrm_algo minix nls_iso8859_1 xfs rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss parport_pc nfs_acl ppdev lockd grace sunrpc lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_blk virtio_scsi virtio_net
[22368.830473] CPU: 32 PID: 39931 Comm: kworker/32:0H Tainted: G W 4.15.0-12-generic #13+leo20180320
[22368.830588] Workqueue: kblockd blk_mq_run_work_fn
[22368.830629] NIP: c00000000068e0f0 LR: c000000000131538 CTR: c00000000068e140
[22368.830688] REGS: c0000001789779d0 TRAP: 0700 Tainted: G W (4.15.0-12-generic)
[22368.830754] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 42002424 XER: 00000000
[22368.830830] CFAR: c00000000068e070 SOFTE: 1
[22368.830830] GPR00: c000000000131538 c000000178977c50 c0000000016eaf00 c0000001346f7800
[22368.830830] GPR04: c0000001346f7880 0000000000000000 0000000000000000 c000000001721ed8
[22368.830830] GPR08: 0000000000000000 0000000000000001 0000000000000000 c0000001727a5730
[22368.830830] GPR12: c00000000068e140 c00000000fa96000 c00000000013a368 c0000001760ca940
[22368.830830] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[22368.830830] GPR20: c00000017edf8180 0000000000000000 0000000000000000 fffffffffffffef7
[22368.830830] GPR24: 0000000000000402 0000000000000000 c00000017edf8500 0000000000000000
[22368.830830] GPR28: c0000001346f7880 c00000017edfc900 c00000017edf8180 c0000001346f7800
[22368.831330] NIP [c00000000068e0f0] __blk_mq_run_hw_queue+0xf0/0x140
[22368.831383] LR [c000000000131538] process_one_work+0x298/0x5a0
[22368.831472] Call Trace:
[22368.831495] [c000000178977c50] [c000000178977ce0] 0xc000000178977ce0 (unreliable)
[22368.831554] [c000000178977c90] [c000000000131538] process_one_work+0x298/0x5a0
[22368.831612] [c000000178977d20] [c0000000001318d8] worker_thread+0x98/0x630
[22368.831662] [c000000178977dc0] [c00000000013a508] kthread+0x1a8/0x1b0
[22368.831714] [c000000178977e30] [c00000000000b528] ret_from_kernel_thread+0x5c/0xb4
[22368.831773] Instruction dump:
[22368.831803] 7fa4eb78 7fc3f378 4bb10611 60000000 eba10028 ebc10030 38210040 e8010010
[22368.831863] ebe1fff8 7c0803a6 4e800020 60420000 <0fe00000> 78290464 8129000c 552902ef
[22368.831924] ---[ end trace 70e3c0daa532c727 ]---
[22368.831972] WARNING: CPU: 32 PID: 39931 at /home/user/ubuntu-bionic/block/blk-mq.c:1211 __blk_mq_run_hw_queue+0xf0/0x140
[22368.832047] Modules linked in: sctp esp4 xfrm4_mode_transport ccm xfrm_user xfrm_algo minix nls_iso8859_1 xfs rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss parport_pc nfs_acl ppdev lockd grace sunrpc lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_blk virtio_scsi virtio_net
[22368.832450] CPU: 32 PID: 39931 Comm: kworker/32:0H Tainted: G W 4.15.0-12-generic #13+leo20180320
[22368.832527] Workqueue: kblockd blk_mq_run_work_fn
[22368.832565] NIP: c00000000068e0f0 LR: c000000000131538 CTR: c00000000068e140
[22368.832624] REGS: c0000001789779d0 TRAP: 0700 Tainted: G W (4.15.0-12-generic)
[22368.832690] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 42002424 XER: 00000000
[22368.832761] CFAR: c00000000068e070 SOFTE: 1
[22368.832761] GPR00: c000000000131538 c000000178977c50 c0000000016eaf00 c0000001346f7800
[22368.832761] GPR04: c0000001346f7880 0000000000000000 0000000000000000 c000000001721ed8
[22368.832761] GPR08: 0000000000000000 0000000000000001 0000000000000000 c0000001727a5730
[22368.832761] GPR12: c00000000068e140 c00000000fa96000 c00000000013a368 c0000001760ca940
[22368.832761] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[22368.832761] GPR20: c00000017edf8180 0000000000000000 0000000000000000 fffffffffffffef7
[22368.832761] GPR24: 0000000000000402 0000000000000000 c00000017edf8500 0000000000000000
[22368.832761] GPR28: c0000001346f7880 c00000017edfc900 c00000017edf8180 c0000001346f7800
[22368.833253] NIP [c00000000068e0f0] __blk_mq_run_hw_queue+0xf0/0x140
[22368.833302] LR [c000000000131538] process_one_work+0x298/0x5a0
[22368.833351] Call Trace:
[22368.833371] [c000000178977c50] [c000000178977c90] 0xc000000178977c90 (unreliable)
[22368.833429] [c000000178977c90] [c000000000131538] process_one_work+0x298/0x5a0
[22368.833488] [c000000178977d20] [c0000000001318d8] worker_thread+0x98/0x630
[22368.833537] [c000000178977dc0] [c00000000013a508] kthread+0x1a8/0x1b0
[22368.833587] [c000000178977e30] [c00000000000b528] ret_from_kernel_thread+0x5c/0xb4
[22368.833642] Instruction dump:
[22368.833673] 7fa4eb78 7fc3f378 4bb10611 60000000 eba10028 ebc10030 38210040 e8010010
[22368.833732] ebe1fff8 7c0803a6 4e800020 60420000 <0fe00000> 78290464 8129000c 552902ef
[22368.833792] ---[ end trace 70e3c0daa532c728 ]---
[22368.913232] WARNING: CPU: 35 PID: 40184 at /home/user/ubuntu-bionic/block/blk-mq.c:1211 __blk_mq_run_hw_queue+0xf0/0x140
[22368.913335] Modules linked in: sctp esp4 xfrm4_mode_transport ccm xfrm_user xfrm_algo minix nls_iso8859_1 xfs rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss parport_pc nfs_acl ppdev lockd grace sunrpc lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_blk virtio_scsi virtio_net
[22368.913750] CPU: 35 PID: 40184 Comm: kworker/35:0H Tainted: G W 4.15.0-12-generic #13+leo20180320
[22368.913830] Workqueue: kblockd blk_mq_run_work_fn
[22368.913869] NIP: c00000000068e0f0 LR: c000000000131538 CTR: c00000000068e140
[22368.913928] REGS: c00000017acc79d0 TRAP: 0700 Tainted: G W (4.15.0-12-generic)
[22368.913996] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 42002424 XER: 00000000
[22368.914070] CFAR: c00000000068e070 SOFTE: 1
[22368.914070] GPR00: c000000000131538 c00000017acc7c50 c0000000016eaf00 c000000142f0fc00
[22368.914070] GPR04: c000000142f0fc80 0000000000000000 0000000000000000 c000000001721ed8
[22368.914070] GPR08: 0000000000000000 0000000000000001 0000000000000000 c000000172428530
[22368.914070] GPR12: c00000000068e140 c00000000fa98100 c00000000013a368 c0000001760c2ac0
[22368.914070] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[22368.914070] GPR20: c00000017eeb8180 0000000000000000 0000000000000000 fffffffffffffef7
[22368.914070] GPR24: 0000000000000402 0000000000000000 c00000017eeb8500 0000000000000000
[22368.914070] GPR28: c000000142f0fc80 c00000017eebc900 c00000017eeb8180 c000000142f0fc00
[22368.914583] NIP [c00000000068e0f0] __blk_mq_run_hw_queue+0xf0/0x140
[22368.914635] LR [c000000000131538] process_one_work+0x298/0x5a0
[

> virsh dumpxml boslcp3g4

root@boslcp3:~# virsh dumpxml boslcp3g4 | grep vcpu
  <vcpu placement='static' current='64'>80</vcpu>
  <vcpus>
    <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='1' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='2' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='3' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='4' enabled='yes' hotpluggable='no' order='2'/>
    <vcpu id='5' enabled='yes' hotpluggable='no' order='2'/>
    <vcpu id='6' enabled='yes' hotpluggable='no' order='2'/>
    <vcpu id='7' enabled='yes' hotpluggable='no' order='2'/>
    <vcpu id='8' enabled='yes' hotpluggable='no' order='3'/>
    <vcpu id='9' enabled='yes' hotpluggable='no' order='3'/>
    <vcpu id='10' enabled='yes' hotpluggable='no' order='3'/>
    <vcpu id='11' enabled='yes' hotpluggable='no' order='3'/>
    <vcpu id='12' enabled='yes' hotpluggable='no' order='4'/>
    <vcpu id='13' enabled='yes' hotpluggable='no' order='4'/>
    <vcpu id='14' enabled='yes' hotpluggable='no' order='4'/>
    <vcpu id='15' enabled='yes' hotpluggable='no' order='4'/>
    <vcpu id='16' enabled='yes' hotpluggable='no' order='5'/>
    <vcpu id='17' enabled='yes' hotpluggable='no' order='5'/>
    <vcpu id='18' enabled='yes' hotpluggable='no' order='5'/>
    <vcpu id='19' enabled='yes' hotpluggable='no' order='5'/>
    <vcpu id='20' enabled='yes' hotpluggable='no' order='6'/>
    <vcpu id='21' enabled='yes' hotpluggable='no' order='6'/>
    <vcpu id='22' enabled='yes' hotpluggable='no' order='6'/>
    <vcpu id='23' enabled='yes' hotpluggable='no' order='6'/>
    <vcpu id='24' enabled='yes' hotpluggable='no' order='7'/>
    <vcpu id='25' enabled='yes' hotpluggable='no' order='7'/>
    <vcpu id='26' enabled='yes' hotpluggable='no' order='7'/>
    <vcpu id='27' enabled='yes' hotpluggable='no' order='7'/>
    <vcpu id='28' enabled='yes' hotpluggable='no' order='8'/>
    <vcpu id='29' enabled='yes' hotpluggable='no' order='8'/>
    <vcpu id='30' enabled='yes' hotpluggable='no' order='8'/>
    <vcpu id='31' enabled='yes' hotpluggable='no' order='8'/>
    <vcpu id='32' enabled='yes' hotpluggable='yes' order='9'/>
    <vcpu id='33' enabled='yes' hotpluggable='yes' order='9'/>
    <vcpu id='34' enabled='yes' hotpluggable='yes' order='9'/>
    <vcpu id='35' enabled='yes' hotpluggable='yes' order='9'/>
    <vcpu id='36' enabled='yes' hotpluggable='yes' order='10'/>
    <vcpu id='37' enabled='yes' hotpluggable='yes' order='10'/>
    <vcpu id='38' enabled='yes' hotpluggable='yes' order='10'/>
    <vcpu id='39' enabled='yes' hotpluggable='yes' order='10'/>
    <vcpu id='40' enabled='yes' hotpluggable='yes' order='11'/>
    <vcpu id='41' enabled='yes' hotpluggable='yes' order='11'/>
    <vcpu id='42' enabled='yes' hotpluggable='yes' order='11'/>
    <vcpu id='43' enabled='yes' hotpluggable='yes' order='11'/>
    <vcpu id='44' enabled='yes' hotpluggable='yes' order='12'/>
    <vcpu id='45' enabled='yes' hotpluggable='yes' order='12'/>
    <vcpu id='46' enabled='yes' hotpluggable='yes' order='12'/>
    <vcpu id='47' enabled='yes' hotpluggable='yes' order='12'/>
    <vcpu id='48' enabled='yes' hotpluggable='yes' order='13'/>
    <vcpu id='49' enabled='yes' hotpluggable='yes' order='13'/>
    <vcpu id='50' enabled='yes' hotpluggable='yes' order='13'/>
    <vcpu id='51' enabled='yes' hotpluggable='yes' order='13'/>
    <vcpu id='52' enabled='yes' hotpluggable='yes' order='14'/>
    <vcpu id='53' enabled='yes' hotpluggable='yes' order='14'/>
    <vcpu id='54' enabled='yes' hotpluggable='yes' order='14'/>
    <vcpu id='55' enabled='yes' hotpluggable='yes' order='14'/>
    <vcpu id='56' enabled='yes' hotpluggable='yes' order='15'/>
    <vcpu id='57' enabled='yes' hotpluggable='yes' order='15'/>
    <vcpu id='58' enabled='yes' hotpluggable='yes' order='15'/>
    <vcpu id='59' enabled='yes' hotpluggable='yes' order='15'/>
    <vcpu id='60' enabled='yes' hotpluggable='yes' order='16'/>
    <vcpu id='61' enabled='yes' hotpluggable='yes' order='16'/>
    <vcpu id='62' enabled='yes' hotpluggable='yes' order='16'/>
    <vcpu id='63' enabled='yes' hotpluggable='yes' order='16'/>
    <vcpu id='64' enabled='no' hotpluggable='yes'/>
    <vcpu id='65' enabled='no' hotpluggable='yes'/>
    <vcpu id='66' enabled='no' hotpluggable='yes'/>
    <vcpu id='67' enabled='no' hotpluggable='yes'/>
    <vcpu id='68' enabled='no' hotpluggable='yes'/>
    <vcpu id='69' enabled='no' hotpluggable='yes'/>
    <vcpu id='70' enabled='no' hotpluggable='yes'/>
    <vcpu id='71' enabled='no' hotpluggable='yes'/>
    <vcpu id='72' enabled='no' hotpluggable='yes'/>
    <vcpu id='73' enabled='no' hotpluggable='yes'/>
    <vcpu id='74' enabled='no' hotpluggable='yes'/>
    <vcpu id='75' enabled='no' hotpluggable='yes'/>
    <vcpu id='76' enabled='no' hotpluggable='yes'/>
    <vcpu id='77' enabled='no' hotpluggable='yes'/>
    <vcpu id='78' enabled='no' hotpluggable='yes'/>
    <vcpu id='79' enabled='no' hotpluggable='yes'/>
  </vcpus>
root@boslcp3:~#

> lspcu output from guest

root@boslcp3g4:~# lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0,4,8,12,16,20,24,28,32-63
Off-line CPU(s) list: 1-3,5-7,9-11,13-15,17-19,21-23,25-27,29-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Model: 2.2 (pvr 004e 1202)
Model name: POWER9 (architected), altivec supported
Hypervisor vendor: KVM
Virtualization type: para
L1d cache: 32K
L1i cache: 32K
NUMA node0 CPU(s): 0,4,8,12,16,20,24,28,32-39
NUMA node1 CPU(s): 40-63
root@boslcp3g4:~#

> Attched guest consoel logs

Hi, there is a patch to remove the WARNING message but it's harmless according to the patch author:

4058ba173eb411422482125e14e7bb12b70c8b38

 blk-mq: turn WARN_ON in __blk_mq_run_hw_queue into printk

 We know this WARN_ON is harmless and in reality it may be trigged,
 so convert it to printk() and dump_stack() to avoid to confusing
 people.

I'll prepare a backport to Canonical but I lowering this importance to medium, since it's not a problem.

Thank you

Revision history for this message
bugproxy (bugproxy) wrote : hotplug CPU dump messages from boslcp3g4

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-165971 severity-medium targetmilestone-inin1804
Revision history for this message
bugproxy (bugproxy) wrote : dmesg message logs during hotplug cpu

Default Comment by Bridge

Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
importance: Undecided → Medium
status: New → Triaged
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
tags: added: triage-g
Seth Forshee (sforshee)
Changed in linux (Ubuntu):
status: New → Fix Committed
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
status: Triaged → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (40.4 KiB)

This bug was fixed in the package linux - 4.15.0-15.16

---------------
linux (4.15.0-15.16) bionic; urgency=medium

  * linux: 4.15.0-15.16 -proposed tracker (LP: #1761177)

  * FFe: Enable configuring resume offset via sysfs (LP: #1760106)
    - PM / hibernate: Make passing hibernate offsets more friendly

  * /dev/bcache/by-uuid links not created after reboot (LP: #1729145)
    - SAUCE: (no-up) bcache: decouple emitting a cached_dev CHANGE uevent

  * Ubuntu18.04:POWER9:DD2.2 - Unable to start a KVM guest with default machine
    type(pseries-bionic) complaining "KVM implementation does not support
    Transactional Memory, try cap-htm=off" (kvm) (LP: #1752026)
    - powerpc: Use feature bit for RTC presence rather than timebase presence
    - powerpc: Book E: Remove unused CPU_FTR_L2CSR bit
    - powerpc: Free up CPU feature bits on 64-bit machines
    - powerpc: Add CPU feature bits for TM bug workarounds on POWER9 v2.2
    - powerpc/powernv: Provide a way to force a core into SMT4 mode
    - KVM: PPC: Book3S HV: Work around transactional memory bugs in POWER9
    - KVM: PPC: Book3S HV: Work around XER[SO] bug in fake suspend mode
    - KVM: PPC: Book3S HV: Work around TEXASR bug in fake suspend state

  * Important Kernel fixes to be backported for Power9 (kvm) (LP: #1758910)
    - powerpc/mm: Fixup tlbie vs store ordering issue on POWER9

  * Ubuntu 18.04 - IO Hang on some namespaces when running HTX with 16
    namespaces (Bolt / NVMe) (LP: #1757497)
    - powerpc/64s: Fix lost pending interrupt due to race causing lost update to
      irq_happened

  * fwts-efi-runtime-dkms 18.03.00-0ubuntu1: fwts-efi-runtime-dkms kernel module
    failed to build (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

linux (4.15.0-14.15) bionic; urgency=medium

  * linux: 4.15.0-14.15 -proposed tracker (LP: #1760678)

  * [Bionic] mlx4 ETH - mlnx_qos failed when set some TC to vendor
    (LP: #1758662)
    - net/mlx4_en: Change default QoS settings

  * AT_BASE_PLATFORM in AUXV is absent on kernels available on Ubuntu 17.10
    (LP: #1759312)
    - powerpc/64s: Fix NULL AT_BASE_PLATFORM when using DT CPU features

  * Bionic update to 4.15.15 stable release (LP: #1760585)
    - net: dsa: Fix dsa_is_user_port() test inversion
    - openvswitch: meter: fix the incorrect calculation of max delta_t
    - qed: Fix MPA unalign flow in case header is split across two packets.
    - tcp: purge write queue upon aborting the connection
    - qed: Fix non TCP packets should be dropped on iWARP ll2 connection
    - sysfs: symlink: export sysfs_create_link_nowarn()
    - net: phy: relax error checking when creating sysfs link netdev->phydev
    - devlink: Remove redundant free on error path
    - macvlan: filter out unsupported feature flags
    - net: ipv6: keep sk status consistent after datagram connect failure
    - ipv6: old_dport should be a __be16 in __ip6_datagram_connect()
    - ipv6: sr: fix NULL pointer dereference when setting encap source address
    - ipv6: sr: fix scheduling in RCU when creating seg6 lwtunnel state
    - mlxsw: spectrum_buffers: Set a minimum quota for CPU port traffic
    - net: phy: Tell caller result ...

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
status: Fix Committed → Fix Released
Brad Figg (brad-figg)
tags: added: cscc
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.