------- Comment From <email address hidden> 2017-04-10 00:00 EDT------- Did try with kdump enabled and updated kernel, still hitting at issue
# uname -a Linux ltc-test-ci1 4.10.0-19-generic #21-Ubuntu SMP Thu Apr 6 17:03:05 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
# [11827.612620] kernel BUG at /build/linux-mYrikn/linux-4.10.0/include/linux/swapops.h:129! [11827.612748] Oops: Exception in kernel mode, sig: 5 [#1] [11827.612796] SMP NR_CPUS=2048 [11827.612797] NUMA [11827.612832] PowerNV [11827.612881] Modules linked in: vhost_net macvtap macvlan xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 kvm_hv kvm_pr kvm tcm_fc libfc usb_f_tcm tcm_usb_gadget libcomposite udc _core tcm_qla2xxx qla2xxx scsi_transport_fc ib_srpt iscsi_target_mod tcm_loop vhost_scsi vhost target_core_user target_core_file target_core_iblock target_core_pscsi target_core_mod ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_tcpudp ipt_REJECT nf_reject_ipv4 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_nat ip 6table_mangle ip6table_raw iptable_security iptable_nat iptable_mangle iptable_raw ebtable_filter ebtables openvswitch ip6table_filter ip6_tables nf_conntrack_ipv6 nf_nat_ipv6 iptabl e_filter nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack [11827.613527] binfmt_misc vmx_crypto ipmi_powernv ipmi_devintf ipmi_msghandler leds_powernv uio_pdrv_genirq powernv_rng uio powernv_op_panel nfsd auth_rpcgss nfs_acl lockd grace su nrpc ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas crc32c_vpmsum tg3 ipr [11827.613935] CPU: 40 PID: 74758 Comm: CPU 17/KVM Not tainted 4.10.0-19-generic #21-Ubuntu [11827.614006] task: c000000e3998c600 task.stack: c000000e39900000 [11827.614065] NIP: c00000000030ea08 LR: c00000000030e918 CTR: 0000000000000000 [11827.614135] REGS: c000000e399033b0 TRAP: 0700 Not tainted (4.10.0-19-generic) [11827.614205] MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE> [11827.614211] CR: 44882882 XER: 00000000 [11827.614293] CFAR: c00000000030eb44 SOFTE: 1 [11827.614293] GPR00: c00000000030e918 c000000e39903630 c00000000145cb00 f000000001fa4770 [11827.614293] GPR04: c0000007e91d2010 f000000001fa4770 000000001f00025c 000000005c02001f [11827.614293] GPR08: c0000000015ccb00 0000000000000001 0000000000000001 0000000000201de9 [11827.614293] GPR12: 0000000000002200 c000000007b56800 0000000000000000 000000000007fe03 [11827.614293] GPR16: 0000000000010000 00003ffede020000 0000000000000000 0000000088000000 [11827.614293] GPR20: 0000000020000000 0000000088000000 0000000022000000 c000000f1b320000 [11827.614293] GPR24: c0000000015ce3d8 c000000e397c6e00 c000000f1d3dfb90 0000000000000000 [11827.614293] GPR28: c000000e39903710 3e00000000055c02 f000000001570080 f000000001fa4770 [11827.614897] NIP [c00000000030ea08] __migration_entry_wait+0x128/0x2a0 [11827.614956] LR [c00000000030e918] __migration_entry_wait+0x38/0x2a0 [11827.615015] Call Trace: [11827.615040] [c000000e39903630] [c00000000030e918] __migration_entry_wait+0x38/0x2a0 (unreliable) [11827.615125] [c000000e39903670] [c0000000002bcf2c] do_swap_page+0x73c/0x9a0 [11827.615185] [c000000e399036f0] [c0000000002c0d58] handle_mm_fault+0xac8/0x1600 [11827.615257] [c000000e399037e0] [c0000000002b53c4] __get_user_pages+0x194/0x4e0 [11827.615329] [c000000e39903890] [c0000000002b5aa4] get_user_pages_unlocked+0xf4/0x280 [11827.615401] [c000000e39903930] [c0000000002b6c6c] get_user_pages_fast+0xac/0x100 [11827.615474] [c000000e39903980] [d00000000f81d074] kvmppc_book3s_hv_page_fault+0x2bc/0xbc0 [kvm_hv] [11827.615559] [c000000e39903a70] [d00000000f819a30] kvmppc_vcpu_run_hv+0xbc8/0x1220 [kvm_hv] [11827.615636] [c000000e39903b80] [d00000000f5e32bc] kvmppc_vcpu_run+0x34/0x48 [kvm] [11827.615712] [c000000e39903ba0] [d00000000f5e036c] kvm_arch_vcpu_ioctl_run+0x64/0x170 [kvm] [11827.615842] [c000000e39903be0] [d00000000f5d3db8] kvm_vcpu_ioctl+0x500/0x780 [kvm] [11827.615981] [c000000e39903d40] [c00000000035c4b4] do_vfs_ioctl+0xd4/0x8c0 [11827.616095] [c000000e39903de0] [c00000000035cd74] SyS_ioctl+0xd4/0xf0 [11827.616210] [c000000e39903e30] [c00000000000b184] system_call+0x38/0xe0 [11827.616322] Instruction dump: [11827.616393] 3d020017 79293448 39481868 ebca0000 7fde4a14 e93e0020 712a0001 4082014c [11827.616533] 7fc9f378 e9290000 7d2948f8 792907e0 <0b090000> 39400000 3bbe001c 39000001 [11827.616672] ---[ end trace 56c09d0670f15647 ]--- [11827.616763] [11827.616829] Sending IPI to other CPUs [11827.617901] IPI complete [11828.619473] kexec: Starting switchover sequence. -> smp_release_cpus() spinning_secondaries = 79 <- smp_release_cpus() [ 5.018872] Processor 1 is stuck. [ 10.020806] Processor 2 is stuck. [ 15.022726] Processor 3 is stuck. [ 20.024680] Processor 4 is stuck. [ 25.026592] Processor 5 is stuck. [ 30.028517] Processor 6 is stuck. [ 35.030444] Processor 7 is stuck.
system is stuck and not in a usable state to get any other info.
------- Comment From <email address hidden> 2017-04-10 00:00 EDT-------
Did try with kdump enabled and updated kernel, still hitting at issue
# uname -a
Linux ltc-test-ci1 4.10.0-19-generic #21-Ubuntu SMP Thu Apr 6 17:03:05 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
# [11827.612620] kernel BUG at /build/ linux-mYrikn/ linux-4. 10.0/include/ linux/swapops. h:129! masquerade_ ipv4 kvm_hv kvm_pr kvm tcm_fc libfc usb_f_tcm tcm_usb_gadget libcomposite udc iscsi ip_tables x_tables autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy EE,ME,IR, DR,RI,LE> entry_wait+ 0x128/0x2a0 entry_wait+ 0x38/0x2a0 entry_wait+ 0x38/0x2a0 (unreliable) page+0x73c/ 0x9a0 mm_fault+ 0xac8/0x1600 pages+0x194/ 0x4e0 pages_unlocked+ 0xf4/0x280 pages_fast+ 0xac/0x100 book3s_ hv_page_ fault+0x2bc/ 0xbc0 [kvm_hv] vcpu_run_ hv+0xbc8/ 0x1220 [kvm_hv] vcpu_run+ 0x34/0x48 [kvm] vcpu_ioctl_ run+0x64/ 0x170 [kvm] ioctl+0x500/ 0x780 [kvm] ioctl+0xd4/ 0x8c0 call+0x38/ 0xe0 secondaries = 79
[11827.612748] Oops: Exception in kernel mode, sig: 5 [#1]
[11827.612796] SMP NR_CPUS=2048
[11827.612797] NUMA
[11827.612832] PowerNV
[11827.612881] Modules linked in: vhost_net macvtap macvlan xt_CHECKSUM ipt_MASQUERADE nf_nat_
_core tcm_qla2xxx qla2xxx scsi_transport_fc ib_srpt iscsi_target_mod tcm_loop vhost_scsi vhost target_core_user target_core_file target_core_iblock target_core_pscsi target_core_mod
ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_tcpudp ipt_REJECT nf_reject_ipv4 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_nat ip
6table_mangle ip6table_raw iptable_security iptable_nat iptable_mangle iptable_raw ebtable_filter ebtables openvswitch ip6table_filter ip6_tables nf_conntrack_ipv6 nf_nat_ipv6 iptabl
e_filter nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack
[11827.613527] binfmt_misc vmx_crypto ipmi_powernv ipmi_devintf ipmi_msghandler leds_powernv uio_pdrv_genirq powernv_rng uio powernv_op_panel nfsd auth_rpcgss nfs_acl lockd grace su
nrpc ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_
async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas crc32c_vpmsum tg3 ipr
[11827.613935] CPU: 40 PID: 74758 Comm: CPU 17/KVM Not tainted 4.10.0-19-generic #21-Ubuntu
[11827.614006] task: c000000e3998c600 task.stack: c000000e39900000
[11827.614065] NIP: c00000000030ea08 LR: c00000000030e918 CTR: 0000000000000000
[11827.614135] REGS: c000000e399033b0 TRAP: 0700 Not tainted (4.10.0-19-generic)
[11827.614205] MSR: 9000000000029033 <SF,HV,
[11827.614211] CR: 44882882 XER: 00000000
[11827.614293] CFAR: c00000000030eb44 SOFTE: 1
[11827.614293] GPR00: c00000000030e918 c000000e39903630 c00000000145cb00 f000000001fa4770
[11827.614293] GPR04: c0000007e91d2010 f000000001fa4770 000000001f00025c 000000005c02001f
[11827.614293] GPR08: c0000000015ccb00 0000000000000001 0000000000000001 0000000000201de9
[11827.614293] GPR12: 0000000000002200 c000000007b56800 0000000000000000 000000000007fe03
[11827.614293] GPR16: 0000000000010000 00003ffede020000 0000000000000000 0000000088000000
[11827.614293] GPR20: 0000000020000000 0000000088000000 0000000022000000 c000000f1b320000
[11827.614293] GPR24: c0000000015ce3d8 c000000e397c6e00 c000000f1d3dfb90 0000000000000000
[11827.614293] GPR28: c000000e39903710 3e00000000055c02 f000000001570080 f000000001fa4770
[11827.614897] NIP [c00000000030ea08] __migration_
[11827.614956] LR [c00000000030e918] __migration_
[11827.615015] Call Trace:
[11827.615040] [c000000e39903630] [c00000000030e918] __migration_
[11827.615125] [c000000e39903670] [c0000000002bcf2c] do_swap_
[11827.615185] [c000000e399036f0] [c0000000002c0d58] handle_
[11827.615257] [c000000e399037e0] [c0000000002b53c4] __get_user_
[11827.615329] [c000000e39903890] [c0000000002b5aa4] get_user_
[11827.615401] [c000000e39903930] [c0000000002b6c6c] get_user_
[11827.615474] [c000000e39903980] [d00000000f81d074] kvmppc_
[11827.615559] [c000000e39903a70] [d00000000f819a30] kvmppc_
[11827.615636] [c000000e39903b80] [d00000000f5e32bc] kvmppc_
[11827.615712] [c000000e39903ba0] [d00000000f5e036c] kvm_arch_
[11827.615842] [c000000e39903be0] [d00000000f5d3db8] kvm_vcpu_
[11827.615981] [c000000e39903d40] [c00000000035c4b4] do_vfs_
[11827.616095] [c000000e39903de0] [c00000000035cd74] SyS_ioctl+0xd4/0xf0
[11827.616210] [c000000e39903e30] [c00000000000b184] system_
[11827.616322] Instruction dump:
[11827.616393] 3d020017 79293448 39481868 ebca0000 7fde4a14 e93e0020 712a0001 4082014c
[11827.616533] 7fc9f378 e9290000 7d2948f8 792907e0 <0b090000> 39400000 3bbe001c 39000001
[11827.616672] ---[ end trace 56c09d0670f15647 ]---
[11827.616763]
[11827.616829] Sending IPI to other CPUs
[11827.617901] IPI complete
[11828.619473] kexec: Starting switchover sequence.
-> smp_release_cpus()
spinning_
<- smp_release_cpus()
[ 5.018872] Processor 1 is stuck.
[ 10.020806] Processor 2 is stuck.
[ 15.022726] Processor 3 is stuck.
[ 20.024680] Processor 4 is stuck.
[ 25.026592] Processor 5 is stuck.
[ 30.028517] Processor 6 is stuck.
[ 35.030444] Processor 7 is stuck.
system is stuck and not in a usable state to get any other info.