---Problem Description---
While rebooting the firestone machine which is having a CAPI card,  just after the reboot command, kernel is getting panicked.
 
---uname output---
Linux ltc84-pkvm1 4.9.0-15-generic #16-Ubuntu SMP Fri Jan 20 15:28:49 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
 
---Additional Hardware Info---
0001:01:00.0 Processing accelerators: IBM Device 0477 (rev 01)
0002:00:00.0 Processing accelerators: IBM Device 4350 (rev 0a) 

 
Machine Type = PowerNV 8335-GTA  
 
---System Hang---
 reset via ipmitool
 
 
---Steps to Reproduce---
 1. on a  Firestone machine with CAPI Nallatech card , run reboot command from the terminal

 
Contact Information = <email address hidden> 

Oops output:
 Ubuntu 17.04.  .  .  .Ubuntu 17.04.  .  .  .Ubuntu 17.04.  .  .  .Ubuntu 17.04.  .  .  .Ubuntu 17.04.  .  .  .Ubuntu 17.04.  .  .  .[  283.410962] Oops: Exception in kernel mode, sig: 5 [#1]
[  283.410962] Oops: Exception in kernel mode, sig: 5 [#1]
[  283.411564] SMP NR_CPUS=2048 [  283.411564] SMP NR_CPUS=2048 [  283.411962] NUMA
[  283.411962] NUMA
[  283.412166] PowerNV
[  283.412166] PowerNV
[  283.412505] Modules linked in:[  283.412505] Modules linked in: kvm_hv kvm_hv kvm_pr kvm_pr kvm kvm usb_f_tcm usb_f_tcm libcomposite libcomposite udc_core udc_core target_core_mod target_core_mod configfs configfs ip_set ip_set nfnetlink nfnetlink bridge bridge stp stp llc llc joydev joydev input_leds input_leds mac_hid mac_hid hid_generic hid_generic usbhid usbhid hid hid ofpart ofpart cmdlinepart cmdlinepart at24 at24 nvmem_core nvmem_core powernv_flash powernv_flash ipmi_powernv ipmi_powernv ipmi_msghandler ipmi_msghandler mtd mtd uio_pdrv_genirq uio_pdrv_genirq uio uio ibmpowernv ibmpowernv vmx_crypto vmx_crypto opal_prd opal_prd powernv_rng powernv_rng binfmt_misc binfmt_misc x_tables x_tables autofs4 autofs4 uas uas usb_storage usb_storage nouveau nouveau crc32c_vpmsum crc32c_vpmsum ast ast i2c_algo_bit i2c_algo_bit ttm ttm ahci ahci drm_kms_helper drm_kms_helper syscopyarea syscopyarea tg3 tg3 libahci libahci sysfillrect sysfillrect sysimgblt sysimgblt fb_sys_fops fb_sys_fops drm drm cxl cxl pnv_php pnv_php [last unloaded: ip_tables] [last unloaded: ip_tables]

[  283.417833] CPU: 19 PID: 1 Comm: systemd-shutdow Tainted: G        W       4.9.0-15-generic #16-Ubuntu
[  283.417833] CPU: 19 PID: 1 Comm: systemd-shutdow Tainted: G        W       4.9.0-15-generic #16-Ubuntu
[  283.418795] task: c0000063bb938600 task.stack: c0000027f1188000
[  283.418795] task: c0000063bb938600 task.stack: c0000027f1188000
[  283.419412] NIP: c00000000064fd40 LR: c000000000630564 CTR: c0000000006304f0
[  283.419412] NIP: c00000000064fd40 LR: c000000000630564 CTR: c0000000006304f0
[  283.420277] REGS: c0000027f118b8b0 TRAP: 0700   Tainted: G        W        (4.9.0-15-generic)
[  283.420277] REGS: c0000027f118b8b0 TRAP: 0700   Tainted: G        W        (4.9.0-15-generic)
[  283.421161] MSR: 900000000282b033 [  283.421161] MSR: 900000000282b033 <<SFSF,HV,HV,VEC,VEC,VSX,VSX,EE,EE,FP,FP,ME,ME,IR,IR,DR,DR,RI,RI,LE,LE>>[  283.422060]   CR: 44288888  XER: 00000000
[  283.422060]   CR: 44288888  XER: 00000000
[  283.422475] CFAR: c000000000630560 [  283.422475] CFAR: c000000000630560 SOFTE: 1 SOFTE: 1
GPR00:
GPR00: c000000000630564 c000000000630564 c0000027f118bb30 c0000027f118bb30 c00000000141a300 c00000000141a300 c0000027f1069000 c0000027f1069000
GPR04:
GPR04: 0000000000000001 0000000000000001 c000000000081100 c000000000081100 900000000280b033 900000000280b033 000000000000001b 000000000000001b
GPR08:
GPR08: 0000000000000000 0000000000000000 0000000000000001 0000000000000001 c0000027f1069288 c0000027f1069288 9000000002803003 9000000002803003
GPR12:
GPR12: 0000000024288882 0000000024288882 c00000000fb8ab00 c00000000fb8ab00 0000000000000004 0000000000000004 0000000000000001 0000000000000001
GPR16:
GPR16: 0000000038af7370 0000000038af7370 0000000000000138 0000000000000138 0000000038af6be0 0000000038af6be0 0000000000000000 0000000000000000
GPR20:
GPR20: 0000000038af6b10 0000000038af6b10 0000000038af6fd0 0000000038af6fd0 00003fffcb9f3f10 00003fffcb9f3f10 0000000038b0f9c8 0000000038b0f9c8
GPR24:
GPR24: 0000000038af7040 0000000038af7040 0000000000000000 0000000000000000 0000000000000000 0000000000000000 c0000027f1069100 c0000027f1069100
GPR28:
GPR28: c000000001731f98 c000000001731f98 c000000001377630 c000000001377630 c0000027f10690a0 c0000027f10690a0 c0000027f1069000 c0000027f1069000

NIP [c00000000064fd40] pci_msi_shutdown+0x60/0x190
NIP [c00000000064fd40] pci_msi_shutdown+0x60/0x190
[  283.431503] LR [c000000000630564] pci_device_shutdown+0x74/0xf0
[  283.431503] LR [c000000000630564] pci_device_shutdown+0x74/0xf0
[  283.432841] Call Trace:
[  283.432841] Call Trace:
[  283.433696] [c0000027f118bb30] [c00000000072a0a4] __pm_runtime_resume+0x94/0xd0[  283.433696] [c0000027f118bb30] [c00000000072a0a4] __pm_runtime_resume+0x94/0xd0 (unreliable) (unreliable)

[  283.435213] [c0000027f118bb70] [c000000000630564] pci_device_shutdown+0x74/0xf0[  283.435213] [c0000027f118bb70] [c000000000630564] pci_device_shutdown+0x74/0xf0

[  283.436626] [c0000027f118bbb0] [c000000000713284] device_shutdown+0x1b4/0x2c0[  283.436626] [c0000027f118bbb0] [c000000000713284] device_shutdown+0x1b4/0x2c0

[  283.438016] [c0000027f118bc50] [c0000000000f9b64] kernel_restart_prepare+0x54/0x70[  283.438016] [c0000027f118bc50] [c0000000000f9b64] kernel_restart_prepare+0x54/0x70

[  283.439467] [c0000027f118bc70] [c0000000000f9cb8] kernel_restart+0x28/0xc0[  283.439467] [c0000027f118bc70] [c0000000000f9cb8] kernel_restart+0x28/0xc0

[  283.440860] [c0000027f118bce0] [c0000000000fa190] SyS_reboot+0x1c0/0x2d0[  283.440860] [c0000027f118bce0] [c0000000000fa190] SyS_reboot+0x1c0/0x2d0

[  283.442241] [c0000027f118be30] [c00000000000bd84] system_call+0x38/0xe0[  283.442241] [c0000027f118be30] [c00000000000bd84] system_call+0x38/0xe0

[  283.443616] Instruction dump:[  283.443616] Instruction dump:

[  283.444635] 4d820020 [  283.444635] 4d820020 7c0802a6 7c0802a6 fbc1fff0 fbc1fff0 fbe1fff8 fbe1fff8 f8010010 f8010010 f821ffc1 f821ffc1 7c691b78 7c691b78 7c7f1b78 7c7f1b78

[  283.446463] e9490289 [  283.446463] e9490289 7d295278 7d295278 7d290074 7d290074 7929d182 7929d182 <0b090000> <0b090000> 88a3004d 88a3004d 80830038 80830038 38c10020 38c10020

[  283.448336] ---[ end trace 503f59bbe300ec1f ]---
[  283.448336] ---[ end trace 503f59bbe300ec1f ]---
[  283.449715]
[  283.449715]
Ubuntu 17.04.  .  .  .Ubuntu 17.04.  .  .  .[  285.451149] systemd-shutdow: 7 output lines suppressed due to ratelimiting
[  285.451149] systemd-shutdow: 7 output lines suppressed due to ratelimiting
[  285.451865] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[  285.451865]
[  285.451865] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[  285.451865]
 
lspci:

root@ltc84-pkvm1:~# lspci
0000:00:00.0 PCI bridge: IBM Device 03dc
0000:01:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0000:02:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0000:02:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0000:03:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
0000:04:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
0001:00:00.0 PCI bridge: IBM Device 03dc
0001:01:00.0 Processing accelerators: IBM Device 0477 (rev 01)
0002:00:00.0 Processing accelerators: IBM Device 4350 (rev 0a)
0020:00:00.0 PCI bridge: IBM Device 03dc
0020:01:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0020:02:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0020:02:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0020:03:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
0020:04:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
0021:00:00.0 PCI bridge: IBM Device 03dc
0021:01:00.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0021:01:00.1 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
0021:01:00.2 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
0021:01:00.3 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
0021:01:00.4 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
0021:02:01.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0021:02:08.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0021:02:09.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0021:02:0a.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0021:02:0b.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0021:02:0c.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0021:0d:00.0 USB controller: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller (rev 02)
0021:0e:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9235 PCIe 2.0 x2 4-port SATA 6 Gb/s Controller (rev 11)
0021:0f:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
0021:10:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
0022:00:00.0 PCI bridge: IBM Device 03dc
0022:01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0022:01:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0022:01:00.2 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0022:01:00.3 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
root@ltc84-pkvm1:~#

OPAL Information:

fru print 47
 Product Name          : OpenPOWER Firmware
 Product Version       : IBM-firestone-ibm-OP8_1.12_2.64
 Product Extra         : 	op-build-f33c80d
 Product Extra         : 	buildroot-211bd05
 Product Extra         : 	skiboot-5.4.3
 Product Extra         : 	hostboot-2eb7706-740e8ce
 Product Extra         : 	linux-4.4.30-openpower1-45b5734
 Product Extra         : 	petitboot-v1.3.2-954eb67
 Product Extra         : 	firestone-xml-2494a43
 Product Extra         : 	occ-d7efe30
 Product Extra         : 	hostboot-bin