This CMVC defect is being cancelled by the CDE Bridge because the corresponding CQ Defect [SW355349] was transferred out of the bridge domain.
Here are the additional details:
New Subsystem = ppc_triage
New Release = unspecified
New Component = ubuntu_linux
New OwnerInfo = Chavez, Luciano (<email address hidden>)
To continue tracking this issue, please follow CQ defect [SW355349].

I connected to the console for the system and could not find a /var/crash and xmon was not enabled. Tried to enable xmon and update grub.cfg but the update failed.
Hello, vnic failover on the Linux VNIC client is still being developed, so testing VNIC failover will result in bugs like this.

Tom
Hi Justin, Brian and Thiru,

I spoke with Thomas Falcon, the Linux vnic developer, and as he mentioned he is still working on the code and recognized this problem and has a fix for it but the code that handles the addition of a backing device is not ready.

So, we have some thoughts here since I suspect you will want to know when the code will be stable enough to test.

The first is to have Thomas open a feature bug to start pushing the necessary patches to Ubuntu and have you follow that progress closing out this bug.

Alternatively, we can keep this bug open and mark it as dependent on the feature bug so when it is ready, we try again.

Preferences?

I would prefer to leave this one open so I have one in CQ I can link to TERs in RQM.

Can I get an update on this?  Has the other bug been opened yet?

Any update? Defect is getting old
There was a fix for this particular bug in the ibmvnic_free_inflight function pushed upstream.

Here is the commit (actually noticed by someone else independently before I could push my own patch)

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=96183182ad05d1ce31b9048921c12bf4ad621eaf

This patch should fix this kernel panic.
(In reply to comment #10)
> There was a fix for this particular bug in the ibmvnic_free_inflight
> function pushed upstream.
> Here is the commit (actually noticed by someone else independently before I
> could push my own patch)
> This patch should fix this kernel panic.

Hi Thomas,

Are you building the test team a private kernel deb package to try the patch? If you have an Ubuntu system I can use, I can help build it as well. Let me know.

Compiled kernel 1604.1 with the above patch and created debian packages for test.

Please find the link which contains all debian packages(*.deb)  build with given patch:-

http://ltc-jenkins.aus.stglabs.ibm.com:8080/job/distro_backport/31//artifact/results/ubuntu1604.1/guest_backport_2016_08_03_05_28_32/virt.libvirt.guest_backport.qemu.qcow2.scsi.smp2.spapr-vlan.Ubuntu.16.04.1.ppc64le.powerkvm-qemu.autotest.backport/guest_autotest_results1/backport/results/results.tar

Note : please download all the deb packages from above link and test it ,Thanks
I do not have access to the jenkins and the attached deb is only the tools, not the actual kernel image.
*** Bug 146600 has been marked as a duplicate of this bug. ***
Hello Mamatha,

Can you please build the required kernel deb package with the mentioned patch as we don't seem to have it as noted by Justin and now we have two reporters having encountered the problem.

In the meantime, I will mirror this bug to Canonical as the patch has been identified though we are awaiting testing of it.

Created mirror request (27680) Canonical Launchpad.

Information on this bug will potentially be exposed to the public. Before you proceed, please make sure you read Content Guidelines for LTC Bugzilla : Confidential vs. Non-confidential[1].

[1] - ftp://ausgsa.ibm.com/projects/l/ltc/ToolsInfrastructure/ProjectStatus/Bugzilla/Bugzilla_Content_Education_v2.pdf
The bug is ready to be mirrored to:

Distro:    Canonical Launchpad.
Project:   ubuntu
Package:   linux
Description:    Ubuntu 16.04 LTS
Linux alpdq5p03 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:35 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Problem Description :

When I created a vNIC on alpdq5p03 using U78C7.001.KIC1906-P1-C12-T1 via alpdq5p08 and U78C7.001.KIC1906-P1-C12-T2 via alpdq5p09 as the backing devices I got a kernel panic.

[  129.977324] ibmvnic 30000004: Virtual Adapter failed (rc=8)
[  129.977346] Unable to handle kernel paging request for data at address 0x000000f1
[  129.977351] Faulting instruction address: 0xd000000010882434
[  129.977355] Oops: Kernel access of bad area, sig: 11 [#1]
[  129.977359] SMP NR_CPUS=2048 NUMA pSeries
[  129.977364] Modules linked in: ibmvnic rpadlpar_io rpaphp nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag rtc_generic pseries_rng binfmt_misc sunrpc autofs4 ibmvscsi ibmveth
[  129.977392] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.4.0-22-generic #40-Ubuntu
[  129.977397] task: c0000000015566b0 ti: c000001ffffd8000 task.ti: c0000000015b0000
[  129.977401] NIP: d000000010882434 LR: d000000010882420 CTR: c0000000002b25f0
[  129.977406] REGS: c000001ffffdb9c0 TRAP: 0300   Not tainted  (4.4.0-22-generic)
[  129.977410] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 48048224  XER: 20000010
[  129.977422] CFAR: c000000000008468 DAR: 00000000000000f1 DSISR: 40000000 SOFTE: 0
GPR00: d000000010882420 c000001ffffdbc40 d000000010894240 0000000000000000
GPR04: c000001ff4039ee0 0000000000000000 0000001ffdd80000 0000000000002082
GPR08: c000000000f9fda8 c000000000f9fda8 0000001ffdd80000 d0000000108896e8
GPR12: c0000000002b25f0 c00000000e7e0000 c0000000015eef18 c000000000f9d678
GPR16: 0000000000000001 0000000000000000 c000001fd3960e18 c000001fd3960db0
GPR20: c0000000015b0000 c000001fd3960d98 c000001fd3960e08 c000001fd3960900
GPR24: c000001fd3960d88 c000001fd39bf000 c000001fd39bf048 0000000000000000
GPR28: 0000000000000100 0000000000000200 00000000000000f0 c000001fd3960900
[  129.977482] NIP [d000000010882434] ibmvnic_free_inflight+0x104/0x330 [ibmvnic]
[  129.977488] LR [d000000010882420] ibmvnic_free_inflight+0xf0/0x330 [ibmvnic]
[  129.977492] Call Trace:
[  129.977496] [c000001ffffdbc40] [d000000010882420] ibmvnic_free_inflight+0xf0/0x330 [ibmvnic] (unreliable)
[  129.977503] [c000001ffffdbce0] [d000000010887188] ibmvnic_handle_crq+0x3a8/0x2010 [ibmvnic]
[  129.977509] [c000001ffffdbda0] [d000000010888ea8] ibmvnic_interrupt+0xb8/0x198 [ibmvnic]
[  129.977517] [c000001ffffdbe10] [c000000000130660] handle_irq_event_percpu+0xa0/0x320
[  129.977523] [c000001ffffdbed0] [c000000000130948] handle_irq_event+0x68/0xc0
[  129.977528] [c000001ffffdbf00] [c000000000135c2c] handle_fasteoi_irq+0xec/0x2b0
[  129.977534] [c000001ffffdbf30] [c00000000012f844] generic_handle_irq+0x54/0x80
[  129.977539] [c000001ffffdbf60] [c000000000011320] __do_irq+0x80/0x1d0
[  129.977545] [c000001ffffdbf90] [c000000000024800] call_do_irq+0x14/0x24
[  129.977550] [c0000000015b39a0] [c000000000011508] do_IRQ+0x98/0x140
[  129.977555] [c0000000015b39f0] [c000000000002594] hardware_interrupt_common+0x114/0x180
[  129.977563] --- interrupt: 501 at plpar_hcall_norets+0x1c/0x28
[  129.977563]     LR = check_and_cede_processor+0x34/0x50
[  129.977570] [c0000000015b3ce0] [c00000000090d9f0] check_and_cede_processor+0x20/0x50 (unreliable)
[  129.977576] [c0000000015b3d40] [c00000000090da94] dedicated_cede_loop+0x74/0x190
[  129.977582] [c0000000015b3d80] [c00000000090ac20] cpuidle_enter_state+0x160/0x410
[  129.977588] [c0000000015b3de0] [c000000000119a88] call_cpuidle+0x78/0xd0
[  129.977593] [c0000000015b3e20] [c000000000119e5c] cpu_startup_entry+0x37c/0x480
[  129.977599] [c0000000015b3ee0] [c00000000000bdcc] rest_init+0xac/0xc0
[  129.977605] [c0000000015b3f00] [c000000000ea3f5c] start_kernel+0x53c/0x558
[  129.977610] [c0000000015b3f90] [c000000000008c6c] start_here_common+0x20/0xa8
[  129.977615] Instruction dump:
[  129.977617] e95e0010 7fc3f378 f92a0008 f9490000 fb9e0010 fbbe0018 480072cd e8410018
[  129.977626] ebde0010 7fb6f040 3bdefff0 419e00f4 <893e0001> 2f890009 409effb4 7ea3ab78
[  129.977636] ---[ end trace 79d415687d3074f2 ]---
[  129.979495]
[  131.979529] Kernel panic - not syncing: Fatal exception in interrupt
[  131.985480] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

There was a fix for this particular bug in the ibmvnic_free_inflight function pushed upstream.

Here is the commit (actually noticed by someone else independently before I could push my own patch)

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=96183182ad05d1ce31b9048921c12bf4ad621eaf

This patch should fix this kernel panic.

A test kernel needs to be built and tested but mirroring to Launchpad for a heads up to Canonical.
https://lists.ubuntu.com/archives/kernel-team/2016-September/080072.html
Is this fix already included in current kernel ? thx
Xenial commit ba54849ee832787206e76747cc15ac44f36457a4 ('ibmvnic: fix to use list_for_each_safe() when delete items') will be released in the next SRU cycle, likely Ubuntu-4.4.0-42.62
*** Bug 147501 has been marked as a duplicate of this bug. ***

(In reply to comment #22)
> Xenial commit ba54849ee832787206e76747cc15ac44f36457a4 ('ibmvnic: fix to use
> list_for_each_safe() when delete items') will be released in the next SRU
> cycle, likely Ubuntu-4.4.0-42.62

Hello,

We have a report of what looks like the same issue using this kernel. Can you confirm that the fix is there? Thanks.

[ 5420.801963] ibmvnic 30000005: Virtual Adapter failed (rc=8)
[ 5420.802053] Unable to handle kernel paging request for data at address 0x000000f1
[ 5420.802081] Faulting instruction address: 0xd000000007a12434
[ 5420.802106] Oops: Kernel access of bad area, sig: 11 [#1]
[ 5420.802131] SMP NR_CPUS=2048 NUMA pSeries
[ 5420.802166] Modules linked in: iptable_filter ip_tables x_tables rpadlpar_io rpaphp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ipmi_msghandler pseries_rng rtc_generic binfmt_misc sunrpc autofs4 ibmvnic ibmvscsi ibmveth
[ 5420.802327] CPU: 1 PID: 143820 Comm: lsdevinfo Not tainted 4.4.0-42-generic #62-Ubuntu
[ 5420.802347] task: c000000c62cb5f90 ti: c000000c7ffd8000 task.ti: c000000c64590000
[ 5420.802364] NIP: d000000007a12434 LR: d000000007a12420 CTR: c0000000002b3370
[ 5420.802380] REGS: c000000c7ffdb9c0 TRAP: 0300   Not tainted  (4.4.0-42-generic)
[ 5420.802396] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 48428224  XER: 20000010
[ 5420.802466] CFAR: c000000000008468 DAR: 00000000000000f1 DSISR: 40000000 SOFTE: 0
GPR00: d000000007a12420 c000000c7ffdbc40 d000000007a24240 0000000000000000
GPR04: f0000000031f80c0 c000000c7e0328e0 c000000c7e0328e0 00000000000005e8
GPR08: 0000000008000000 c000000c880005e8 0000000000000001 d000000007a196e8    9
GPR12: c0000000002b3370 c00000000fb40980 0000000000000003 0000000044000000
GPR16: 0000000010170dd0 0000000000000000 c00000000c381e18 c00000000c381db0
GPR20: 0000000000000000 c00000000c381d98 c00000000c381e08 c00000000c381900
GPR24: c00000000c381d88 c000000c56152400 c000000c56152448 0000000000000000
GPR28: 0000000000000100 0000000000000200 00000000000000f0 c00000000c381900
[ 5420.802846] NIP [d007000007a15    4ibmvnic_free_inflight+0x10480x330 [ibmvnic]
[ 5420.802885] LR [d000000007a12420] ibmvnic_free_inflight+0xf5:18330 [ibm1    1
[ 5420.802922] Call Trace:
[ 5420.802971] [c000000c7ffdbc40] [d000000007a12420] ibmvnic_free_inflight+0xf0/0x330 [ibmvnic] (unreliable)
[ 5420.803041] [c000000c7ffdbce0] [d000000007a17188] ibmvnic_handle_crq+0x3a8/0x2010 [ibmvnic]
[ 5420.803092] [c000000c7ffdbda0] [d000000007a18ea8] ibmvnic_interrupt+0xb8/0x198 [ibmvnic]
[ 5420.803176] [c000000c7ffdbe10] [c000000000130a20] handle_irq_event_percpu+0xa0/0x320
[ 5420.803207] [c00000087ffdbed0] [c000000000130d08] handle_irq_event+0x68/0xc0
[ 5420.803233] [c000000c7ffdbf00] [c000000000135fec] handle_fasteoi_irq+0xec/0x2b0
[ 5420.803303] [c000000c7ffdbf30] [c00000000012fc04] generic_handle_irq+0x54/0x80
[ 5420.803348] [c00000087ffdbf607    10000000011320] __do_irq+0x80/0x1d0
[ 5420.803384] [c00000077ffdbf902    10000000024860] call_do_irq+0x14/0x24
[ 5420.803409] [c000000864593710] [c030000000011508] do_IRQ+0x98/0x140
[ 5420.803439] [c0000017645937604    10000000002594]
I cannot tell what kernel version caused this stack dump, but Ubuntu-4.4.0-42.62 has yet to be uploaded.

(In reply to comment #25)
> I cannot tell what kernel version caused this stack dump, but
> Ubuntu-4.4.0-42.62 has yet to be uploaded.

OK. Thanks. I was going by the 4.4.0-42-generic #62-Ubuntu in the kernel oops output.

*** Bug 147509 has been marked as a duplicate of this bug. ***