STC860:alpine-pdq:alpdq5p03: kernel panic when adding vnic

[bugproxy]

Description: Ubuntu 16.04 LTS
Linux alpdq5p03 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:35 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Problem Description :

When I created a vNIC on alpdq5p03 using U78C7.001.KIC1906-P1-C12-T1 via alpdq5p08 and U78C7.001.KIC1906-P1-C12-T2 via alpdq5p09 as the backing devices I got a kernel panic.

[ 129.977324] ibmvnic 30000004: Virtual Adapter failed (rc=8)
[ 129.977346] Unable to handle kernel paging request for data at address 0x000000f1
[ 129.977351] Faulting instruction address: 0xd000000010882434
[ 129.977355] Oops: Kernel access of bad area, sig: 11 [#1]
[ 129.977359] SMP NR_CPUS=2048 NUMA pSeries
[ 129.977364] Modules linked in: ibmvnic rpadlpar_io rpaphp nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag rtc_generic pseries_rng binfmt_misc sunrpc autofs4 ibmvscsi ibmveth
[ 129.977392] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.4.0-22-generic #40-Ubuntu
[ 129.977397] task: c0000000015566b0 ti: c000001ffffd8000 task.ti: c0000000015b0000
[ 129.977401] NIP: d000000010882434 LR: d000000010882420 CTR: c0000000002b25f0
[ 129.977406] REGS: c000001ffffdb9c0 TRAP: 0300 Not tainted (4.4.0-22-generic)
[ 129.977410] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 48048224 XER: 20000010
[ 129.977422] CFAR: c000000000008468 DAR: 00000000000000f1 DSISR: 40000000 SOFTE: 0
GPR00: d000000010882420 c000001ffffdbc40 d000000010894240 0000000000000000
GPR04: c000001ff4039ee0 0000000000000000 0000001ffdd80000 0000000000002082
GPR08: c000000000f9fda8 c000000000f9fda8 0000001ffdd80000 d0000000108896e8
GPR12: c0000000002b25f0 c00000000e7e0000 c0000000015eef18 c000000000f9d678
GPR16: 0000000000000001 0000000000000000 c000001fd3960e18 c000001fd3960db0
GPR20: c0000000015b0000 c000001fd3960d98 c000001fd3960e08 c000001fd3960900
GPR24: c000001fd3960d88 c000001fd39bf000 c000001fd39bf048 0000000000000000
GPR28: 0000000000000100 0000000000000200 00000000000000f0 c000001fd3960900
[ 129.977482] NIP [d000000010882434] ibmvnic_free_inflight+0x104/0x330 [ibmvnic]
[ 129.977488] LR [d000000010882420] ibmvnic_free_inflight+0xf0/0x330 [ibmvnic]
[ 129.977492] Call Trace:
[ 129.977496] [c000001ffffdbc40] [d000000010882420] ibmvnic_free_inflight+0xf0/0x330 [ibmvnic] (unreliable)
[ 129.977503] [c000001ffffdbce0] [d000000010887188] ibmvnic_handle_crq+0x3a8/0x2010 [ibmvnic]
[ 129.977509] [c000001ffffdbda0] [d000000010888ea8] ibmvnic_interrupt+0xb8/0x198 [ibmvnic]
[ 129.977517] [c000001ffffdbe10] [c000000000130660] handle_irq_event_percpu+0xa0/0x320
[ 129.977523] [c000001ffffdbed0] [c000000000130948] handle_irq_event+0x68/0xc0
[ 129.977528] [c000001ffffdbf00] [c000000000135c2c] handle_fasteoi_irq+0xec/0x2b0
[ 129.977534] [c000001ffffdbf30] [c00000000012f844] generic_handle_irq+0x54/0x80
[ 129.977539] [c000001ffffdbf60] [c000000000011320] __do_irq+0x80/0x1d0
[ 129.977545] [c000001ffffdbf90] [c000000000024800] call_do_irq+0x14/0x24
[ 129.977550] [c0000000015b39a0] [c000000000011508] do_IRQ+0x98/0x140
[ 129.977555] [c0000000015b39f0] [c000000000002594] hardware_interrupt_common+0x114/0x180
[ 129.977563] --- interrupt: 501 at plpar_hcall_norets+0x1c/0x28
[ 129.977563] LR = check_and_cede_processor+0x34/0x50
[ 129.977570] [c0000000015b3ce0] [c00000000090d9f0] check_and_cede_processor+0x20/0x50 (unreliable)
[ 129.977576] [c0000000015b3d40] [c00000000090da94] dedicated_cede_loop+0x74/0x190
[ 129.977582] [c0000000015b3d80] [c00000000090ac20] cpuidle_enter_state+0x160/0x410
[ 129.977588] [c0000000015b3de0] [c000000000119a88] call_cpuidle+0x78/0xd0
[ 129.977593] [c0000000015b3e20] [c000000000119e5c] cpu_startup_entry+0x37c/0x480
[ 129.977599] [c0000000015b3ee0] [c00000000000bdcc] rest_init+0xac/0xc0
[ 129.977605] [c0000000015b3f00] [c000000000ea3f5c] start_kernel+0x53c/0x558
[ 129.977610] [c0000000015b3f90] [c000000000008c6c] start_here_common+0x20/0xa8
[ 129.977615] Instruction dump:
[ 129.977617] e95e0010 7fc3f378 f92a0008 f9490000 fb9e0010 fbbe0018 480072cd e8410018
[ 129.977626] ebde0010 7fb6f040 3bdefff0 419e00f4 <893e0001> 2f890009 409effb4 7ea3ab78
[ 129.977636] ---[ end trace 79d415687d3074f2 ]---
[ 129.979495]
[ 131.979529] Kernel panic - not syncing: Fatal exception in interrupt
[ 131.985480] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

There was a fix for this particular bug in the ibmvnic_free_inflight function pushed upstream.

Here is the commit (actually noticed by someone else independently before I could push my own patch)

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=96183182ad05d1ce31b9048921c12bf4ad621eaf

This patch should fix this kernel panic.

A test kernel needs to be built and tested but mirroring to Launchpad for a heads up to Canonical.

[Tim Gardner (timg-tpi)]

https://lists.ubuntu.com/archives/kernel-team/2016-September/080072.html

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-10-06 04:54 EDT-------
Is this fix already included in current kernel ? thx

[Tim Gardner (timg-tpi)]

Xenial commit ba54849ee832787206e76747cc15ac44f36457a4 ('ibmvnic: fix to use list_for_each_safe() when delete items') will be released in the next SRU cycle, likely Ubuntu-4.4.0-42.62

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-10-11 16:14 EDT-------
*** Bug 147501 has been marked as a duplicate of this bug. ***

------- Comment From <email address hidden> 2016-10-11 16:16 EDT-------
(In reply to comment #22)
> Xenial commit ba54849ee832787206e76747cc15ac44f36457a4 ('ibmvnic: fix to use
> list_for_each_safe() when delete items') will be released in the next SRU
> cycle, likely Ubuntu-4.4.0-42.62

Hello,

We have a report of what looks like the same issue using this kernel. Can you confirm that the fix is there? Thanks.

[ 5420.801963] ibmvnic 30000005: Virtual Adapter failed (rc=8)
[ 5420.802053] Unable to handle kernel paging request for data at address 0x000000f1
[ 5420.802081] Faulting instruction address: 0xd000000007a12434
[ 5420.802106] Oops: Kernel access of bad area, sig: 11 [#1]
[ 5420.802131] SMP NR_CPUS=2048 NUMA pSeries
[ 5420.802166] Modules linked in: iptable_filter ip_tables x_tables rpadlpar_io rpaphp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ipmi_msghandler pseries_rng rtc_generic binfmt_misc sunrpc autofs4 ibmvnic ibmvscsi ibmveth
[ 5420.802327] CPU: 1 PID: 143820 Comm: lsdevinfo Not tainted 4.4.0-42-generic #62-Ubuntu
[ 5420.802347] task: c000000c62cb5f90 ti: c000000c7ffd8000 task.ti: c000000c64590000
[ 5420.802364] NIP: d000000007a12434 LR: d000000007a12420 CTR: c0000000002b3370
[ 5420.802380] REGS: c000000c7ffdb9c0 TRAP: 0300 Not tainted (4.4.0-42-generic)
[ 5420.802396] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 48428224 XER: 20000010
[ 5420.802466] CFAR: c000000000008468 DAR: 00000000000000f1 DSISR: 40000000 SOFTE: 0
GPR00: d000000007a12420 c000000c7ffdbc40 d000000007a24240 0000000000000000
GPR04: f0000000031f80c0 c000000c7e0328e0 c000000c7e0328e0 00000000000005e8
GPR08: 0000000008000000 c000000c880005e8 0000000000000001 d000000007a196e8 9
GPR12: c0000000002b3370 c00000000fb40980 0000000000000003 0000000044000000
GPR16: 0000000010170dd0 0000000000000000 c00000000c381e18 c00000000c381db0
GPR20: 0000000000000000 c00000000c381d98 c00000000c381e08 c00000000c381900
GPR24: c00000000c381d88 c000000c56152400 c000000c56152448 0000000000000000
GPR28: 0000000000000100 0000000000000200 00000000000000f0 c00000000c381900
[ 5420.802846] NIP [d007000007a15 4ibmvnic_free_inflight+0x10480x330 [ibmvnic]
[ 5420.802885] LR [d000000007a12420] ibmvnic_free_inflight+0xf5:18330 [ibm1 1
[ 5420.802922] Call Trace:
[ 5420.802971] [c000000c7ffdbc40] [d000000007a12420] ibmvnic_free_inflight+0xf0/0x330 [ibmvnic] (unreliable)
[ 5420.803041] [c000000c7ffdbce0] [d000000007a17188] ibmvnic_handle_crq+0x3a8/0x2010 [ibmvnic]
[ 5420.803092] [c000000c7ffdbda0] [d000000007a18ea8] ibmvnic_interrupt+0xb8/0x198 [ibmvnic]
[ 5420.803176] [c000000c7ffdbe10] [c000000000130a20] handle_irq_event_percpu+0xa0/0x320
[ 5420.803207] [c00000087ffdbed0] [c000000000130d08] handle_irq_event+0x68/0xc0
[ 5420.803233] [c000000c7ffdbf00] [c000000000135fec] handle_fasteoi_irq+0xec/0x2b0
[ 5420.803303] [c000000c7ffdbf30] [c00000000012fc04] generic_handle_irq+0x54/0x80
[ 5420.803348] [c00000087ffdbf607 10000000011320] __do_irq+0x80/0x1d0
[ 5420.803384] [c00000077ffdbf902 10000000024860] call_do_irq+0x14/0x...

[Tim Gardner (timg-tpi)]

I cannot tell what kernel version caused this stack dump, but Ubuntu-4.4.0-42.62 has yet to be uploaded.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-10-12 10:47 EDT-------
(In reply to comment #25)
> I cannot tell what kernel version caused this stack dump, but
> Ubuntu-4.4.0-42.62 has yet to be uploaded.

OK. Thanks. I was going by the 4.4.0-42-generic #62-Ubuntu in the kernel oops output.

------- Comment From <email address hidden> 2016-10-12 10:51 EDT-------
*** Bug 147509 has been marked as a duplicate of this bug. ***

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-10-12 11:10 EDT-------
<Automated Update> The severity of defect SW355349 was increased from 2 to 1 because defect SW368742 was rejected as the duplicate of defect SW355349 and the severity of defect SW368742 was higher than 2

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-10-12 11:26 EDT-------
<Automated Update> The severity of defect SW355349 was increased from 2 to 1 because defect SW368742 was rejected as the duplicate of defect SW355349 and the severity of defect SW368742 was higher than 2

[Tim Gardner (timg-tpi)]

<email address hidden> - I was looking at the stack dump in the description, and not the one in #4. It appears that the fix commit hasn't been released in a version yet. The earliest that it will get packaged is Ubuntu-4.4.0-44.64

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-10-12 14:29 EDT-------
Ok. Thanks!

[Seth Forshee (sforshee)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[bugproxy (bugproxy)]

Reposting internal comment to make it visible to Canonical and hopefully with Thomas' help determine if this should be marked as verification-failed-xenial and additional patches will be necessary.

> Att: Linux team,
>
> I installed the latest ubuntu package. Here is the details.
>
> hmc alphmc4
> cec : rdx2fp1
> lpar : rdx2p03
>
> # uname -a
> Linux rdx2p03 4.4.0-44-generic #64-Ubuntu SMP Mon Oct 17 08:43:50 UTC 2016
> ppc64le ppc64le ppc64le GNU/Linux
>
> (0) root @ rdx2p03: /root
> # apt-get update
> Hit:1 http://us.ports.ubuntu.com/ubuntu-ports xenial InRelease
> Hit:2 http://us.ports.ubuntu.com/ubuntu-ports xenial-updates InRelease
> Hit:3 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease
> Hit:4 http://us.ports.ubuntu.com/ubuntu-ports xenial-backports InRelease
> Hit:5 http://ports.ubuntu.com/ubuntu-ports xenial-proposed InRelease
> Reading package lists... Done
>
> (0) root @ rdx2p03: /root
> # apt-get upgrade
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Calculating upgrade... Done
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>
> (0) root @ rdx2p03: /root
> # apt-get dist-upgrade
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Calculating upgrade... Done
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>
> (0) root @ rdx2p03: /root
>
> I was able to create vNIC and backing device.
>
> No response for lspci.
>
> # ip link
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode
> DEFAULT group default qlen 1
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> 2: ibmveth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UNKNOWN mode DEFAULT group default qlen 1000
> link/ether 06:73:67:dd:71:02 brd ff:ff:ff:ff:ff:ff
> 3: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT
> group default qlen 1000
> link/ether 06:73:67:dd:71:07 brd ff:ff:ff:ff:ff:ff
> 4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT
> group default qlen 1000
> link/ether 06:73:67:dd:71:08 brd ff:ff:ff:ff:ff:ff
>
> But build_net help failed with oops msg
>
> Automation script will now try to detect your network topology.
> In case of errors manually edit bpt file and run build_net bpt to setup
> hxecom networks
> [ 140.400376] Unable to handle kernel paging request for data at address
> 0x00000000
> [ 140.400386] Faulting instruction address: 0xd0000000017b6534
> [ 140.400391] Oops: Kernel access of bad area, sig: 11 [#1]
> [ 140.400393] SMP NR_CPUS=2048 NUMA pSeries
> [ 140.400398] ibmvnic 30000007: Error 7 in QUERY_MAP_RSP
> [ 140.400403] Modules linked in: rpadlpar_io rpaphp dccp_diag dccp tcp_diag
> udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc
> pseries_rng rtc_generic dm_round_robin sunrpc dm_multipath autofs4 ibmvnic
> ibmveth ibmvscsi
> [ 140.400421] CPU: 5 PID: 13808 Comm: ip Tainted: G W
> 4.4.0-44-generic #64-Ubuntu
> [ 140.400424] task: c00000014b4d26e0 ti: c00000014b5b4000 task.ti:
> c00000014b5b4000
> [ 140.400427] NIP: d0000000017b6534 LR: d0000000017b65f0 CTR:
> 00...

[bugproxy (bugproxy)]

cde00 (<email address hidden>) deleted native attachment sosreport.tar.xz on 2016-10-20 09:13:22

cde00 (<email address hidden>) added native attachment /tmp/AIXOS05875122/sosreport.tar.xz on 2016-10-20 09:13:25

[Tim Gardner (timg-tpi)]

Marking verification failed. The fix commit did not fix the panic, nor did it make it any worse.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-47.68

---------------
linux (4.4.0-47.68) xenial; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1636941

  * Add a driver for Amazon Elastic Network Adapters (ENA) (LP: #1635721)
    - lib/bitmap.c: conversion routines to/from u32 array
    - net: ethtool: add new ETHTOOL_xLINKSETTINGS API
    - net: ena: Add a driver for Amazon Elastic Network Adapters (ENA)
    - [config] enable CONFIG_ENA_ETHERNET=m (Amazon ENA driver)

  * unexpectedly large memory usage of mounted snaps (LP: #1636847)
    - [Config] switch squashfs to single threaded decode

 -- Kamal Mostafa <email address hidden> Wed, 26 Oct 2016 10:47:55 -0700

[bugproxy (bugproxy)]

==== State: Working by: carp on 08 November 2016 14:50:18 ====

#=#=# 2016-11-08 14:50:16 (CST) #=#=#
New Fix_Potential = [F860.20W]
#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#=#

[bugproxy (bugproxy)]

==== State: Working by: justinmc on 30 May 2017 06:50:20 ====

The CQ side is badly desynced from the BZ side. This is fixed and is marked as closed in BZ, but is still showing as Working in CQ. Assuming this comment actually makes it to BZ, can we put the BZ side back in verify to see if that makes the CQ side behave?

[bugproxy (bugproxy)]

==== State: Verify by: justinmc on 30 May 2017 08:43:46 ====

Okay, thanks Luciano. I think I understand the state on the CQ side now. I need Peruvaje, Manohara (<email address hidden>) to accept the verify record he has now and then CQ should close this again. I've sent him a note so hopefully this will be resolved tomorrow.

[bugproxy (bugproxy)]

This CMVC defect is being cancelled by the CDE Bridge because the corresponding CQ Defect [SW355349] was transferred out of the bridge domain.
Here are the additional details:
New Subsystem = ppc_triage
New Release = unspecified
New Component = ubuntu_linux
New OwnerInfo = Chavez, Luciano (<email address hidden>)
To continue tracking this issue, please follow CQ defect [SW355349].

I connected to the console for the system and could not find a /var/crash and xmon was not enabled. Tried to enable xmon and update grub.cfg but the update failed.
Hello, vnic failover on the Linux VNIC client is still being developed, so testing VNIC failover will result in bugs like this.

Tom
Hi Justin, Brian and Thiru,

I spoke with Thomas Falcon, the Linux vnic developer, and as he mentioned he is still working on the code and recognized this problem and has a fix for it but the code that handles the addition of a backing device is not ready.

So, we have some thoughts here since I suspect you will want to know when the code will be stable enough to test.

The first is to have Thomas open a feature bug to start pushing the necessary patches to Ubuntu and have you follow that progress closing out this bug.

Alternatively, we can keep this bug open and mark it as dependent on the feature bug so when it is ready, we try again.

Preferences?

I would prefer to leave this one open so I have one in CQ I can link to TERs in RQM.

Can I get an update on this? Has the other bug been opened yet?

Any update? Defect is getting old
There was a fix for this particular bug in the ibmvnic_free_inflight function pushed upstream.

Here is the commit (actually noticed by someone else independently before I could push my own patch)

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=96183182ad05d1ce31b9048921c12bf4ad621eaf

This patch should fix this kernel panic.
(In reply to comment #10)
> There was a fix for this particular bug in the ibmvnic_free_inflight
> function pushed upstream.
> Here is the commit (actually noticed by someone else independently before I
> could push my own patch)
> This patch should fix this kernel panic.

Hi Thomas,

Are you building the test team a private kernel deb package to try the patch? If you have an Ubuntu system I can use, I can help build it as well. Let me know.

Compiled kernel 1604.1 with the above patch and created debian packages for test.

Please find the link which contains all debian packages(*.deb) build with given patch:-

http://ltc-jenkins.aus.stglabs.ibm.com:8080/job/distro_backport/31//artifact/results/ubuntu1604.1/guest_backport_2016_08_03_05_28_32/virt.libvirt.guest_backport.qemu.qcow2.scsi.smp2.spapr-vlan.Ubuntu.16.04.1.ppc64le.powerkvm-qemu.autotest.backport/guest_autotest_results1/backport/results/results.tar

Note : please download all the deb packages from above link and test it ,Thanks
I do not have access to the jenkins and the attached deb is only the tools, not the actual kernel image.
*** Bug 146600 has been marked as a duplicate of this bug. ***
Hello Mamatha,

Can you please build the required kernel deb package with the mentioned patch as we don't seem to have it as noted by Justin and now w...