ISST-LTE:pKVM:Ubuntu1804: rcu_sched self-detected stall on CPU follow by CPU ATTEMPT TO RE-ENTER FIRMWARE!

[bugproxy]

== Comment: #0 - Application Cdeadmin <email address hidden> - 2018-03-20 14:10:53 ==

== Comment: #1 - Application Cdeadmin <email address hidden> - 2018-03-20 14:10:54 ==
== Comment: #2 - Application Cdeadmin <email address hidden> - 2018-03-20 14:10:56 ==
------- Comment From dougmill-ibm 2018-03-20 13:51:47 EDT -------
This problem is not tied to a Linux distro. It will be fixed in firmware, as I understand it. Let us close any redundant issues for this same problem. Mark them as duplicate.

== Comment: #3 - Application Cdeadmin <email address hidden> - 2018-03-20 15:50:54 ==
------- Comment From mzipse 2018-03-20 15:44:26 EDT -------
@stewart-ibm @svaidy , I need to you take a first look. The stop fixes that Vaidy had previously highlighted in a recent note are included in the 3/15 PNOR.

== Comment: #5 - Application Cdeadmin <email address hidden> - 2018-04-04 16:10:56 ==
------- Comment From haochanh 2018-04-04 16:04:07 EDT -------
We update to 0330, bmc=1.18, then we hit bug 1134. Currently we are running with disable stop5 but still see the watchdog: hard lockup.
After 2 hours of test run, I am seeing the "Watchdog: Lockup' and "became unstuck"
****************************
[Wed Apr 4 13:38:25 2018] Watchdog CPU:42 Hard LOCKUP
[Wed Apr 4 13:38:25 2018] Modules linked in: vhost_net vhost macvtap macvlan tap xfs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 nfs fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) cxl pnv_php mlx4_en(OE) mlx4_ib(OE) ib_core(OE) mlx4_core(OE) devlink mlx_compat(OE) kvm_hv kvm binfmt_misc dm_service_time dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua input_leds joydev mac_hid idt_89hpesx ipmi_powernv
[Wed Apr 4 13:38:25 2018] vmx_crypto ipmi_devintf at24 ofpart uio_pdrv_genirq cmdlinepart uio powernv_flash ipmi_msghandler mtd crct10dif_vpmsum opal_prd ibmpowernv nfsd sch_fq_codel auth_rpcgss nfs_acl lockd grace sunrpc knem(OE) ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq ses enclosure scsi_transport_sas hid_generic usbhid hid lpfc ast i2c_algo_bit ttm drm_kms_helper nvmet_fc syscopyarea sysfillrect nvmet sysimgblt fb_sys_fops nvme_fc nvme_fabrics crc32c_vpmsum drm i40e scsi_transport_fc aacraid [last unloaded: mlxfw]
[Wed Apr 4 13:38:25 2018] CPU: 42 PID: 0 Comm: swapper/42 Tainted: G OE 4.15.0-12-generic #13
[Wed Apr 4 13:38:25 2018] NIP: c0000000000a3ca4 LR: c0000000000a3ca4 CTR: c000000000008000
[Wed Apr 4 13:38:25 2018] REGS: c000000ff596fc40 TRAP: 0100 Tainted: G OE (4.15.0-12-generic)
[Wed Apr 4 13:38:25 2018] MSR: 9000000000001033 <SF,HV,ME,IR,DR,RI,LE> CR: 24004482 XER: 20040000
[Wed Apr 4 13:38:25 2018] CFAR: c000000ff596fda0 SOFTE: 42
                           GPR00: c0000000000a3ca4 c000000ff596fda0 c0000000016eb200 c000000ff596fc40
                           GPR04: b000000000001033 c0000000000a3690 0000000024004484 0000000ffa450000
                           GPR08: 0000000000000001 c000000000d10ed8 00000000000000ff 0000000000000000
                           GPR12: 9000000000121033 c000000007a3ce00 c000000ff596ff90 0000000000000000
                           GPR16: 0000000000000000 c000000000047840 c000000000047810 c0000000011b5380
                           GPR20: 0000000000000800 c000000001722484 000000000000002a 0000000000000000
                           GPR24: 00000000000000a8 0000000000000007 0000000000000000 0000000000000007
                           GPR28: c00000000161d270 c000000ffb666fd8 c00000000161d528 0000000000000007
[Wed Apr 4 13:38:25 2018] NIP [c0000000000a3ca4] power9_idle_type+0x24/0x40
[Wed Apr 4 13:38:25 2018] LR [c0000000000a3ca4] power9_idle_type+0x24/0x40
[Wed Apr 4 13:38:25 2018] Call Trace:
[Wed Apr 4 13:38:25 2018] [c000000ff596fda0] [c0000000000a3ca4] power9_idle_type+0x24/0x40 (unreliable)
[Wed Apr 4 13:38:25 2018] [c000000ff596fdc0] [c000000000ad1240] stop_loop+0x40/0x5c
[Wed Apr 4 13:38:25 2018] [c000000ff596fdf0] [c000000000acd9a4] cpuidle_enter_state+0xa4/0x450
[Wed Apr 4 13:38:25 2018] [c000000ff596fe50] [c00000000017195c] call_cpuidle+0x4c/0x90
[Wed Apr 4 13:38:25 2018] [c000000ff596fe70] [c000000000171d70] do_idle+0x2b0/0x330
[Wed Apr 4 13:38:25 2018] [c000000ff596fec0] [c000000000172028] cpu_startup_entry+0x38/0x50
[Wed Apr 4 13:38:25 2018] [c000000ff596fef0] [c000000000049c30] start_secondary+0x4f0/0x510
[Wed Apr 4 13:38:25 2018] [c000000ff596ff90] [c00000000000aa6c] start_secondary_prolog+0x10/0x14
[Wed Apr 4 13:38:25 2018] Instruction dump:
[Wed Apr 4 13:38:25 2018] ebe1fff8 7c0803a6 4e800020 3c4c0164 38427580 7c0802a6 60000000 7c0802a6
[Wed Apr 4 13:38:25 2018] f8010010 f821ffe1 4bfff97d 4bf732d9 <60000000> 38210020 e8010010 7c0803a6
[Wed Apr 4 13:38:25 2018] Watchdog CPU:43 Hard LOCKUP
[Wed Apr 4 13:38:25 2018] Modules linked in: vhost_net vhost macvtap macvlan tap xfs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 nfs fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) cxl pnv_php mlx4_en(OE) mlx4_ib(OE) ib_core(OE) mlx4_core(OE) devlink mlx_compat(OE) kvm_hv kvm binfmt_misc dm_service_time dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua input_leds joydev mac_hid idt_89hpesx ipmi_powernv
[Wed Apr 4 13:38:25 2018] vmx_crypto ipmi_devintf at24 ofpart uio_pdrv_genirq cmdlinepart uio powernv_flash ipmi_msghandler mtd crct10dif_vpmsum opal_prd ibmpowernv nfsd sch_fq_codel auth_rpcgss nfs_acl lockd grace sunrpc knem(OE) ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq ses enclosure scsi_transport_sas hid_generic usbhid hid lpfc ast i2c_algo_bit ttm drm_kms_helper nvmet_fc syscopyarea sysfillrect nvmet sysimgblt fb_sys_fops nvme_fc nvme_fabrics crc32c_vpmsum drm i40e scsi_transport_fc aacraid [last unloaded: mlxfw]
[Wed Apr 4 13:38:25 2018] CPU: 43 PID: 0 Comm: swapper/43 Tainted: G OE 4.15.0-12-generic #13
[Wed Apr 4 13:38:25 2018] NIP: c0000000000a3ca4 LR: c0000000000a3ca4 CTR: c000000000008000
[Wed Apr 4 13:38:25 2018] REGS: c000000ff597fc40 TRAP: 0100 Tainted: G OE (4.15.0-12-generic)
[Wed Apr 4 13:38:25 2018] MSR: 9000000000001033 <SF,HV,ME,IR,DR,RI,LE> CR: 24004482 XER: 00000000
[Wed Apr 4 13:38:25 2018] CFAR: c000000ff597fda0 SOFTE: 43
                           GPR00: c0000000000a3ca4 c000000ff597fda0 c0000000016eb200 c000000ff597fc40
                           GPR04: b000000000001033 c0000000000a3690 0000000024004484 ffffffffffffffbf
                           GPR08: 000000000000007f c000000000d10ed8 00000000000000ff ffffffffffffffdf
                           GPR12: 9000000000121033 c000000007a3d900 c000000ff597ff90 0000000000000000
                           GPR16: 0000000000000000 c000000000047840 c000000000047810 c0000000011b5380
                           GPR20: 0000000000000800 c000000001722484 000000000000002b 0000000000000000
                           GPR24: 00000000000000ac 0000000000000007 0000000000000000 0000000000000007
                           GPR28: c00000000161d270 c000000ffb6a6fd8 c00000000161d528 0000000000000007
[Wed Apr 4 13:38:25 2018] NIP [c0000000000a3ca4] power9_idle_type+0x24/0x40
[Wed Apr 4 13:38:25 2018] LR [c0000000000a3ca4] power9_idle_type+0x24/0x40
[Wed Apr 4 13:38:25 2018] Call Trace:
[Wed Apr 4 13:38:25 2018] [c000000ff597fda0] [c0000000000a3ca4] power9_idle_type+0x24/0x40 (unreliable)
[Wed Apr 4 13:38:25 2018] [c000000ff597fdc0] [c000000000ad1240] stop_loop+0x40/0x5c
[Wed Apr 4 13:38:25 2018] [c000000ff597fdf0] [c000000000acd9a4] cpuidle_enter_state+0xa4/0x450
[Wed Apr 4 13:38:25 2018] [c000000ff597fe50] [c00000000017195c] call_cpuidle+0x4c/0x90
[Wed Apr 4 13:38:25 2018] [c000000ff597fe70] [c000000000171d70] do_idle+0x2b0/0x330
[Wed Apr 4 13:38:25 2018] [c000000ff597fec0] [c000000000172028] cpu_startup_entry+0x38/0x50
[Wed Apr 4 13:38:25 2018] [c000000ff597fef0] [c000000000049c30] start_secondary+0x4f0/0x510
[Wed Apr 4 13:38:25 2018] [c000000ff597ff90] [c00000000000aa6c] start_secondary_prolog+0x10/0x14
[Wed Apr 4 13:38:25 2018] Instruction dump:
[Wed Apr 4 13:38:25 2018] ebe1fff8 7c0803a6 4e800020 3c4c0164 38427580 7c0802a6 60000000 7c0802a6
[Wed Apr 4 13:38:25 2018] f8010010 f821ffe1 4bfff97d 4bf732d9 <60000000> 38210020 e8010010 7c0803a6
[Wed Apr 4 13:38:27 2018] Watchdog CPU:42 became unstuck
[Wed Apr 4 13:38:27 2018] Watchdog CPU:41 became unstuck
[Wed Apr 4 13:38:27 2018] Watchdog CPU:43 became unstuck

== Comment: #6 - Application Cdeadmin <email address hidden> - 2018-04-04 16:50:56 ==
------- Comment From youhour 2018-04-04 16:44:55 EDT -------
pegas 1.1 seems to fix my problem above by @haochanh. Upgrade your OS and see if that will help.

== Comment: #7 - Michael Neuling <email address hidden> - 2018-04-05 16:30:31 ==
So we've seen something similar on on other bugs (like https://github.com/open-power/boston-openpower/issues/1084#issuecomment-377122303)

It's looks like we may have taken an RCU stall which causes an NMI interrupt to be sent to the stalled CPU. This then interrupts a CPU which is in OPAL, which the kernel doesn't do a good job of recovering from. There are two patches that can help:

The first one removes the NMI on RCU stalls here https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?id=47712a921bb781caf69fca9eae43be19968816cb

The second improves the kernel handling of taking an NMI/sreset inside OPAL http://patchwork.ozlabs.org/patch/886688/ (not upstream).

== Comment: #15 - Application Cdeadmin <email address hidden> - 2018-04-12 08:41:01 ==
------- Comment From youhour 2018-04-12 08:32:21 EDT -------
@mikey Do we have commit for the fix yet?

== Comment: #16 - Gustavo Luiz Ferreira Walbon <email address hidden> - 2018-04-12 13:46:51 ==
All,

There are three of four from the original patchset that were approved on the upstream. Missing the patch '[RFC,4/4] powerpc/xmon: Detect if OPAL was interrupted and mark unrecoverable' (https://patchwork.ozlabs.org/patch/886691/)

[ATTENTION] The ubuntu kernel freeze is coming this week.

[1/4] https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/arch?h=next&id=15b4dd7981496f51c5f9262a5e0761e48de6655f

powerpc/64s: return more carefully from sreset NMI
System Reset, being an NMI, must return more carefully than other
interrupts. It has traditionally returned via the nromal return
from exception path, but that has a number of problems.

- r13 does not get restored if returning to kernel. This is for
  interrupts which may cause a context switch, which sreset will
  never do. Interrupting OPAL (which uses a different r13) is one
  place where this causes breakage.

- It may cause several other problems returning to kernel with
  preempt or TIF_EMULATE_STACK_STORE if it hits at the wrong time.

It's safer just to have a simple restore and return, like machine
check which is the other NMI.

[2/4] https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/arch?h=next&id=d40b6768e45bd9213139b2d91d30c7692b6007b1

powerpc/64s: sreset panic if there is no debugger or crash dump handlers
system_reset_exception does most of its own crash handling now,
invoking the debugger or crash dumps if they are registered. If not,
then it goes through to die() to print stack traces, and then is
supposed to panic (according to comments).

However after die() prints oopses, it does its own handling which
doesn't allow system_reset_exception to panic (e.g., it may just
kill the current process). This patch causes sreset exceptions to
return from die after it prints messages but before acting.

This also stops die from invoking the debugger on 0x100 crashes.
system_reset_exception similarly calls the debugger. It had been
thought this was harmless (because if the debugger was disabled,
neither call would fire, and if it was enabled the first call
would return). However in some cases like xmon 'X' command, the
debugger returns 0, which currently causes it to be entered
again (first in system_reset_exception, then in die), which is
confusing.

[3/4] https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/arch?h=next&id=741de617661794246f84a21a02fc5e327bffc9ad

powerpc/powernv: Handle unknown OPAL errors in opal_nvram_write()
opal_nvram_write currently just assumes success if it encounters an
error other than OPAL_BUSY or OPAL_BUSY_EVENT. Have it return -EIO
on other errors instead.

Fixes: 628daa8d5abf ("powerpc/powernv: Add RTC and NVRAM support plus RTAS fallbacks")

== Comment: #17 - Application Cdeadmin <email address hidden> - 2018-04-12 17:30:58 ==
------- Comment From youhour 2018-04-12 17:30:25 EDT -------
Stewart mentioned that these patches need to be picked by the distros.

@bwmashak Ben do you know who from the distros needs to be informed?

== Comment: #18 - Michael Y. Lim <email address hidden> - 2018-04-13 10:12:40 ==
Gustavo, please let us know which kernel version has this patch. Thank you!

== Comment: #19 - Gustavo Luiz Ferreira Walbon <email address hidden> - 2018-04-13 14:22:06 ==
(In reply to comment #18)
> Gustavo, please let us know which kernel version has this patch. Thank you!

Hello Michael,

So, There is no official distro with those patches, they are on upstream yet.
I have generated a build with just asked patch set here which it's based on the ubuntu kernel v4.15.0-12.13.
http://pokgsa.ibm.com/gsa/pokgsa/home/g/w/gwalbon/web/public/Bug165882/v2/

== Comment: #21 - Benjamin W. Mashak <email address hidden> - 2018-04-24 15:59:09 ==
Gustavo Luiz Ferreira Walbon, what's the outlook to upstream and close this BZ? Its currently on the must-fix list for upcoming GA in May.

== Comment: #22 - Gustavo Luiz Ferreira Walbon <email address hidden> - 2018-04-25 08:28:01 ==
(In reply to comment #21)
> Gustavo Luiz Ferreira Walbon, what's the outlook to upstream and close this
> BZ? Its currently on the must-fix list for upcoming GA in May.

Benjamin,

This missing patch was a RFC by Nicholas Piggin, as a RFC just the 3 of 4 patches was judged as relevant and they were added to powerpc tree.

I hope this 3 patches were enough.

== Comment: #24 - Gustavo Luiz Ferreira Walbon <email address hidden> - 2018-04-25 08:34:35 ==
Adding a patch series to fix a CPU lockup in UbuntuKVM 18.04.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-04-30 13:07 EDT-------
Submitted
https://lists.ubuntu.com/archives/kernel-team/2018-April/092069.html
https://lists.ubuntu.com/archives/kernel-team/2018-April/092070.html
https://lists.ubuntu.com/archives/kernel-team/2018-April/092071.html

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-07 12:43 EDT-------
Hi Canonical,

What is wrong in my request to add those patches?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-07 12:51 EDT-------
I'm a bit confuse regarding this issue 1105 and issue 1257.

@haochanh Chan can you post the plc log here also.

[Frank Heimes (fheimes)]

@Gustavo - in case the patches are upstream accepted, nothing is wrong with your request.
But keep in mind that 18.04 was recently released, hence all patches that should be added to such an already released Ubuntu version now need to follow the SRU process.
https://wiki.ubuntu.com/StableReleaseUpdates
This is a more structured process where various considerations and approvals are needed - hence it is more time consuming compared to adding patches to an Ubuntu version that is still in development.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-09 17:11 EDT-------
@haochanh please verify and close.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-14 16:51 EDT-------
Ubuntu issue, so removing LC GA1 Mustfix.

[Breno Leitão (breno-leitao)]

Hey Frank,

I understand all of these patches are upstreamed already on Linus' tree and most of the patches seem to be clean cherry picks (except 15b4dd7981496f51c5f9262a5e0761e48de6655f, which was backported).

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-29 13:09 EDT-------
Chanh, are you able to verify the proposed kernel?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-29 13:42 EDT-------
Canonical: the "verification-needed-bionic" tag was removed - is this intentional?

[Frank Heimes (fheimes)]

The activity log: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1767927/+activity
shows that it was removed by LP user 'bugproxy' - probably an accident of the Bugzilla bridge?!

Looks like the Bugzilla bridge removed several tags (see log entry '2018-05-29 17:19:23'.
@IBM please verify if the bridge behavior is correct and than expected.

For now I added the tags again ...

[Stefan Bader (smb)]

Any progress on the verification for this?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-07 16:54 EDT-------
Hi,

I have looking for a machine to reproduce then update all of thing to validate it. Tomorrow I hope a boston LC available for testing back.

sorry for delay.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-08 12:57 EDT-------
I got a Boston

I used a kernel with proposed patches and I did not see a hang of error in dmesg after ipmi test in KVM guest.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-23.25

---------------
linux (4.15.0-23.25) bionic; urgency=medium

  * linux: 4.15.0-23.25 -proposed tracker (LP: #1772927)

  * arm64 SDEI support needs trampoline code for KPTI (LP: #1768630)
    - arm64: mmu: add the entry trampolines start/end section markers into
      sections.h
    - arm64: sdei: Add trampoline code for remapping the kernel

  * Some PCIe errors not surfaced through rasdaemon (LP: #1769730)
    - ACPI: APEI: handle PCIe AER errors in separate function
    - ACPI: APEI: call into AER handling regardless of severity

  * qla2xxx: Fix page fault at kmem_cache_alloc_node() (LP: #1770003)
    - scsi: qla2xxx: Fix session cleanup for N2N
    - scsi: qla2xxx: Remove unused argument from qlt_schedule_sess_for_deletion()
    - scsi: qla2xxx: Serialize session deletion by using work_lock
    - scsi: qla2xxx: Serialize session free in qlt_free_session_done
    - scsi: qla2xxx: Don't call dma_free_coherent with IRQ disabled.
    - scsi: qla2xxx: Fix warning in qla2x00_async_iocb_timeout()
    - scsi: qla2xxx: Prevent relogin trigger from sending too many commands
    - scsi: qla2xxx: Fix double free bug after firmware timeout
    - scsi: qla2xxx: Fixup locking for session deletion

  * Several hisi_sas bug fixes (LP: #1768974)
    - scsi: hisi_sas: dt-bindings: add an property of signal attenuation
    - scsi: hisi_sas: support the property of signal attenuation for v2 hw
    - scsi: hisi_sas: fix the issue of link rate inconsistency
    - scsi: hisi_sas: fix the issue of setting linkrate register
    - scsi: hisi_sas: increase timer expire of internal abort task
    - scsi: hisi_sas: remove unused variable hisi_sas_devices.running_req
    - scsi: hisi_sas: fix return value of hisi_sas_task_prep()
    - scsi: hisi_sas: Code cleanup and minor bug fixes

  * [bionic] machine stuck and bonding not working well when nvmet_rdma module
    is loaded (LP: #1764982)
    - nvmet-rdma: Don't flush system_wq by default during remove_one
    - nvme-rdma: Don't flush delete_wq by default during remove_one

  * Warnings/hang during error handling of SATA disks on SAS controller
    (LP: #1768971)
    - scsi: libsas: defer ata device eh commands to libata

  * Hotplugging a SATA disk into a SAS controller may cause crash (LP: #1768948)
    - ata: do not schedule hot plug if it is a sas host

  * ISST-LTE:pKVM:Ubuntu1804: rcu_sched self-detected stall on CPU follow by CPU
    ATTEMPT TO RE-ENTER FIRMWARE! (LP: #1767927)
    - powerpc/powernv: Handle unknown OPAL errors in opal_nvram_write()
    - powerpc/64s: return more carefully from sreset NMI
    - powerpc/64s: sreset panic if there is no debugger or crash dump handlers

  * fsnotify: Fix fsnotify_mark_connector race (LP: #1765564)
    - fsnotify: Fix fsnotify_mark_connector race

  * Hang on network interface removal in Xen virtual machine (LP: #1771620)
    - xen-netfront: Fix hang on device removal

  * HiSilicon HNS NIC names are truncated in /proc/interrupts (LP: #1765977)
    - net: hns: Avoid action name truncation

  * Ubuntu 18.04 kernel crashed while in degraded mode (LP: #1770849)
    - SAUCE: powerpc/perf: Fix memory allocation for...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-20 17:50 EDT-------
Ubuntu issue. Removing IBM FW tag.