Hard LOCKUP observed on stressing Ubuntu 18 04

[bugproxy]

== SRU Justification ==
IBM has seen hard lockups when running stress tests against Bionic.
These test sometimes lead to rcu_stalls. IBM found that this bug is
resolved with commits 6bed3237624e and ac61c1156623.

== Fixes ==
6bed3237624e ("powerpc: use NMI IPI for smp_send_stop")
ac61c1156623 ("powerpc: Fix smp_send_stop NMI IPI handling")

== Regression Potential ==
Low. Limited to powerpc.

== Test Case ==
A test kernel was built with these patches and tested by the original bug reporter.
The bug reporter states the test kernel resolved the bug.

--Problem Description--
Hard LOCKUP on stressing Ubuntu 18 04

---Issue observed---
Hard LOCKUP on stressing Ubuntu 18 04 using Ubuntu 18 04, sometimes leads to rcu_stalls.

Apr 17 00:00:23 lep8d kernel: [ 4309.786755] Watchdog CPU:3 Hard LOCKUP
Apr 17 00:00:23 lep8d kernel: [ 4309.786759] Modules linked in: algif_rng salsa20_generic userio camellia_generic cast6_generic cast_common snd_seq snd_seq_device snd_timer snd soundcore vhost_net serpent_generic tap twofish_generic twofish_common vhost_vsock vmw_vsock_virtio_transport_common vhost vsock lrw unix_diag algif_skcipher cuse sctp tgr192 wp512 rmd320 rmd256 rmd160 hci_vhci rmd128 bluetooth ecdh_generic dccp_ipv4 md4 uhid hid algif_hash dccp af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter kvm_hv kvm binfmt_misc uio_pdrv_genirq uio vmx_crypto ibmpowernv powernv_op_panel ipmi_powernv
Apr 17 00:00:23 lep8d kernel: [ 4309.786899] ipmi_devintf ipmi_msghandler powernv_rng leds_powernv crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas btrfs xor zstd_compress raid6_pq uas usb_storage crc32c_vpmsum tg3 ipr
Apr 17 00:00:23 lep8d kernel: [ 4309.786944] CPU: 3 PID: 28361 Comm: stress-ng-hrtim Not tainted 4.15.0-15-generic #16-Ubuntu
Apr 17 00:00:23 lep8d kernel: [ 4309.786950] NIP: c000000000d0c8b8 LR: c000000000120dbc CTR: c000000000024480
Apr 17 00:00:23 lep8d kernel: [ 4309.786956] REGS: c000000007f7fd80 TRAP: 0900 Not tainted (4.15.0-15-generic)
Apr 17 00:00:23 lep8d kernel: [ 4309.786957] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28000442 XER: 20000000
Apr 17 00:00:23 lep8d kernel: [ 4309.786972] CFAR: c000000000120db8 SOFTE: 0
Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR00: c000000000120dbc c000002d51377ba0 c0000000016eb400 c000002d512b0f88
Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR04: 0000000000000000 0000000000000001 0000000001f40668 0000000000000001
Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR08: 0000000000000001 0000000000000000 0000000080000003 0000000000000000
Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR12: c000000000024480 c000000007a22100 0000000000000000 00000000000186a0
Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR16: 00007fffed3687e0 00000abda98e5db8 0000000000008005 0000000000040100
Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR20: 0000000000000000 00000000418004fc 00000000003c0000 0000000008430000
Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR24: c000002d512b0f88 0000000000000000 c000002d51377d30 c000002d52d47c00
Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR28: c000002d51377d50 c000002d51377d50 c000002d52f04500 c000002d512b0f88
Apr 17 00:00:23 lep8d kernel: [ 4309.787035] NIP [c000000000d0c8b8] _raw_spin_lock+0x38/0xe0
Apr 17 00:00:23 lep8d kernel: [ 4309.787045] LR [c000000000120dbc] dequeue_signal+0xcc/0x260
Apr 17 00:00:23 lep8d kernel: [ 4309.787046] Call Trace:
Apr 17 00:00:23 lep8d kernel: [ 4309.787052] [c000002d51377bd0] [c000000000120dac] dequeue_signal+0xbc/0x260
Apr 17 00:00:23 lep8d kernel: [ 4309.787059] [c000002d51377c20] [c00000000012459c] get_signal+0x13c/0x7a0
Apr 17 00:00:23 lep8d kernel: [ 4309.787066] [c000002d51377d10] [c00000000001dacc] do_signal+0x7c/0x2c0
Apr 17 00:00:23 lep8d kernel: [ 4309.787072] [c000002d51377e00] [c00000000001deb0] do_notify_resume+0xd0/0x100
Apr 17 00:00:23 lep8d kernel: [ 4309.787083] [c000002d51377e30] [c00000000000b7c4] ret_from_except_lite+0x70/0x74
Apr 17 00:00:23 lep8d kernel: [ 4309.787085] Instruction dump:
Apr 17 00:00:23 lep8d kernel: [ 4309.787090] 7c0802a6 60000000 fbe1fff8 f821ffd1 7c7f1b78 39400000 994d028c 814d0008
Apr 17 00:00:23 lep8d kernel: [ 4309.787102] 7d201829 2c090000 40c20010 7d40192d <40c2fff0> 7c2004ac 2fa90000 409e001c
Apr 17 00:00:23 lep8d kernel: [ 4313.015781] kauditd_printk_skb: 13 callbacks suppressed

---uname output---
# uname -a
Linux lep8d 4.15.0-15-generic #16-Ubuntu SMP Wed Apr 4 13:57:51 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = Power 8 BML/Tuleta

----Additional Info-----
Hard LOCKUP is also seen on garri BML.
syslog is attached.

Reproducible : 90%

---Steps to Reproduce---
1. wget https://github.com/ColinIanKing/stress-ng/archive/master.zip
2. unzip master.zip; cd stress-ng-master;
3. make; make install;
4. Run the following command multiple times
stress-ng --all <nr_cpus> --vm-bytes 80% --aggressive --maximize --oomable --timeout 300 --verify --syslog --metrics --times

Issue is observed on Power 9 BML machines as well.

[Tue Apr 17 04:13:42 2018] Watchdog CPU:37 Hard LOCKUP
[Tue Apr 17 04:13:42 2018] Modules linked in: vsock lrw algif_skcipher tgr192 wp512 rmd320 rmd256 hci_vhci unix_diag bluetooth rmd160 sctp rmd128 ecdh_generic md4 dccp_ipv4 algif_hash cuse dccp af_alg vhost_net vhost tap xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc kvm_hv kvm dm_crypt dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua idt_89hpesx joydev input_leds mac_hid ofpart cmdlinepart vmx_crypto ipmi_powernv ipmi_devintf at24 uio_pdrv_genirq ibmpowernv opal_prd crct10dif_vpmsum powernv_flash ipmi_msghandler mtd uio sch_fq_codel ib_iser rdma_cm iw_cm ib_cm
[Tue Apr 17 04:13:42 2018] ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi jc42 ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq ses enclosure scsi_transport_sas hid_generic ast i2c_algo_bit ttm drm_kms_helper usbhid hid syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e aacraid
[Tue Apr 17 04:13:42 2018] CPU: 37 PID: 11524 Comm: stress-ng-hrtim Not tainted 4.15.0-15-generic #16-Ubuntu
[Tue Apr 17 04:13:42 2018] NIP: c00000000012058c LR: c000000000120554 CTR: c00000000002bd30
[Tue Apr 17 04:13:42 2018] REGS: c000000007debd80 TRAP: 0900 Not tainted (4.15.0-15-generic)
[Tue Apr 17 04:13:42 2018] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 22000442 XER: 00000000
[Tue Apr 17 04:13:42 2018] CFAR: c00000000011f8c4 SOFTE: 0
                           GPR00: c000000000120554 c0000004cfc7bd10 c0000000016eb400 c0000004cfb54200
                           GPR04: c0000004cfc7be00 0000000042000442 0000000000007338 0000000000040000
                           GPR08: c0000004cfc78080 c0000004cfc78000 0000000000000002 c000000000d10f78
                           GPR12: c00000000002bd30 c000000007a39700 0000000000000000 00000000000186a0
                           GPR16: 00007fffd2cab1d0 00000edac67c5db8 00007fffd2cab1c8 00000edac67c5dc0
                           GPR20: 00007fffd2cab1cc 00007fffd2cab48b ffffffffffffffff 00007fffd2cab48a
                           GPR24: 0000000000010000 000072164cd80000 00007fffd2cab0c4 00007fffd2cab1d8
                           GPR28: 0000000000000000 c0000004cfc7be00 c0000004cfc7be00 c0000004cfb54200
[Tue Apr 17 04:13:42 2018] NIP [c00000000012058c] recalc_sigpending+0x5c/0x90
[Tue Apr 17 04:13:42 2018] LR [c000000000120554] recalc_sigpending+0x24/0x90
[Tue Apr 17 04:13:42 2018] Call Trace:
[Tue Apr 17 04:13:42 2018] [c0000004cfc7bd10] [c00000000001db60] do_signal+0x110/0x2c0 (unreliable)
[Tue Apr 17 04:13:42 2018] [c0000004cfc7bd30] [c000000000121658] __set_task_blocked+0x48/0x90
[Tue Apr 17 04:13:42 2018] [c0000004cfc7bd70] [c000000000124ed8] __set_current_blocked+0x58/0xb0
[Tue Apr 17 04:13:42 2018] [c0000004cfc7bda0] [c00000000002be18] sys_rt_sigreturn+0xe8/0x270
[Tue Apr 17 04:13:42 2018] [c0000004cfc7be30] [c00000000000b184] system_call+0x58/0x6c
[Tue Apr 17 04:13:42 2018] Instruction dump:
[Tue Apr 17 04:13:42 2018] e86d0260 3d220020 3929def8 81290000 2f890000 409e0030 78290464 39400002
[Tue Apr 17 04:13:42 2018] 39090080 7ce040a8 7ce75078 7ce041ad <40c2fff4> 38210020 e8010010 7c0803a6

Continuous lockups are observed.

- Harish

== ==
Hardware: P9 Boston/ P8 Tuleta

DD revision: P9 DD2.2

Operating Env.: BML

PNOR: version-SUPERMICRO-P9DSU-V1.10-20180413-imp

Host OS: Ubuntu 18.04

====
(In reply to comment #5)
> hi Harish,
>
> Does this bug happen if you set powersave=off? I am wondering if this
> problem might be related to the stop state issue.

Following Issue is seen with powersave=off. This is on Power 8 BML.

[ 517.480153] Modules linked in: salsa20_generic(+) camellia_generic cast6_generic cast_common serpent_generic twofish_generic twofish_common lrw algif_skcipher cuse hci_vhci bluetooth snd_seq snd_seq_device tgr192 ecdh_generic snd_timer snd wp512 soundcore rmd320 rmd256 rmd160 uhid hid vhost_net tap sctp unix_diag userio rmd128 vhost_vsock vmw_vsock_virtio_transport_common md4 dccp_ipv4 vhost vsock algif_hash dccp af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter kvm_hv kvm binfmt_misc ipmi_powernv leds_powernv ipmi_devintf vmx_crypto uio_pdrv_genirq ipmi_msghandler
[ 517.480305] powernv_rng crct10dif_vpmsum uio ibmpowernv powernv_op_panel sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas btrfs xor zstd_compress raid6_pq tg3 uas crc32c_vpmsum ipr usb_storage
[ 517.480343] CPU: 73 PID: 4388 Comm: stress-ng-dev Not tainted 4.15.0-15-generic #16-Ubuntu
[ 517.480347] NIP: c00000000000a724 LR: c000000000016e74 CTR: 0000000030061154
[ 517.480352] REGS: c000001fef1db890 TRAP: 0901 Not tainted (4.15.0-15-generic)
[ 517.480353] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48002224 XER: 20000000
[ 517.480368] CFAR: c000000000d0c8fc SOFTE: 1
[ 517.480368] GPR00: c00000000018c5bc c000001fef1dbb10 c0000000016eb400 0000000000000500
[ 517.480368] GPR04: 0000000000000000 c0000000000a2608 9000000000001033 0000000000000004
[ 517.480368] GPR08: c000000007a52300 0000000000000000 0000000080000049 9000000000001003
[ 517.480368] GPR12: 0000000000000040 c000000007a52300
[ 517.480406] NIP [c00000000000a724] replay_interrupt_return+0x0/0x4
[ 517.480411] LR [c000000000016e74] arch_local_irq_restore+0x74/0x90
[ 517.480412] Call Trace:
[ 517.480420] [c000001fef1dbb10] [c0000000018bd940] log_first_seq+0x0/0x8 (unreliable)
[ 517.480427] [c000001fef1dbb30] [c00000000018c5bc] console_unlock+0x2fc/0x6c0
[ 517.480432] [c000001fef1dbc20] [c00000000018ccec] vprintk_emit+0x36c/0x420
[ 517.480437] [c000001fef1dbc90] [c00000000018ec54] vprintk_func+0x64/0xf0
[ 517.480442] [c000001fef1dbcb0] [c00000000018e354] printk+0x40/0x54
[ 517.480455] [c000001fef1dbcd0] [d00000001efc2f20] vsock_dev_do_ioctl.isra.4+0xb8/0xe0 [vsock]
[ 517.480462] [c000001fef1dbd40] [c0000000003efc34] do_vfs_ioctl+0xd4/0xa00
[ 517.480467] [c000001fef1dbde0] [c0000000003f0624] SyS_ioctl+0xc4/0x130
[ 517.480473] [c000001fef1dbe30] [c00000000000b184] system_call+0x58/0x6c
[ 517.480475] Instruction dump:
[ 517.480480] 7d8000a6 e9628008 7d200026 618c8000 2c030900 4182e7f8 2c030500 4182e310
[ 517.480491] 2c030a00 4182ffa4 2c030e60 4182f090 <4e800020> 7c781b78 48000359 48000371

which leads to the Hard LOCKUP and rcu_stalls.

[ 629.383369] Watchdog CPU:73 Hard LOCKUP
[ 629.383372] Modules linked in: salsa20_generic(+) camellia_generic cast6_generic cast_common serpent_generic twofish_generic twofish_common lrw algif_skcipher cuse hci_vhci bluetooth snd_seq snd_seq_device tgr192 ecdh_generic snd_timer snd wp512 soundcore rmd320 rmd256 rmd160 uhid hid vhost_net tap sctp unix_diag userio rmd128 vhost_vsock vmw_vsock_virtio_transport_common md4 dccp_ipv4 vhost vsock algif_hash dccp af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter kvm_hv kvm binfmt_misc ipmi_powernv leds_powernv ipmi_devintf vmx_crypto uio_pdrv_genirq ipmi_msghandler
[ 629.383451] powernv_rng crct10dif_vpmsum uio ibmpowernv powernv_op_panel sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas btrfs xor zstd_compress raid6_pq tg3 uas crc32c_vpmsum ipr usb_storage
[ 629.383475] CPU: 73 PID: 4388 Comm: stress-ng-dev Tainted: G L 4.15.0-15-generic #16-Ubuntu
[ 629.383478] NIP: c0000000000a259c LR: c00000000009df5c CTR: 0000000030036830
[ 629.383481] REGS: c000000007c37d80 TRAP: 0900 Tainted: G L (4.15.0-15-generic)
[ 629.383482] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48002222 XER: 20000000
[ 629.383494] CFAR: c00000000009df48 SOFTE: 0
[ 629.383497] GPR00: 0000000030005128 c000001fef1dba30 c0000000016eb400 0000000000000000
[ 629.383504] GPR04: 0000000048002222 c0000000000a259c 9000000000009033 00000000000000f1
[ 629.383510] GPR08: 0000000000000000 00000000300b0218 c00000000009df70 9000000000001003
[ 629.383517] GPR12: c00000000009df48 c000000007a52300 0000714a9655a560 0000000000000000
[ 629.383524] GPR16: 0000000000000027 0000000000000027 c000000001572a00 0000000000000000
[ 629.383530] GPR20: 20c49ba5e353f7cf 0000000000000000 0000000000000017 000000000000000d
[ 629.383537] GPR24: fffffffffffffff5 0000000000000000 0000000000000010 c0000000018a8d18
[ 629.383544] GPR28: 0000000000000000 0000000000000010 c000001fef1dbad0 0000000000000010
[ 629.383551] NIP [c0000000000a259c] opal_put_chars+0x19c/0x280
[ 629.383553] LR [c00000000009df5c] opal_return+0x14/0x48
[ 629.383554] Call Trace:
[ 629.383556] [c000001fef1dba30] [c0000000000a259c] opal_put_chars+0x19c/0x280 (unreliable)
[ 629.383561] [c000001fef1dbab0] [c0000000008089b0] hvc_console_print+0xd0/0x210
[ 629.383564] [c000001fef1dbb30] [c00000000018c59c] console_unlock+0x2dc/0x6c0
[ 629.383568] [c000001fef1dbc20] [c00000000018ccec] vprintk_emit+0x36c/0x420
[ 629.383571] [c000001fef1dbc90] [c00000000018ec54] vprintk_func+0x64/0xf0
[ 629.383574] [c000001fef1dbcb0] [c00000000018e354] printk+0x40/0x54
[ 629.383577] [c000001fef1dbcd0] [d00000001efc2f20] vsock_dev_do_ioctl.isra.4+0xb8/0xe0 [vsock]
[ 629.383581] [c000001fef1dbd40] [c0000000003efc34] do_vfs_ioctl+0xd4/0xa00
[ 629.383584] [c000001fef1dbde0] [c0000000003f0624] SyS_ioctl+0xc4/0x130
[ 629.383587] [c000001fef1dbe30] [c00000000000b184] system_call+0x58/0x6c
[ 629.383589] Instruction dump:
[ 629.383591] 7f03c378 38210080 eb01ffc0 eb61ffd8 4e800020 7f63db78 7f24cb78 48c6a5e1
[ 629.383600] 60000000 38600000 3b00fff5 4bffc01d <60000000> e8010090 eb210048 eb810060
[ 638.478556] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 638.478581] 73-....: (149 ticks this GP) idle=0aa/140000000000000/0 softirq=14950/14950 fqs=15015
[ 638.478586] (detected by 83, t=37345 jiffies, g=2733, c=2732, q=9230463)
[ 638.478645] Sending NMI from CPU 83 to CPUs 73:

So, actual issue might be with powersave. These issue are due to huge data dumped to console.

- Harish

== Harish Sriram <email address hidden> ==
With powersave=off on a P9 DD 2.2 Boston LC, the following is observed.

[ 119.200587] Watchdog CPU:60 Hard LOCKUP
[ 119.204932] Watchdog CPU:48 Hard LOCKUP
[ 119.207911] Watchdog CPU:52 Hard LOCKUP
[ 119.208454] Watchdog CPU:46 Hard LOCKUP
....
....

syslog will be attached.

- Harish

== MAHESH J. SALGAONKAR <email address hidden> ==

> > hi Harish,
> >
> > Does this bug happen if you set powersave=off? I am wondering if this
> > problem might be related to the stop state issue.
>
> Following Issue is seen with powersave=off. This is on Power 8 BML.
>
> [ 517.480153] Modules linked in: salsa20_generic(+) camellia_generic
> cast6_generic cast_common serpent_generic twofish_generic twofish_common lrw
> algif_skcipher cuse hci_vhci bluetooth snd_seq snd_seq_device tgr192
> ecdh_generic snd_timer snd wp512 soundcore rmd320 rmd256 rmd160 uhid hid
> vhost_net tap sctp unix_diag userio rmd128 vhost_vsock
> vmw_vsock_virtio_transport_common md4 dccp_ipv4 vhost vsock algif_hash dccp
> af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
> iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack
> nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc
> devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter
> kvm_hv kvm binfmt_misc ipmi_powernv leds_powernv ipmi_devintf vmx_crypto
> uio_pdrv_genirq ipmi_msghandler
> [ 517.480305] powernv_rng crct10dif_vpmsum uio ibmpowernv powernv_op_panel
> sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas
> btrfs xor zstd_compress raid6_pq tg3 uas crc32c_vpmsum ipr usb_storage
> [ 517.480343] CPU: 73 PID: 4388 Comm: stress-ng-dev Not tainted
> 4.15.0-15-generic #16-Ubuntu
> [ 517.480347] NIP: c00000000000a724 LR: c000000000016e74 CTR:
> 0000000030061154
> [ 517.480352] REGS: c000001fef1db890 TRAP: 0901 Not tainted
> (4.15.0-15-generic)
> [ 517.480353] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR:
> 48002224 XER: 20000000
> [ 517.480368] CFAR: c000000000d0c8fc SOFTE: 1
> [ 517.480368] GPR00: c00000000018c5bc c000001fef1dbb10 c0000000016eb400
> 0000000000000500
> [ 517.480368] GPR04: 0000000000000000 c0000000000a2608 9000000000001033
> 0000000000000004
> [ 517.480368] GPR08: c000000007a52300 0000000000000000 0000000080000049
> 9000000000001003
> [ 517.480368] GPR12: 0000000000000040 c000000007a52300
> [ 517.480406] NIP [c00000000000a724] replay_interrupt_return+0x0/0x4
> [ 517.480411] LR [c000000000016e74] arch_local_irq_restore+0x74/0x90
> [ 517.480412] Call Trace:
> [ 517.480420] [c000001fef1dbb10] [c0000000018bd940] log_first_seq+0x0/0x8
> (unreliable)
> [ 517.480427] [c000001fef1dbb30] [c00000000018c5bc]
> console_unlock+0x2fc/0x6c0
> [ 517.480432] [c000001fef1dbc20] [c00000000018ccec] vprintk_emit+0x36c/0x420
> [ 517.480437] [c000001fef1dbc90] [c00000000018ec54] vprintk_func+0x64/0xf0
> [ 517.480442] [c000001fef1dbcb0] [c00000000018e354] printk+0x40/0x54
> [ 517.480455] [c000001fef1dbcd0] [d00000001efc2f20]
> vsock_dev_do_ioctl.isra.4+0xb8/0xe0 [vsock]
> [ 517.480462] [c000001fef1dbd40] [c0000000003efc34] do_vfs_ioctl+0xd4/0xa00
> [ 517.480467] [c000001fef1dbde0] [c0000000003f0624] SyS_ioctl+0xc4/0x130
> [ 517.480473] [c000001fef1dbe30] [c00000000000b184] system_call+0x58/0x6c
> [ 517.480475] Instruction dump:
> [ 517.480480] 7d8000a6 e9628008 7d200026 618c8000 2c030900 4182e7f8
> 2c030500 4182e310
> [ 517.480491] 2c030a00 4182ffa4 2c030e60 4182f090 <4e800020> 7c781b78
> 48000359 48000371
>
> which leads to the Hard LOCKUP and rcu_stalls.
>
> [ 629.383369] Watchdog CPU:73 Hard LOCKUP
> [ 629.383372] Modules linked in: salsa20_generic(+) camellia_generic
> cast6_generic cast_common serpent_generic twofish_generic twofish_common lrw
> algif_skcipher cuse hci_vhci bluetooth snd_seq snd_seq_device tgr192
> ecdh_generic snd_timer snd wp512 soundcore rmd320 rmd256 rmd160 uhid hid
> vhost_net tap sctp unix_diag userio rmd128 vhost_vsock
> vmw_vsock_virtio_transport_common md4 dccp_ipv4 vhost vsock algif_hash dccp
> af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
> iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack
> nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc
> devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter
> kvm_hv kvm binfmt_misc ipmi_powernv leds_powernv ipmi_devintf vmx_crypto
> uio_pdrv_genirq ipmi_msghandler
> [ 629.383451] powernv_rng crct10dif_vpmsum uio ibmpowernv powernv_op_panel
> sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas
> btrfs xor zstd_compress raid6_pq tg3 uas crc32c_vpmsum ipr usb_storage
> [ 629.383475] CPU: 73 PID: 4388 Comm: stress-ng-dev Tainted: G
> L 4.15.0-15-generic #16-Ubuntu
> [ 629.383478] NIP: c0000000000a259c LR: c00000000009df5c CTR:
> 0000000030036830
> [ 629.383481] REGS: c000000007c37d80 TRAP: 0900 Tainted: G L
> (4.15.0-15-generic)
> [ 629.383482] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR:
> 48002222 XER: 20000000
> [ 629.383494] CFAR: c00000000009df48 SOFTE: 0
> [ 629.383497] GPR00: 0000000030005128 c000001fef1dba30 c0000000016eb400
> 0000000000000000
> [ 629.383504] GPR04: 0000000048002222 c0000000000a259c 9000000000009033
> 00000000000000f1
> [ 629.383510] GPR08: 0000000000000000 00000000300b0218 c00000000009df70
> 9000000000001003
> [ 629.383517] GPR12: c00000000009df48 c000000007a52300 0000714a9655a560
> 0000000000000000
> [ 629.383524] GPR16: 0000000000000027 0000000000000027 c000000001572a00
> 0000000000000000
> [ 629.383530] GPR20: 20c49ba5e353f7cf 0000000000000000 0000000000000017
> 000000000000000d
> [ 629.383537] GPR24: fffffffffffffff5 0000000000000000 0000000000000010
> c0000000018a8d18
> [ 629.383544] GPR28: 0000000000000000 0000000000000010 c000001fef1dbad0
> 0000000000000010
> [ 629.383551] NIP [c0000000000a259c] opal_put_chars+0x19c/0x280
> [ 629.383553] LR [c00000000009df5c] opal_return+0x14/0x48
> [ 629.383554] Call Trace:
> [ 629.383556] [c000001fef1dba30] [c0000000000a259c]
> opal_put_chars+0x19c/0x280 (unreliable)
> [ 629.383561] [c000001fef1dbab0] [c0000000008089b0]
> hvc_console_print+0xd0/0x210
> [ 629.383564] [c000001fef1dbb30] [c00000000018c59c]
> console_unlock+0x2dc/0x6c0
> [ 629.383568] [c000001fef1dbc20] [c00000000018ccec] vprintk_emit+0x36c/0x420
> [ 629.383571] [c000001fef1dbc90] [c00000000018ec54] vprintk_func+0x64/0xf0
> [ 629.383574] [c000001fef1dbcb0] [c00000000018e354] printk+0x40/0x54
> [ 629.383577] [c000001fef1dbcd0] [d00000001efc2f20]
> vsock_dev_do_ioctl.isra.4+0xb8/0xe0 [vsock]
> [ 629.383581] [c000001fef1dbd40] [c0000000003efc34] do_vfs_ioctl+0xd4/0xa00
> [ 629.383584] [c000001fef1dbde0] [c0000000003f0624] SyS_ioctl+0xc4/0x130
> [ 629.383587] [c000001fef1dbe30] [c00000000000b184] system_call+0x58/0x6c
> [ 629.383589] Instruction dump:
> [ 629.383591] 7f03c378 38210080 eb01ffc0 eb61ffd8 4e800020 7f63db78
> 7f24cb78 48c6a5e1
> [ 629.383600] 60000000 38600000 3b00fff5 4bffc01d <60000000> e8010090
> eb210048 eb810060
> [ 638.478556] INFO: rcu_sched detected stalls on CPUs/tasks:
> [ 638.478581] 73-....: (149 ticks this GP) idle=0aa/140000000000000/0
> softirq=14950/14950 fqs=15015
> [ 638.478586] (detected by 83, t=37345 jiffies, g=2733, c=2732,
> q=9230463)
> [ 638.478645] Sending NMI from CPU 83 to CPUs 73:
>
> So, actual issue might be with powersave. These issue are due to huge data
> dumped to console.
>
> - Harish

== Breno Leitao <email address hidden> ==
> I think we probably need below upstream commit:
>
> commit 15b4dd7981496f51c5f9262a5e0761e48de6655f
> Author: Nicholas Piggin <email address hidden>
> Date: Tue Mar 27 01:01:03 2018 +1000
>
> powerpc/64s: return more carefully from sreset NMI

This patch apply cleanly on top of bionic kernel, but we definitely need to test it before sending to canonical for inclusion. Holding the mirror until we have an ack that this is the patch we need for this issue.

== Gustavo Luiz Ferreira Walbon <email address hidden> ==
Breno,

Two patches are missing,

6bed3237624e3faad1592543952907cd01a42c83 powerpc: use NMI IPI for smp_send_stop
ac61c1156623455c46701654abd8c99720bceea1 powerpc: Fix smp_send_stop NMI IPI handling

I have rebased the tree with latest proposed kernel and built the packages again :
http://pokgsa.ibm.com/gsa/pokgsa/home/g/w/gwalbon/web/public/Bug166791/

Patches : https://github.com/walbon/ubuntu-bionic/commits/bug166791

== Harish Sriram <email address hidden>==

> > Hard Lockups are observed with stressing the 18.04.1 kernel.
> >
> > # uname -a
> > Linux ltc-wspoon12 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 17:59:00 UTC
> > 2018 ppc64le ppc64le ppc64le GNU/Linux
> >
> > [ 5295.047807] Watchdog CPU:64 Hard LOCKUP
> > [ 5295.047817] Modules linked in: snd_seq snd_seq_device snd_timer snd
> > soundcore twofish_generic twofish_common lrw uhid hid algif_skcipher
> > vhost_net tgr192 vhost tap wp512 hci_vhci bluetooth rmd320 ecdh_generic
> > rmd256 cuse rmd160 rmd128 btrfs zstd_compress md4 algif_hash xor unix_diag
> > binfmt_misc raid6_pq sctp af_alg userio dccp_ipv4 dccp xt_CHECKSUM
> > iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4
> > nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c
> > ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables
> > ip6table_filter ip6_tables iptable_filter idt_89hpesx at24 ofpart
> > uio_pdrv_genirq ipmi_powernv ipmi_devintf ipmi_msghandler opal_prd uio
> > cmdlinepart powernv_flash mtd ibmpowernv vmx_crypto crct10dif_vpmsum kvm_hv
> > kvm sch_fq_codel
> > [ 5295.047928] ip_tables x_tables autofs4 mlx5_ib ib_core nouveau ast
> > i2c_algo_bit ttm mlx5_core drm_kms_helper syscopyarea sysfillrect sysimgblt
> > fb_sys_fops ahci mlxfw crc32c_vpmsum drm tg3 libahci devlink
> > [ 5295.047956] CPU: 64 PID: 15874 Comm: stress-ng-hrtim Not tainted
> > 4.15.0-23-generic #25-Ubuntu
> > [ 5295.047958] NIP: c000000000cffc04 LR: c000000000126468 CTR:
> > 0000000000000000
> > [ 5295.047962] REGS: c00000003fcffd80 TRAP: 0900 Not tainted
> > (4.15.0-23-generic)
> > [ 5295.047963] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR:
> > 28000448 XER: 00000000
> > [ 5295.047973] CFAR: c000000000126464 SOFTE: 0
> > GPR00: c000000000126468 c0002036131f7c70 c0000000016eaf00
> > c0002038f4c4b208
> > GPR04: c0002036131f7d30 0000000000000000 00000ebedb3a39a4
> > 00000ebedb3bc840
> > GPR08: 0000000000000021 0000000000000000 0000000080000040
> > 0000000052000442
> > GPR12: 0000000000000001 c00000000faac000 0000000000000000
> > 00000000000186a0
> > GPR16: 00007ffff15aaec0 00000ebedb366800 00007ffff15aaeb8
> > 00000ebedb366808
> > GPR20: 00007ffff15aaebc 00007ffff15ab17b ffffffffffffffff
> > 00007ffff15ab17a
> > GPR24: 0000000000010000 00007ab613340000 00007ffff15aad68
> > 00007ffff15aaec8
> > GPR28: 0000000000000000 0000000000040002 c0002036131f7cf0
> > c0002038f4c4b208
> > [ 5295.048016] NIP [c000000000cffc04] _raw_spin_lock_irq+0x44/0xe0
> > [ 5295.048021] LR [c000000000126468] __set_current_blocked+0x48/0xb0
> > [ 5295.048022] Call Trace:
> > [ 5295.048027] [c0002036131f7ca0] [00007ffff15aaebc] 0x7ffff15aaebc
> > [ 5295.048031] [c0002036131f7cd0] [c000000000126584]
> > signal_setup_done+0x84/0xe0
> > [ 5295.048036] [c0002036131f7d10] [c00000000001dc60] do_signal+0x110/0x2c0
> > [ 5295.048040] [c0002036131f7e00] [c00000000001dfb0]
> > do_notify_resume+0xd0/0x100
> > [ 5295.048045] [c0002036131f7e30] [c00000000000b8c4]
> > ret_from_except_lite+0x70/0x74
> > [ 5295.048046] Instruction dump:
> > [ 5295.048050] f821ffd1 7c7f1b78 39400000 892d028b 994d028b 39400000
> > 994d028d 814d0008
> > [ 5295.048074] 7d201829 2c090000 40c20010 7d40192d <40c2fff0> 7c2004ac
> > 2fa90000 409e0010
> > [ 5295.272204] Watchdog CPU:64 became unstuck
> >
> > This is on WSP DD 2.2.
> > PNOR:
> > Version of System Firmware :
> > Product Name : OpenPOWER Firmware
> > Product Extra : version-witherspoon-ibm-OP9-v2.0-2.14
> > Product Extra : occ-77bb5e6
> > Product Extra : skiboot-v6.0.1
> > Product Extra : buildroot-2018.02.1-6-ga8d1126
>
> It would be interesting to run an irqsoff latency tracer with this stress
> test. It might require kernel recompile to add the irqsoff tracer option.
>
> Where is this hrtimer stress test from?

This test from stress ng.

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Joseph Salisbury (jsalisbury)]

Just to confirm, are these the only two patches needed for this bug:
6bed3237624e3faad1592543952907cd01a42c83 powerpc: use NMI IPI for smp_send_stop
ac61c1156623455c46701654abd8c99720bceea1 powerpc: Fix smp_send_stop NMI IPI handling

[Joseph Salisbury (jsalisbury)]

I built a test kernel with commits 6bed3237624e and ac61c1156623. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1777194

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-19 05:39 EDT-------
(In reply to comment #34)
>
> I built a test kernel with commits 6bed3237624e and ac61c1156623. The test
> kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1777194
>
> Can you test this kernel and see if it resolves this bug?
>
> Note about installing test kernels:
> ? If the test kernel is prior to 4.15(Bionic) you need to install the
> linux-image and linux-image-extra .deb packages.
> ? If the test kernel is 4.15(Bionic) or newer, you need to install the
> linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.
>
> Thanks in advance!

Issue is not observed with the unsigned kernel provided above.

# uname -a
Linux ltc-wspoon12 4.15.0-23-generic #26~lp1777194 SMP Mon Jun 18 23:45:25 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Thanks,
Harish

[Joseph Salisbury (jsalisbury)]

SRU request submitted:
https://lists.ubuntu.com/archives/kernel-team/2018-June/093481.html

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-01 10:35 EDT-------
(In reply to comment #36)
> SRU request submitted:
> https://lists.ubuntu.com/archives/kernel-team/2018-June/093481.html

Do we get this fix added to 18.04.1 builds?

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-18 14:09 EDT-------
Able to reproduce the issue with the latest proposed kernel on Tuleta machine. Seems the issue is not resolved with the provided patches.

[ 155.121412] Watchdog CPU:65 Hard LOCKUP
[ 155.121416] Modules linked in: rmd256 rmd160 rmd128 vhost_net md4 vhost dccp_ipv4(+) sctp(+) unix_diag algif_hash tap dccp af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter kvm_hv kvm binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua uio_pdrv_genirq uio ipmi_powernv ipmi_devintf ipmi_msghandler powernv_rng ibmpowernv vmx_crypto leds_powernv powernv_op_panel crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas btrfs xor zstd_compress raid6_pq uas usb_storage crc32c_vpmsum ipr tg3
[ 155.121531] CPU: 65 PID: 7391 Comm: stress-ng-hrtim Not tainted 4.15.0-29-generic #31-Ubuntu
[ 155.121537] NIP: c000000000d01444 LR: c0000000001269c8 CTR: 0000000000000000
[ 155.121544] REGS: c000000007c97d80 TRAP: 0900 Not tainted (4.15.0-29-generic)
[ 155.121545] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28000448 XER: 00000000
[ 155.121560] CFAR: c0000000001269c4 SOFTE: 0
GPR00: c0000000001269c8 c000001fe581fc70 c0000000016eaf00 c000001fe57ad408
GPR04: c000001fe581fd30 0000000000000000 0000007cd2603a7c 0000007cd261c940
GPR08: 0000000000000021 0000000000000000 0000000080000041 0000000042000448
GPR12: 0000000000000001 c000000007a4cb00 0000000000000000 0000000000000000
GPR16: 00000000000186a0 00007fffd75e1ef0 0000007cd25c7040 00007fffd75e1ee8
GPR20: 0000007cd25c7048 00007fffd75e21ab ffffffffffffffff 00007fffd75e21aa
GPR24: 0000000000010000 00007896e1ca0000 00007fffd75e1da8 00007fffd75e1ef8
GPR28: 0000000000000000 0000000000040002 c000001fe581fcf0 c000001fe57ad408
[ 155.121625] NIP [c000000000d01444] _raw_spin_lock_irq+0x44/0xe0
[ 155.121631] LR [c0000000001269c8] __set_current_blocked+0x48/0xb0
[ 155.121633] Call Trace:
[ 155.121640] [c000001fe581fca0] [0000007cd25c7048] 0x7cd25c7048
[ 155.121648] [c000001fe581fcd0] [c000000000126ae4] signal_setup_done+0x84/0xe0
[ 155.121657] [c000001fe581fd10] [c00000000001da20] do_signal+0x110/0x2c0
[ 155.121665] [c000001fe581fe00] [c00000000001dd70] do_notify_resume+0xd0/0x100
[ 155.121672] [c000001fe581fe30] [c00000000000b8c4] ret_from_except_lite+0x70/0x74
[ 155.121673] Instruction dump:
[ 155.121677] f821ffd1 7c7f1b78 39400000 892d028b 994d028b 39400000 994d028d 814d0008
[ 155.121692] 7d201829 2c090000 40c20010 7d40192d <40c2fff0> 7c2004ac 2fa90000 409e0010

# uname -a
Linux lep8d 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:37:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Thanks,
Harish

[Joseph Salisbury (jsalisbury)]

I just confirmed the two requested patches are in the Ubuntu-4.15.0-29 kernel:
eeada334156f powerpc: use NMI IPI for smp_send_stop
4190e27bf0ee powerpc: Fix smp_send_stop NMI IPI handling

Can you confirm the kernel posted in comment #5 fixes this bug on the same machine?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-19 09:54 EDT-------
(In reply to comment #40)
> I just confirmed the two requested patches are in the Ubuntu-4.15.0-29
> kernel:
> eeada334156f powerpc: use NMI IPI for smp_send_stop
> 4190e27bf0ee powerpc: Fix smp_send_stop NMI IPI handling
>
> Can you confirm the kernel posted in comment #5 fixes this bug on the same
> machine?

As this is observed on Tuleta, i am not able to boot this kernel even after I used "apt-setup/proposed=true" while booting. Can you build a deb with the same?

[Joseph Salisbury (jsalisbury)]

I built a .deb version of the kernel that is currently in -proposed.
The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1777194

Can you test this kernel and see if it resolves this bug?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-20 01:38 EDT-------
(In reply to comment #42)
> I built a .deb version of the kernel that is currently in -proposed.
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1777194
>
> Can you test this kernel and see if it resolves this bug?

I am seeing the following LOCKUP on the above kernel.

# uname -a
Linux lep8d 4.15.0-29-generic #32~lp1777194 SMP Thu Jul 19 16:21:31 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

[ 349.854120] Watchdog CPU:16 Hard LOCKUP
[ 349.854129] Modules linked in: userio(E) serpent_generic(E) hci_vhci(E) bluetooth(E) ecdh_generic(E) twofish_generic(E) twofish_common(E) cuse(E) vhost_vsock(E) vmw_vsock_virtio_transport_common(E) vsock(E) lrw(E) uhid(E) hid(E) algif_skcipher(E) tgr192(E) wp512(E) rmd320(E) snd_seq(E) snd_seq_device(E) unix_diag(E) snd_timer(E) sctp(E) rmd256(E) snd(E) soundcore(E) rmd160(E) rmd128(E) md4(E) vhost_net(E) vhost(E) tap(E) algif_hash(E) dccp_ipv4(E) dccp(E) af_alg(E) xt_CHECKSUM(E) iptable_mangle(E) ipt_MASQUERADE(E) nf_nat_masquerade_ipv4(E) iptable_nat(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) xt_conntrack(E) nf_conntrack(E) libcrc32c(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_tcpudp(E) bridge(E) stp(E) llc(E) ebtable_filter(E) ebtables(E) devlink(E) ip6table_filter(E) ip6_tables(E)
[ 349.854595] iptable_filter(E) kvm_hv(E) kvm(E) binfmt_misc(E) dm_multipath(E) scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E) uio_pdrv_genirq(E) uio(E) ipmi_powernv(E) ipmi_devintf(E) ipmi_msghandler(E) vmx_crypto(E) powernv_rng(E) powernv_op_panel(E) ibmpowernv(E) leds_powernv(E) crct10dif_vpmsum(E) nfsd(E) sch_fq_codel(E) auth_rpcgss(E) nfs_acl(E) lockd(E) grace(E) sunrpc(E) ip_tables(E) x_tables(E) autofs4(E) ses(E) enclosure(E) scsi_transport_sas(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) uas(E) usb_storage(E) crc32c_vpmsum(E) tg3(E) ipr(E)
[ 349.854940] CPU: 16 PID: 1634 Comm: ureadahead Tainted: G E 4.15.0-29-generic #32~lp1777194
[ 349.854949] NIP: c00000000029738c LR: c000000000297430 CTR: c0000000000aa8a0
[ 349.854961] REGS: c000000007ee3d80 TRAP: 0900 Tainted: G E (4.15.0-29-generic)
[ 349.854993] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28002848 XER: 00000000
[ 349.855081] CFAR: c00000000029742c SOFTE: 0
GPR00: c000000000297430 c000002ffff13cc0 c0000000016eaf00 c000000fff924360
GPR04: 0000000000000010 c000000fff924360 c00000000171dd78 000000000000b000
GPR08: c000000001721ee0 0000000000000003 0000000000000000 c000000ff9004928
GPR12: c0000000000aa8a0 c000000007a2b000 0000000000000000 0000000000000001
GPR16: 0000000000240000 00000e8aa6080160 0000000000000014 0000000000000000
GPR20: 0000000000000040 c000000fa02a0000 c000000fa02a01d0 0000000000000001
GPR24: 0000000000000002 0000000000000010 c000002ff3171800 c000002ffff13ef0
GPR28: fffffffffffffff8 0000000000000002 0000000000000003 c000000ffa110450
[ 349.855354] NIP [c00000000029738c] irq_work_run_list+0x8c/0x100
[ 349.855373] LR [c000000000297430] irq_work_run+0x30/0x60
[ 349.855384] Call Trace:
[ 349.855400] [c000002ffff13cc0] [c0000000002973b0] irq_work_run_list+0xb0/0x100 (unreliable)
[ 34...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-20 02:36 EDT-------
Following lockups are observed

Jul 20 01:15:35 lep8d kernel: [ 483.899693] Watchdog CPU:126 Hard LOCKUP
Jul 20 01:15:35 lep8d kernel: [ 483.899700] Modules linked in: serpent_generic cuse twofish_generic twofish_common vhost_net tap vhost_vsock vmw_vsock_virtio_transport_common vhost vsock lrw algif_skcipher tgr192 unix_diag wp512 sctp rmd320 rmd256 snd_seq snd_seq_device snd_timer rmd160 snd soundcore rmd128 dccp_ipv4 md4 dccp algif_hash af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc devlink ip6table_filter ip6_tables iptable_filter uio_pdrv_genirq uio kvm_hv kvm leds_powernv powernv_rng vmx_crypto ipmi_powernv ipmi_devintf ibmpowernv ipmi_msghandler binfmt_misc sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables
Jul 20 01:15:35 lep8d kernel: [ 483.900118] x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear uas usb_storage crct10dif_vpmsum crc32c_vpmsum ipr tg3 [last unloaded: ebtables]
Jul 20 01:15:35 lep8d kernel: [ 483.900292] CPU: 126 PID: 70582 Comm: stress-ng-fstat Not tainted 4.15.0-29-generic #32~lp1777194
Jul 20 01:15:35 lep8d kernel: [ 483.900300] NIP: c000000000d02364 LR: c0000000001145ac CTR: c000000000162300
Jul 20 01:15:35 lep8d kernel: [ 483.900305] REGS: c00000003fa17d80 TRAP: 0900 Not tainted (4.15.0-29-generic)
Jul 20 01:15:35 lep8d kernel: [ 483.900312] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 22004244 XER: 00000000
Jul 20 01:15:35 lep8d kernel: [ 483.900360] CFAR: c000000000d0236c SOFTE: 0
Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR00: c0000000001145ac c000000fa8147d20 c0000000016eaf00 c000000001712180
Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR04: 000000000000000d 0000000000000040 0000000000000000 0000000000000842
Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR08: 0000002ffbad0000 0000000000000000 000000008000007e c000000fa81bc830
Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR12: 0000000000004400 c00000000fad6a00 0000026380759318 000074efb46d0000
Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR16: 0000026380759210 000074efbae24410 000074efbae24420 000074efb4edf240
Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR20: 0000000000000000 00007fffe93cbe40 00007fffe93cbe78 0000000000000000
Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR24: c000000fa81bc780 0000000000000000 0000000000000000 c000000001712180
Jul 20 01:15:35 lep8d kernel: [ 483.900360] GPR28: c000002d55ace160 c000000fa8147db0 0000000000000000 c000000001712180
Jul 20 01:15:35 lep8d kernel: [ 483.900551] NIP [c000000000d02364] _raw_write_lock_irq+0x74/0xe0
Jul 20 01:15:35 lep8d kernel: [ 483.900557] LR [c0000000001145ac] do_exit+0x41c/0xc80
Jul 20 01:15:35 lep8d kernel: [ 483.900565] Call Trace:
Jul 20 01:15:35 lep8d kernel: [ 483.900574] [c000000fa8147d20] [c000000fa8147d50] 0xc000000fa8147d50 (unreliable)
Jul 20 01:15:35...

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2018-07-20 02:37 EDT-------

[Kleber Sacilotto de Souza (kleber-souza)]

According to our understanding of the comments and logs provided, the original hard lockup is different to the one latest reported. So it seems safe to say testing shows the original problem is resolved and a new problem which is much less serious is observed.

So I will mark the verification of this bug as "done".

Thank you.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-29.31

---------------
linux (4.15.0-29.31) bionic; urgency=medium

  * linux: 4.15.0-29.31 -proposed tracker (LP: #1782173)

  * [SRU Bionic][Cosmic] kernel panic in ipmi_ssif at msg_done_handler
    (LP: #1777716)
    - ipmi_ssif: Fix kernel panic at msg_done_handler

  * Update to ocxl driver for 18.04.1 (LP: #1775786)
    - misc: ocxl: use put_device() instead of device_unregister()
    - powerpc: Add TIDR CPU feature for POWER9
    - powerpc: Use TIDR CPU feature to control TIDR allocation
    - powerpc: use task_pid_nr() for TID allocation
    - ocxl: Rename pnv_ocxl_spa_remove_pe to clarify it's action
    - ocxl: Expose the thread_id needed for wait on POWER9
    - ocxl: Add an IOCTL so userspace knows what OCXL features are available
    - ocxl: Document new OCXL IOCTLs
    - ocxl: Fix missing unlock on error in afu_ioctl_enable_p9_wait()

  * Critical upstream bugfix missing in Ubuntu 18.04 - frequent Xorg crash after
    suspend (LP: #1776887)
    - ocxl: Document the OCXL_IOCTL_GET_METADATA IOCTL

  * Hard LOCKUP observed on stressing Ubuntu 18 04 (LP: #1777194)
    - powerpc: use NMI IPI for smp_send_stop
    - powerpc: Fix smp_send_stop NMI IPI handling

  * IPL: ppc64_cpu --frequency hang with INFO: rcu_sched detected stalls on
    CPUs/tasks on w34 and wsbmc016 with 920.1714.20170330n (LP: #1773964)
    - rtc: opal: Fix OPAL RTC driver OPAL_BUSY loops

  * [Regression] EXT4-fs error (device sda2): ext4_validate_block_bitmap:383:
    comm stress-ng: bg 4705: bad block bitmap checksum (LP: #1781709)
    - SAUCE: Revert "UBUNTU: SAUCE: ext4: fix ext4_validate_inode_bitmap: comm
      stress-ng: Corrupt inode bitmap"
    - SAUCE: ext4: check for allocation block validity with block group locked

linux (4.15.0-28.30) bionic; urgency=medium

  * linux: 4.15.0-28.30 -proposed tracker (LP: #1781433)

  * Cannot set MTU higher than 1500 in Xen instance (LP: #1781413)
    - xen-netfront: Fix mismatched rtnl_unlock
    - xen-netfront: Update features after registering netdev

linux (4.15.0-27.29) bionic; urgency=medium

  * linux: 4.15.0-27.29 -proposed tracker (LP: #1781062)

  * [Regression] EXT4-fs error (device sda1): ext4_validate_inode_bitmap:99:
    comm stress-ng: Corrupt inode bitmap (LP: #1780137)
    - SAUCE: ext4: fix ext4_validate_inode_bitmap: comm stress-ng: Corrupt inode
      bitmap

linux (4.15.0-26.28) bionic; urgency=medium

  * linux: 4.15.0-26.28 -proposed tracker (LP: #1780112)

  * failure to boot with linux-image-4.15.0-24-generic (LP: #1779827) // Cloud-
    init causes potentially huge boot delays with 4.15 kernels (LP: #1780062)
    - random: Make getrandom() ready earlier

linux (4.15.0-25.27) bionic; urgency=medium

  * linux: 4.15.0-25.27 -proposed tracker (LP: #1779354)

  * hisi_sas_v3_hw: internal task abort: timeout and not done. (LP: #1777736)
    - scsi: hisi_sas: Update a couple of register settings for v3 hw

  * hisi_sas: Add missing PHY spinlock init (LP: #1777734)
    - scsi: hisi_sas: Add missing PHY spinlock init

  * hisi_sas: improve read performance by pre-allocating slot DMA buffers
    (LP: #1777727)
    - scsi: hisi_sas: use dma_zalloc_cohe...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-26 04:12 EDT-------
(In reply to comment #46)
> According to our understanding of the comments and logs provided, the
> original hard lockup is different to the one latest reported. So it seems
> safe to say testing shows the original problem is resolved and a new problem
> which is much less serious is observed.
>
> So I will mark the verification of this bug as "done".
>
> Thank you.

Does this mean we need to open another Bug to track the issue observed in comment 15(LP: 1777194) ?

[Joseph Salisbury (jsalisbury)]

Yes, it would probably be best to open a new bug, so we can track the new lockup separately.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-29 08:54 EDT-------
(In reply to comment #50)
> Yes, it would probably be best to open a new bug, so we can track the new
> lockup separately.

Harish: please open a new defect and let us close this as working.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-31 03:30 EDT-------
(In reply to comment #51)
> (In reply to comment #50)
> > Yes, it would probably be best to open a new bug, so we can track the new
> > lockup separately.
>
> Harish: please open a new defect and let us close this as working.

Closing this bug as this issue is not seen anymore. Will open a new bug for the other issues.

Thanks.