Ubuntu 18.04 kdump does not work on bionic-proposed 4.15 ppc64el with AC922

[bugproxy]

no guest was running, only Ubuntu 18.04 Host installed and trying kdump only on the host

===========
tried on the ubuntu released proposed kernel 4.15.0-10-generic and kdump process was able to proceed (did not dump complete vmcore though) but hit issue, please check below console log
========

root@ltciofvtr-spoon4:~# dmesg | grep -i crash
[ 0.000000] Reserving 6144MB of memory at 128MB for crashkernel (System RAM: 524288MB)
[ 0.000000] Kernel command line: root=UUID=749429f9-83b9-4776-9a3e-147e297cdf99 ro quiet splash crashkernel=6144M
root@ltciofvtr-spoon4:~# dmesg | grep -i reserv
[ 0.000000] Reserving 6144MB of memory at 128MB for crashkernel (System RAM: 524288MB)
[ 0.000000] cma: Reserved 26224 MiB at 0x0000203995000000
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] Memory: 502952192K/536870912K available (13376K kernel code, 2048K rwdata, 3648K rodata, 4800K init, 3037K bss, 7065344K reserved, 26853376K cma-reserved)
root@ltciofvtr-spoon4:~# uname -a
Linux ltciofvtr-spoon4 4.15.0-10-generic #11-Ubuntu SMP Tue Feb 13 18:21:52 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
root@ltciofvtr-spoon4:~# kdump
kdump kdump-config
root@ltciofvtr-spoon4:~# kdump
kdump kdump-config
root@ltciofvtr-spoon4:~# kdump-config
Usage: /usr/sbin/kdump-config {help|test|show|status|load|unload|savecore|propagate|symlinks kernel-version}
root@ltciofvtr-spoon4:~# kdump-config status
current state : ready to kdump
root@ltciofvtr-spoon4:~# echo 1 > /proc/sys/kernel/sysrq
root@ltciofvtr-spoon4:~# echo c > /proc/sysrq-trigger
[ 164.604204] sysrq: SysRq : Trigger a crash
[ 164.604264] Unable to handle kernel paging request for data at address 0x00000000
[ 164.604395] Faulting instruction address: 0xc0000000007ea268
[ 164.604481] Oops: Kernel access of bad area, sig: 11 [#1]
[ 164.604559] LE SMP NR_CPUS=2048 NUMA PowerNV
[ 164.604628] Modules linked in: idt_89hpesx ofpart ipmi_powernv ipmi_devintf ipmi_msghandler ibmpowernv cmdlinepart vmx_crypto powernv_flash at24 uio_pdrv_genirq uio mtd opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 mlx5_ib ib_core mlx5_core bnx2x ast i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mdio ahci mlxfw libcrc32c crc32c_vpmsum drm tg3 libahci devlink
[ 164.605176] CPU: 72 PID: 3444 Comm: bash Not tainted 4.15.0-10-generic #11-Ubuntu
[ 164.605305] NIP: c0000000007ea268 LR: c0000000007eb1a8 CTR: c0000000007ea240
[ 164.605413] REGS: c0002038f8c2f9f0 TRAP: 0300 Not tainted (4.15.0-10-generic)
[ 164.605496] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28222222 XER: 20040000
[ 164.605641] CFAR: c0000000007eb1a4 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1
[ 164.605641] GPR00: c0000000007eb1a8 c0002038f8c2fc70 c0000000016ea600 0000000000000063
[ 164.605641] GPR04: c000203993d0ce18 c000203993d24368 9000000000009033 0000000000000808
[ 164.605641] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003
[ 164.605641] GPR12: c0000000007ea240 c000000007a51800 000008f7fbb8e008 0000000000000000
[ 164.605641] GPR16: 000008f7fb19e9f0 000008f7fb231998 000008f7fb2319d0 000008f7fb268204
[ 164.605641] GPR20: 0000000000000000 0000000000000001 0000000000000000 00007fffed1e1b54
[ 164.605641] GPR24: 00007fffed1e1b50 000008f7fb26afc4 c0000000015e9930 0000000000000002
[ 164.605641] GPR28: 0000000000000063 0000000000000004 c000000001572a9c c0000000015e9cd0
[ 164.606714] NIP [c0000000007ea268] sysrq_handle_crash+0x28/0x30
[ 164.606796] LR [c0000000007eb1a8] __handle_sysrq+0xf8/0x2c0
[ 164.606865] Call Trace:
[ 164.606919] [c0002038f8c2fc70] [c0000000007eb188] __handle_sysrq+0xd8/0x2c0 (unreliable)
[ 164.607043] [c0002038f8c2fd10] [c0000000007eb9b4] write_sysrq_trigger+0x64/0x90
[ 164.607152] [c0002038f8c2fd40] [c00000000047c548] proc_reg_write+0x88/0xd0
[ 164.607252] [c0002038f8c2fd70] [c0000000003cf8dc] __vfs_write+0x3c/0x70
[ 164.607347] [c0002038f8c2fd90] [c0000000003cfb38] vfs_write+0xd8/0x220
[ 164.607445] [c0002038f8c2fde0] [c0000000003cfe58] SyS_write+0x68/0x110
[ 164.607545] [c0002038f8c2fe30] [c00000000000b184] system_call+0x58/0x6c
[ 164.607640] Instruction dump:
[ 164.607689] 4bfff9f1 4bfffe50 3c4c00f0 384203c0 7c0802a6 60000000 39200001 3d42001c
[ 164.607805] 394a76b0 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f0 38420390
[ 164.607925] ---[ end trace d9ded5212faa751b ]---
[ 165.612963]
[ 165.613070] Sending IPI to other CPUs
[ 165.623168] IPI complete
[ 165.627790] kexec: waiting for cpu 1 (physical 1) to enter OPAL
[ 165.629875] kexec: waiting fo[ 419.363342082,5] OPAL: Switch to big-endian OS
r cpu 5 (physical 5) to enter[ 423.007803159,5] OPAL: Switch to little-endian OS
 OPAL
[ 165.635949] kexec: waiting for cpu 7 (physical 7) to enter OPAL
[ 167.322686] kexec: Starting switchover sequence.
[ 1.245751] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.245755] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.314527] vio vio: uevent: failed to send synthetic uevent
/dev/sda2: recovering journal
/dev/sda2: clean, 134971/61054976 files, 5545559/244188416 blocks
[ 6.725980] vio vio: uevent: failed to send synthetic uevent
[ OK ] Started Show Plymouth Boot Screen.
plymouth-start.service
[ OK ] Reached target Local Encrypted Volumes.
[ OK ] Started Forward Password Requests to Plymouth Directory Watch.
[ OK ] Started Network Time Synchronization.
systemd-timesyncd.service
[ OK ] Reached target System Time Synchronized.
[ OK ] Started AppArmor initialization.
apparmor.service
[ OK ] Reached target System Initialization.
[ OK ] Started Wait for Network to be Configured.
systemd-networkd-wait-online.service
[ OK ] Reached target Network is Online.
         Starting Kernel crash dump capture service...
[ 24.498357] kdump-tools[1424]: Starting kdump-tools: * running makedumpfile -c -d 31 /proc/vmcore /var/crash/201802160010/dump-incomplete
[ 41.042895] Fatal Hypervisor Maintenance interrupt [Not recovered]
[ 41.042962] Error detail: Malfunction Alert
[ 41.043003] HMER: 8040000000000000
[ 41.043060] Unknown Malfunction Alert of type 0
[ 41.043113] opal: Hardware platform error: Unrecoverable HMI exception
[ 41.849182] WARNING: CPU: 2 PID: 55 at /build/linux-jWa1Fv/linux-4.15.0/kernel/sched/core.c:1188 set_task_cpu+0x240/0x250
[ 41.849297] Modules linked in: mlx5_ib ibmpowernv vmx_crypto ipmi_powernv ofpart crct10dif_vpmsum ipmi_devintf ipmi_msghandler cmdlinepart powernv_flash opal_prd mtd idt_89hpesx ib_core bnx2x at24 mdio mlx5_core mlxfw libcrc32c devlink tg3 sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum ast i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci uio_pdrv_genirq drm uio libahci
[ 41.849810] CPU: 2 PID: 55 Comm: kworker/2:1 Not tainted 4.15.0-10-generic #11-Ubuntu
[ 41.849945] Workqueue: events hmi_event_handler
[ 41.850025] NIP: c000000008149aa0 LR: c00000000814a6cc CTR: c000000008156600
[ 41.850154] REGS: c00000016feeeeb0 TRAP: 0700 Not tainted (4.15.0-10-generic)
[ 41.850270] MSR: 900000000282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28002244 XER: 00000000
[ 41.850408] CFAR: c00000000814990c SOFTE: 0
[ 41.850408] GPR00: c00000000814a6cc c00000016feef130 c0000000096ea600 c00000016f432b00
[ 41.850408] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 41.850408] GPR08: c000000009721ed8 0000000000000000 0000000000000000 0000000000000002
[ 41.850408] GPR12: 0000000028002244 c00000000fff1600 c000000008138be8 c00000016f54cf00
[ 41.850408] GPR16: 0000000000000000 0000000000000000 0000000000000000 000000017e0a0000
[ 41.850408] GPR20: c0000001872635c0 0000000000000000 0000000000000000 0000000000000000
[ 41.850408] GPR24: c00000016f432f28 c00000000971dd78 c0000000091d8580 0000000000000000
[ 41.850408] GPR28: 0000000000000004 0000000000000000 0000000000000000 c00000016f432b00
[ 41.851427] NIP [c000000008149aa0] set_task_cpu+0x240/0x250
[ 41.851507] LR [c00000000814a6cc] try_to_wake_up+0x1bc/0x660
[ 41.851605] Call Trace:
[ 41.851649] [c00000016feef130] [c0000000091d8580] runqueues+0x0/0xc00 (unreliable)
[ 41.851791] [c00000016feef170] [c00000000814a6cc] try_to_wake_up+0x1bc/0x660
[ 41.851921] [c00000016feef1f0] [c00000000816deb0] __wake_up_common+0xd0/0x200
[ 41.852053] [c00000016feef260] [c00000000843f020] ep_poll_callback+0xe0/0x2f0
[ 41.852159] [c00000016feef2c0] [c00000000816deb0] __wake_up_common+0xd0/0x200
[ 41.852271] [c00000016feef330] [c00000000816e09c] __wake_up_common_lock+0xbc/0x110
[ 41.852379] [c00000016feef3c0] [c00000000818ace0] wake_up_klogd_work_func+0x60/0xc0
[ 41.852492] [c00000016feef3f0] [c000000008291ef0] irq_work_run_list+0xb0/0x100
[ 41.852597] [c00000016feef430] [c0000000081b34e0] update_process_times+0x60/0x90
[ 41.852705] [c00000016feef460] [c0000000081cb1d4] tick_sched_handle.isra.5+0x34/0xd0
[ 41.852817] [c00000016feef490] [c0000000081cb2d0] tick_sched_timer+0x60/0xe0
[ 41.852921] [c00000016feef4d0] [c0000000081b4034] __hrtimer_run_queues+0x144/0x370
[ 41.853029] [c00000016feef550] [c0000000081b4f9c] hrtimer_interrupt+0xfc/0x350
[ 41.853137] [c00000016feef620] [c000000008024950] __timer_interrupt+0x90/0x260
[ 41.853234] [c00000016feef670] [c000000008024d68] timer_interrupt+0x98/0xe0
[ 41.853331] [c00000016feef6a0] [c000000008009014] decrementer_common+0x114/0x120
[ 41.853446] --- interrupt: 901 at replay_interrupt_return+0x0/0x4
[ 41.853446] LR = arch_local_irq_restore+0x74/0x90
[ 41.853602] [c00000016feef990] [c00000016feef9d0] 0xc00000016feef9d0 (unreliable)
[ 41.853711] [c00000016feef9b0] [c000000008d03d00] _raw_spin_unlock_irqrestore+0x40/0xa0
[ 41.853825] [c00000016feef9d0] [c00000000857923c] pstore_dump+0x31c/0x3e0
[ 41.853922] [c00000016feefb10] [c00000000818af04] kmsg_dump+0x134/0x1a0
[ 41.854017] [c00000016feefb70] [c0000000080a1214] pnv_platform_error_reboot+0x94/0x110
[ 41.854127] [c00000016feefbe0] [c0000000080a648c] hmi_event_handler+0x1bc/0x1c0
[ 41.854236] [c00000016feefc90] [c00000000812fdb8] process_one_work+0x298/0x5a0
[ 41.854341] [c00000016feefd20] [c000000008130158] worker_thread+0x98/0x630
[ 41.854438] [c00000016feefdc0] [c000000008138d88] kthread+0x1a8/0x1b0
[ 41.854533] [c00000016feefe30] [c00000000800b528] ret_from_kernel_thread+0x5c/0xb4
[ 41.854642] Instruction dump:
[ 41.854691] 7faa3670 7d4a0194 57a706be 7d4a07b4 794a1f24 7d28502a 7d293c36 71290001
[ 41.854797] 4082fe80 60000000 60000000 60420000 <0fe00000> 4bfffe6c 60000000 60420000
[ 41.854927] ---[ end trace c9f2cb34f3ff824e ]---
[ 497.382416495,0] OPAL: Reboot requested due to Platform error.
[ 497.382470331,3] OPAL: Reboot requested due to Platform error.

--== Welcome to Hostboot hostboot-7050d0a/hbicore.bin ==--

  3.99470|secure|SecureROM valid - enabling functionality
  3.99474|secure|Booting in non-secure mode.
  5.18577|ERRL|Dumping errors reported prior to registration
  5.20139|================================================
  5.20139|Error reported by ipmi (0x2500) PLID 0x90033DC5
  5.20949| Requested sensor is not present.
  5.20949| ModuleId 0x03 IPMI::MOD_IPMISENSOR
  5.20949| ReasonCode 0x2507 IPMI::RC_SENSOR_NOT_PRESENT
  5.20950| UserData1 BMC IPMI Completion code. : 0x00000000000000cb
  5.20951| UserData2 bytes [0-1]sensor name bytes [2-3]sensor number bytes [4-7]HUID of target. : 0xca22000b00010000
  5.20952|------------------------------------------------
  5.20952| Callout type : Procedure Callout
  5.20952| Procedure : EPUB_PRC_HB_CODE
  5.20953| Priority : SRCI_PRIORITY_HIGH
  5.20953|------------------------------------------------
  5.20954| Hostboot Build ID: hostboot-7050d0a/hbicore.bin
  5.20954|================================================
  6.20986|ISTEP 6. 5 - host_init_fsi
  6.34649|ISTEP 6. 6 - host_set_ipl_parms
  6.36715|ISTEP 6. 7 - host_discover_targets
  6.84607|HWAS|PRESENT> DIMM[03]=AAAA000000000000
  6.84608|HWAS|PRESENT> Proc[05]=8800000000000000
  6.84609|HWAS|PRESENT> Core[07]=CCFC3FFF03F30000
  6.86942|ISTEP 6. 8 - host_update_master_tpm

== Comment: #9 - NAVEED A. UPPINANGADY SALIH <email address hidden> - 2018-02-15 23:55:28 ==
root@ltciofvtr-spoon4:/var/crash# ls -lrht 201802160038
total 251M
-rw------- 1 root root 448M Feb 16 00:38 dump-incomplete

== Comment: #14 - Hari Krishna Bathini <email address hidden> - 2018-02-22 01:40:19 ==
The following 4 patches are missing in kexec-tools package leading to HMIs

  commit 69431282f075ab723c4886f20aa248976920aaae
  Author: Hari Bathini <email address hidden>
  Date: Tue Aug 29 23:08:02 2017 +0530

    kexec-tools: ppc64: fix leak while checking for coherent device memory

    Signed-off-by: Hari Bathini <email address hidden>
    Signed-off-by: Simon Horman <email address hidden>

  commit aec4d0f7a2502a13fc21e90ff32dc306b0ad1190
  Author: Hari Bathini <email address hidden>
  Date: Thu Aug 17 18:01:51 2017 +0530

    kexec-tools: ppc64: avoid adding coherent memory regions to crash memory ranges

    Accelerator devices like GPU and FPGA cards contain onboard memory. This
    onboard memory is represented as a memory only NUMA node, integrating it
    with core memory subsystem. Since, the link through which these devices
    are integrated to core memory goes down after a system crash and they are
    meant for user workloads, avoid adding coherent device memory regions to
    crash memory ranges. Without this change, makedumpfile tool tries to save
    unaccessible coherent device memory regions, crashing the system.

    Signed-off-by: Hari Bathini <email address hidden>
    Tested-by: Pingfan Liu <email address hidden>
    Signed-off-by: Simon Horman <email address hidden>

  commit 21eb397a5fc9227cd95d23e8c74a49cf6a293e57
  Author: Hari Bathini <email address hidden>
  Date: Wed Aug 9 23:47:42 2017 +0530

    kexec-tools: powerpc: fix command line overflow error

    Since kernel commit a5980d064fe2 ("powerpc: Bump COMMAND_LINE_SIZE
    to 2048"), powerpc bumped command line size to 2048 but the size
    used here is still the default value of 512. Bump it to 2048 to
    fix command line overflow errors observed when command line length
    is above 512 bytes. Also, get rid of the multiple definitions of
    COMMAND_LINE_SIZE macro in ppc architecture.

    Signed-off-by: Hari Bathini <email address hidden>
    Signed-off-by: Simon Horman <email address hidden>

  commit 47478ea66d4301b12a07862aebc8447a2932f0ed
  Author: Hari Bathini <email address hidden>
  Date: Wed Jul 26 22:49:41 2017 +0530

    kexec-tools: ppc64: fix how RMA top is deduced

    Hang was observed, in purgatory, on a machine configured with
    single LPAR. This was because one of the segments was loaded
    outside the actual Real Memory Area (RMA) due to wrongly
    deduced RMA top value.

    Currently, top of real memory area, which is crucial for loading
    kexec/kdump kernel, is obtained by iterating through mem nodes
    and setting its value based on the base and size values of the
    last mem node in the iteration. That can't always be correct as
    the order of iteration may not be same and RMA base & size are
    always based on the first memory property. Fix this by setting
    RMA top value based on the base and size values of the memory
    node that has the smallest base value (first memory property)
    among all the memory nodes.

    Also, correct the misnomers rmo_base and rmo_top to rma_base
    and rma_top respectively.

    While how RMA top is deduced was broken for sometime, the issue
    may not have been seen so far, for couple of possible reasons:

        1. Only one mem node was available.
        2. First memory property has been the last node in
           iteration when multiple mem nodes were present.

    Fixes: 02f4088ffded ("kexec fix ppc64 device-tree mem node")
    Reported-by: Ankit Kumar <email address hidden>
    Cc: Michael Ellerman <email address hidden>
    Cc: Geoff Levand <email address hidden>
    Signed-off-by: Hari Bathini <email address hidden>
    Signed-off-by: Simon Horman <email address hidden>
--

I would recommend upgrading to the latest kexec-tools verison 2.0.16 (which includes
this patches) instead of cherry-picking these patches..

Thanks
Hari

[bugproxy (bugproxy)]

Default Comment by Bridge

[Thadeu Lima de Souza Cascardo (cascardo)]

This is a duplicate of LP#1743529. As mentioned there, latest kexec-tools has been merged from Debian, as is awaiting for sponsorship. It is available from my personal ppa for testing.

Cascardo.

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-29 10:19 EDT-------
closing as per comment #32

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.