| 2018-06-27 05:39:41 |
bugproxy |
bug |
|
|
added bug |
| 2018-06-27 05:39:43 |
bugproxy |
tags |
|
architecture-ppc64le bugnameltc-169067 severity-high targetmilestone-inin--- |
|
| 2018-06-27 05:39:44 |
bugproxy |
attachment added |
|
kdump_consolelogs https://bugs.launchpad.net/bugs/1778844/+attachment/5157003/+files/kdump_notworking_ubunu180401.txt |
|
| 2018-06-27 05:39:47 |
bugproxy |
attachment added |
|
/var/log/syslog https://bugs.launchpad.net/bugs/1778844/+attachment/5157004/+files/bz169067_syslog |
|
| 2018-06-27 05:39:49 |
bugproxy |
attachment added |
|
firmware/opal log https://bugs.launchpad.net/bugs/1778844/+attachment/5157005/+files/bz169067_opal_log |
|
| 2018-06-27 05:39:51 |
bugproxy |
attachment added |
|
KDump failing with initrd meant for it https://bugs.launchpad.net/bugs/1778844/+attachment/5157006/+files/kdump-failure-with-kdump-initrd.log |
|
| 2018-06-27 05:39:53 |
bugproxy |
attachment added |
|
KDump kernel captures dump successfully using regular/default initrd https://bugs.launchpad.net/bugs/1778844/+attachment/5157007/+files/kdump-success-with-default-initrd.log |
|
| 2018-06-27 05:39:54 |
bugproxy |
ubuntu: assignee |
|
Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) |
|
| 2018-06-27 05:39:56 |
bugproxy |
affects |
ubuntu |
linux (Ubuntu) |
|
| 2018-06-27 07:03:31 |
Andrew Cloke |
bug task added |
|
ubuntu-power-systems |
|
| 2018-06-27 07:03:45 |
Andrew Cloke |
ubuntu-power-systems: assignee |
|
Canonical Kernel Team (canonical-kernel-team) |
|
| 2018-06-27 07:03:57 |
Andrew Cloke |
tags |
architecture-ppc64le bugnameltc-169067 severity-high targetmilestone-inin--- |
architecture-ppc64le bugnameltc-169067 p9 severity-high targetmilestone-inin--- triage-g |
|
| 2018-06-27 07:04:04 |
Andrew Cloke |
ubuntu-power-systems: importance |
Undecided |
High |
|
| 2018-07-02 14:31:45 |
Frank Heimes |
ubuntu-power-systems: status |
New |
Triaged |
|
| 2018-07-02 14:31:53 |
Frank Heimes |
tags |
architecture-ppc64le bugnameltc-169067 p9 severity-high targetmilestone-inin--- triage-g |
architecture-ppc64le bugnameltc-169067 kernel-key p9 severity-high targetmilestone-inin--- triage-g |
|
| 2018-07-02 14:32:26 |
Frank Heimes |
tags |
architecture-ppc64le bugnameltc-169067 kernel-key p9 severity-high targetmilestone-inin--- triage-g |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g |
|
| 2018-07-09 14:06:29 |
Manoj Iyer |
linux (Ubuntu): importance |
Undecided |
Critical |
|
| 2018-07-09 14:06:41 |
Manoj Iyer |
linux (Ubuntu): assignee |
Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) |
Canonical Kernel Team (canonical-kernel-team) |
|
| 2018-07-09 14:06:48 |
Manoj Iyer |
ubuntu-power-systems: importance |
High |
Critical |
|
| 2018-07-09 14:07:32 |
Manoj Iyer |
tags |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-a |
|
| 2018-07-16 23:32:47 |
Thadeu Lima de Souza Cascardo |
nominated for series |
|
Ubuntu Bionic |
|
| 2018-07-16 23:32:47 |
Thadeu Lima de Souza Cascardo |
bug task added |
|
linux (Ubuntu Bionic) |
|
| 2018-07-16 23:33:08 |
Thadeu Lima de Souza Cascardo |
bug task added |
|
makedumpfile (Ubuntu) |
|
| 2018-07-16 23:33:49 |
Thadeu Lima de Souza Cascardo |
bug task added |
|
initramfs-tools (Ubuntu) |
|
| 2018-07-20 21:56:43 |
Andrew Cloke |
ubuntu-power-systems: status |
Triaged |
Incomplete |
|
| 2018-07-23 11:19:56 |
bugproxy |
attachment added |
|
woo_kdump_nvmedisk_console logs https://bugs.launchpad.net/bugs/1778844/+attachment/5166656/+files/woo_kdump_nvmedisk_consolelogs.txt |
|
| 2018-07-23 11:19:58 |
bugproxy |
attachment added |
|
dmesg logs_initramfs state_woo https://bugs.launchpad.net/bugs/1778844/+attachment/5166657/+files/woo_initramfstate_dmeslogs.txt |
|
| 2018-07-23 12:21:13 |
bugproxy |
attachment added |
|
/tmp/initrd.img file https://bugs.launchpad.net/bugs/1778844/+attachment/5166669/+files/initrd.img |
|
| 2018-07-23 12:21:28 |
bugproxy |
attachment added |
|
sosreport_moduels=set https://bugs.launchpad.net/bugs/1778844/+attachment/5166670/+files/sosreport-woo-20180723065534.tar.xz |
|
| 2018-07-23 12:24:18 |
Andrew Cloke |
ubuntu-power-systems: status |
Incomplete |
Triaged |
|
| 2018-07-23 14:00:09 |
Manoj Iyer |
makedumpfile (Ubuntu): assignee |
|
Canonical Kernel Team (canonical-kernel-team) |
|
| 2018-07-23 14:00:22 |
Manoj Iyer |
initramfs-tools (Ubuntu): assignee |
|
Canonical Kernel Team (canonical-kernel-team) |
|
| 2018-07-23 14:00:26 |
Manoj Iyer |
initramfs-tools (Ubuntu): importance |
Undecided |
Critical |
|
| 2018-07-23 14:00:30 |
Manoj Iyer |
makedumpfile (Ubuntu): importance |
Undecided |
Critical |
|
| 2018-07-23 14:02:49 |
Frank Heimes |
tags |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-a |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g |
|
| 2018-07-30 13:40:48 |
Manoj Iyer |
ubuntu-power-systems: status |
Triaged |
Incomplete |
|
| 2018-09-20 11:41:37 |
bugproxy |
attachment added |
|
Consolelogs_wax_kdump_initrmafs state https://bugs.launchpad.net/bugs/1778844/+attachment/5190944/+files/169067_bugrecreation_wax_180401_latets_kernel.txt |
|
| 2018-09-20 11:41:50 |
bugproxy |
attachment added |
|
sosreport_kdump_initramfstsate https://bugs.launchpad.net/bugs/1778844/+attachment/5190945/+files/sosreport-wax-20180920041434.tar.xz |
|
| 2018-09-20 11:42:39 |
bugproxy |
attachment added |
|
/initrd.img file https://bugs.launchpad.net/bugs/1778844/+attachment/5190946/+files/initrd.img |
|
| 2018-09-24 14:00:45 |
Andrew Cloke |
ubuntu-power-systems: status |
Incomplete |
Triaged |
|
| 2018-09-27 19:40:28 |
Thadeu Lima de Souza Cascardo |
summary |
ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump is not working and system enters into initramfs state |
nvme multipath does not report path relationships |
|
| 2018-09-27 19:47:01 |
Thadeu Lima de Souza Cascardo |
nominated for series |
|
Ubuntu Cosmic |
|
| 2018-09-27 19:47:01 |
Thadeu Lima de Souza Cascardo |
bug task added |
|
initramfs-tools (Ubuntu Cosmic) |
|
| 2018-09-27 19:47:01 |
Thadeu Lima de Souza Cascardo |
bug task added |
|
linux (Ubuntu Cosmic) |
|
| 2018-09-27 19:47:01 |
Thadeu Lima de Souza Cascardo |
bug task added |
|
makedumpfile (Ubuntu Cosmic) |
|
| 2018-09-27 19:47:15 |
Thadeu Lima de Souza Cascardo |
makedumpfile (Ubuntu Cosmic): status |
New |
Invalid |
|
| 2018-09-27 19:47:28 |
Thadeu Lima de Souza Cascardo |
makedumpfile (Ubuntu Bionic): status |
New |
Invalid |
|
| 2018-09-27 19:47:42 |
Thadeu Lima de Souza Cascardo |
linux (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2018-09-27 19:47:53 |
Thadeu Lima de Souza Cascardo |
linux (Ubuntu Cosmic): status |
New |
Confirmed |
|
| 2018-09-27 19:48:07 |
Thadeu Lima de Souza Cascardo |
initramfs-tools (Ubuntu Cosmic): status |
New |
Confirmed |
|
| 2018-09-27 19:48:17 |
Thadeu Lima de Souza Cascardo |
initramfs-tools (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2018-10-01 13:48:39 |
Manoj Iyer |
ubuntu-power-systems: status |
Triaged |
In Progress |
|
| 2018-11-12 19:32:54 |
Terry Rudd |
bug |
|
|
added subscriber Terry Rudd |
| 2019-01-17 09:28:48 |
Andrew Cloke |
ubuntu-power-systems: status |
In Progress |
Incomplete |
|
| 2019-02-06 07:15:20 |
bugproxy |
attachment added |
|
woo_kdump_console logs https://bugs.launchpad.net/bugs/1778844/+attachment/5236276/+files/woo_consolelogs_with_patch.txt |
|
| 2019-02-06 07:43:13 |
Andrew Cloke |
ubuntu-power-systems: status |
Incomplete |
Confirmed |
|
| 2019-02-11 15:27:17 |
Andrew Cloke |
ubuntu-power-systems: status |
Confirmed |
In Progress |
|
| 2019-02-12 07:33:27 |
Andrew Cloke |
ubuntu-power-systems: status |
In Progress |
Won't Fix |
|
| 2019-02-12 07:33:57 |
Andrew Cloke |
initramfs-tools (Ubuntu): status |
Confirmed |
Invalid |
|
| 2019-02-12 07:34:08 |
Andrew Cloke |
initramfs-tools (Ubuntu Bionic): status |
Confirmed |
Invalid |
|
| 2019-02-12 07:34:16 |
Andrew Cloke |
initramfs-tools (Ubuntu Cosmic): status |
Confirmed |
Invalid |
|
| 2019-02-12 07:34:25 |
Andrew Cloke |
linux (Ubuntu): status |
Confirmed |
Invalid |
|
| 2019-02-12 07:34:34 |
Andrew Cloke |
linux (Ubuntu Bionic): status |
Confirmed |
Invalid |
|
| 2019-02-12 07:34:43 |
Andrew Cloke |
linux (Ubuntu Cosmic): status |
Confirmed |
Invalid |
|
| 2019-02-12 10:31:18 |
Thadeu Lima de Souza Cascardo |
initramfs-tools (Ubuntu): status |
Invalid |
In Progress |
|
| 2019-02-12 10:31:29 |
Thadeu Lima de Souza Cascardo |
nominated for series |
|
Ubuntu Disco |
|
| 2019-02-12 10:31:29 |
Thadeu Lima de Souza Cascardo |
bug task added |
|
initramfs-tools (Ubuntu Disco) |
|
| 2019-02-12 10:31:29 |
Thadeu Lima de Souza Cascardo |
bug task added |
|
linux (Ubuntu Disco) |
|
| 2019-02-12 10:31:29 |
Thadeu Lima de Souza Cascardo |
bug task added |
|
makedumpfile (Ubuntu Disco) |
|
| 2019-02-12 10:31:42 |
Thadeu Lima de Souza Cascardo |
initramfs-tools (Ubuntu Cosmic): status |
Invalid |
In Progress |
|
| 2019-02-12 10:31:55 |
Thadeu Lima de Souza Cascardo |
initramfs-tools (Ubuntu Bionic): status |
Invalid |
In Progress |
|
| 2019-03-26 17:52:18 |
Thadeu Lima de Souza Cascardo |
attachment added |
|
initramfs-tools_0.131ubuntu19.debdiff https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+attachment/5249521/+files/initramfs-tools_0.131ubuntu19.debdiff |
|
| 2019-03-26 17:55:20 |
Thadeu Lima de Souza Cascardo |
description |
Problem Description:
===================
After triggering crash ,kdump is not working & system enters into initramfs state
Steps to re-create:
==================
>. woo is installed ubuntu180401 kernel
root@woo:~# uname -a
Linux woo 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 17:59:00 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
root@woo:~#
>. Crashkernel value as below
root@woo:~# free -h
total used free shared buff/cache available
Mem: 503G 2.0G 501G 13M 279M 499G
Swap: 2.0G 0B 2.0G
root@woo:~# cat /proc/cmdline
root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet crashkernel=8192M
> kdump status
root@woo:~# kdump-config status
current state : ready to kdump
root@woo:~# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.15.0-23-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.15.0-23-generic
current state: ready to kdump
kexec command:
/sbin/kexec -p --command-line="root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet nr_cpus=1 systemd.unit=kdump-tools.service irqpoll noirqdistrib nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
root@woo:~# dmesg | grep Reser
[ 0.000000] Reserving 8192MB of memory at 128MB for crashkernel (System RAM: 524288MB)
[ 0.000000] cma: Reserved 26224 MiB at 0x0000203995000000
[ 3.545490] Copyright (C) 2017-2018 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries.
> Triggered crash
root@woo:~# echo 1 > /proc/sys/kernel/sysrq
root@woo:~# echo c > /proc/sysrq-trigger
[ 73.056308] sysrq: SysRq : Trigger a crash
[ 73.056357] Unable to handle kernel paging request for data at address 0x00000000
[ 73.056459] Faulting instruction address: 0xc0000000007f24c8
[ 73.056543] Oops: Kernel access of bad area, sig: 11 [#1]
[ 73.056609] LE SMP NR_CPUS=2048 NUMA PowerNV
[ 73.056668] Modules linked in: rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc idt_89hpesx vmx_crypto crct10dif_vpmsum ofpart cmdlinepart ipmi_powernv ipmi_devintf at24 powernv_flash ipmi_msghandler ibmpowernv mtd opal_prd uio_pdrv_genirq uio nfsd auth_rpcgss nfs_acl lockd sch_fq_codel grace sunrpc knem(OE) ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq mlx5_ib(OE) ib_core(OE) nouveau lpfc ast i2c_algo_bit ttm mlx5_core(OE) drm_kms_helper mlxfw(OE) nvmet_fc devlink syscopyarea nvmet mlx_compat(OE) sysfillrect cxl nvme_fc sysimgblt fb_sys_fops nvme_fabrics nvme ahci crc32c_vpmsum drm scsi_transport_fc
[ 73.057601] tg3 libahci nvme_core pnv_php
[ 73.057652] CPU: 44 PID: 4626 Comm: bash Tainted: G OE 4.15.0-23-generic #25-Ubuntu
[ 73.057767] NIP: c0000000007f24c8 LR: c0000000007f3568 CTR: c0000000007f24a0
[ 73.057868] REGS: c000003f8269f9f0 TRAP: 0300 Tainted: G OE (4.15.0-23-generic)
[ 73.057986] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28222222 XER: 20040000
[ 73.058099] CFAR: c0000000007f3564 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1
[ 73.058099] GPR00: c0000000007f3568 c000003f8269fc70 c0000000016eaf00 0000000000000063
[ 73.058099] GPR04: c000003fef47ce18 c000003fef494368 9000000000009033 0000000031da0058
[ 73.058099] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003
[ 73.058099] GPR12: c0000000007f24a0 c000000007a2e400 00000e4fa497c900 0000000000000000
[ 73.058099] GPR16: 00000e4f79cc94b0 00000e4f79d567e0 00000e4f79d88204 00000e4f79d56818
[ 73.058099] GPR20: 00000e4f79d8d5d8 0000000000000001 0000000000000000 00007ffffefce644
[ 73.058099] GPR24: 00007ffffefce640 00000e4f79d8afb4 c0000000015e9aa8 0000000000000002
[ 73.058099] GPR28: 0000000000000063 0000000000000004 c000000001572b1c c0000000015e9e68
[ 73.059060] NIP [c0000000007f24c8] sysrq_handle_crash+0x28/0x30
[ 73.059142] LR [c0000000007f3568] __handle_sysrq+0xf8/0x2c0
[ 73.059215] Call Trace:
[ 73.059254] [c000003f8269fc70] [c0000000007f3548] __handle_sysrq+0xd8/0x2c0 (unreliable)
[ 73.059358] [c000003f8269fd10] [c0000000007f3d74] write_sysrq_trigger+0x64/0x90
[ 73.059456] [c000003f8269fd40] [c000000000481248] proc_reg_write+0x88/0xd0
[ 73.059543] [c000003f8269fd70] [c0000000003d43fc] __vfs_write+0x3c/0x70
[ 73.059627] [c000003f8269fd90] [c0000000003d4658] vfs_write+0xd8/0x220
[ 73.059716] [c000003f8269fde0] [c0000000003d4978] SyS_write+0x68/0x110
[ 73.059809] [c000003f8269fe30] [c00000000000b284] system_call+0x58/0x6c
[ 73.059896] Instruction dump:
[ 73.059940] 4bfff9f1 4bfffe50 3c4c00f0 38428a60 7c0802a6 60000000 39200001 3d42001c
[ 73.060040] 394a6db0 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f0 38428a30
[ 73.060159] ---[ end trace e116d2421d2f59a5 ]---
[ 74.067059]
[ 74.067172] Sending IPI to other CPUs
[ 75.851509[ 202.275317797,5] OPAL: Switch to big-endian OS
] IPI complet[ 207.151277658,5] OPAL: Switch to little-endian OS
[ 232.159296542,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
e
[ 78.164472] kexec: Starting switchover sequence.
[ 1.412463] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.412468] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.481335] vio vio: uevent: failed to send synthetic uevent
[ 2.534732] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.534847] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.534967] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.535144] nouveau 0035:04:00.0: unknown chipset (140000a1)
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
(initramfs)
== Comment: #1 - INDIRA P. JOGA <> - 2018-06-21 01:19:41 ==
Attached woo console logs for kdump issue
== Comment: #4 - INDIRA P. JOGA <> - 2018-06-26 01:51:47 ==
I have triggered crash & sits here
[ 1032.259696471,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
omplete
[ 823.882048] kexec: Starting switchover sequence.
[ 1.154056] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.154060] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.222719] vio vio: uevent: failed to send synthetic uevent
[ 2.212065] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.214995] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.215259] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.215408] nouveau 0035:04:00.0: unknown chipset (140000a1)
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
== Comment: #5 - INDIRA P. JOGA <> - 2018-06-26 02:40:56 ==
NOTE:
Used nvme disk as root disk here.
== Comment: #8 - Hari Krishna Bathini <> - 2018-06-26 06:06:13 ==
The dump target (/var/crash) is on NVMe device (also, the root disk).
But the kdump initrd is not being built with nvme driver modules.
Eventually, nvme disk is not found and kdump kernel is hitting the
initramfs shell. Using the default initrd, which has the nvme driver
modules included, dump was captured successfully.
Can someone from Canonical take a look at this and comment on why
nvme modules are not included in kdump initrd despite it being the
root disk..
Thanks
Hari |
[Impact]
initramfs created with MODULES=dep or kdump initrd won't boot a system with root filesystem on a multipath nvme.
[Test case]
Systems with nvme multipath were able to boot with the created initramfs. Also tested on systems with non-multipath nvme, and non nvme systems.
[Regression potential]
A system could fail to boot because the generated initramfs was broken. The code should just add modules, which is safer than removing modules or doing any other changes. In any case, it was tested to boot on multipath nvme, non multipath nvme and non nvme systems.
-----------------
Problem Description:
===================
After triggering crash ,kdump is not working & system enters into initramfs state
Steps to re-create:
==================
>. woo is installed ubuntu180401 kernel
root@woo:~# uname -a
Linux woo 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 17:59:00 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
root@woo:~#
>. Crashkernel value as below
root@woo:~# free -h
total used free shared buff/cache available
Mem: 503G 2.0G 501G 13M 279M 499G
Swap: 2.0G 0B 2.0G
root@woo:~# cat /proc/cmdline
root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet crashkernel=8192M
> kdump status
root@woo:~# kdump-config status
current state : ready to kdump
root@woo:~# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.15.0-23-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.15.0-23-generic
current state: ready to kdump
kexec command:
/sbin/kexec -p --command-line="root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet nr_cpus=1 systemd.unit=kdump-tools.service irqpoll noirqdistrib nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
root@woo:~# dmesg | grep Reser
[ 0.000000] Reserving 8192MB of memory at 128MB for crashkernel (System RAM: 524288MB)
[ 0.000000] cma: Reserved 26224 MiB at 0x0000203995000000
[ 3.545490] Copyright (C) 2017-2018 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries.
> Triggered crash
root@woo:~# echo 1 > /proc/sys/kernel/sysrq
root@woo:~# echo c > /proc/sysrq-trigger
[ 73.056308] sysrq: SysRq : Trigger a crash
[ 73.056357] Unable to handle kernel paging request for data at address 0x00000000
[ 73.056459] Faulting instruction address: 0xc0000000007f24c8
[ 73.056543] Oops: Kernel access of bad area, sig: 11 [#1]
[ 73.056609] LE SMP NR_CPUS=2048 NUMA PowerNV
[ 73.056668] Modules linked in: rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc idt_89hpesx vmx_crypto crct10dif_vpmsum ofpart cmdlinepart ipmi_powernv ipmi_devintf at24 powernv_flash ipmi_msghandler ibmpowernv mtd opal_prd uio_pdrv_genirq uio nfsd auth_rpcgss nfs_acl lockd sch_fq_codel grace sunrpc knem(OE) ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq mlx5_ib(OE) ib_core(OE) nouveau lpfc ast i2c_algo_bit ttm mlx5_core(OE) drm_kms_helper mlxfw(OE) nvmet_fc devlink syscopyarea nvmet mlx_compat(OE) sysfillrect cxl nvme_fc sysimgblt fb_sys_fops nvme_fabrics nvme ahci crc32c_vpmsum drm scsi_transport_fc
[ 73.057601] tg3 libahci nvme_core pnv_php
[ 73.057652] CPU: 44 PID: 4626 Comm: bash Tainted: G OE 4.15.0-23-generic #25-Ubuntu
[ 73.057767] NIP: c0000000007f24c8 LR: c0000000007f3568 CTR: c0000000007f24a0
[ 73.057868] REGS: c000003f8269f9f0 TRAP: 0300 Tainted: G OE (4.15.0-23-generic)
[ 73.057986] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28222222 XER: 20040000
[ 73.058099] CFAR: c0000000007f3564 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1
[ 73.058099] GPR00: c0000000007f3568 c000003f8269fc70 c0000000016eaf00 0000000000000063
[ 73.058099] GPR04: c000003fef47ce18 c000003fef494368 9000000000009033 0000000031da0058
[ 73.058099] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003
[ 73.058099] GPR12: c0000000007f24a0 c000000007a2e400 00000e4fa497c900 0000000000000000
[ 73.058099] GPR16: 00000e4f79cc94b0 00000e4f79d567e0 00000e4f79d88204 00000e4f79d56818
[ 73.058099] GPR20: 00000e4f79d8d5d8 0000000000000001 0000000000000000 00007ffffefce644
[ 73.058099] GPR24: 00007ffffefce640 00000e4f79d8afb4 c0000000015e9aa8 0000000000000002
[ 73.058099] GPR28: 0000000000000063 0000000000000004 c000000001572b1c c0000000015e9e68
[ 73.059060] NIP [c0000000007f24c8] sysrq_handle_crash+0x28/0x30
[ 73.059142] LR [c0000000007f3568] __handle_sysrq+0xf8/0x2c0
[ 73.059215] Call Trace:
[ 73.059254] [c000003f8269fc70] [c0000000007f3548] __handle_sysrq+0xd8/0x2c0 (unreliable)
[ 73.059358] [c000003f8269fd10] [c0000000007f3d74] write_sysrq_trigger+0x64/0x90
[ 73.059456] [c000003f8269fd40] [c000000000481248] proc_reg_write+0x88/0xd0
[ 73.059543] [c000003f8269fd70] [c0000000003d43fc] __vfs_write+0x3c/0x70
[ 73.059627] [c000003f8269fd90] [c0000000003d4658] vfs_write+0xd8/0x220
[ 73.059716] [c000003f8269fde0] [c0000000003d4978] SyS_write+0x68/0x110
[ 73.059809] [c000003f8269fe30] [c00000000000b284] system_call+0x58/0x6c
[ 73.059896] Instruction dump:
[ 73.059940] 4bfff9f1 4bfffe50 3c4c00f0 38428a60 7c0802a6 60000000 39200001 3d42001c
[ 73.060040] 394a6db0 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f0 38428a30
[ 73.060159] ---[ end trace e116d2421d2f59a5 ]---
[ 74.067059]
[ 74.067172] Sending IPI to other CPUs
[ 75.851509[ 202.275317797,5] OPAL: Switch to big-endian OS
] IPI complet[ 207.151277658,5] OPAL: Switch to little-endian OS
[ 232.159296542,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
e
[ 78.164472] kexec: Starting switchover sequence.
[ 1.412463] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.412468] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.481335] vio vio: uevent: failed to send synthetic uevent
[ 2.534732] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.534847] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.534967] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.535144] nouveau 0035:04:00.0: unknown chipset (140000a1)
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
(initramfs)
== Comment: #1 - INDIRA P. JOGA <> - 2018-06-21 01:19:41 ==
Attached woo console logs for kdump issue
== Comment: #4 - INDIRA P. JOGA <> - 2018-06-26 01:51:47 ==
I have triggered crash & sits here
[ 1032.259696471,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
omplete
[ 823.882048] kexec: Starting switchover sequence.
[ 1.154056] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.154060] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.222719] vio vio: uevent: failed to send synthetic uevent
[ 2.212065] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.214995] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.215259] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.215408] nouveau 0035:04:00.0: unknown chipset (140000a1)
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
== Comment: #5 - INDIRA P. JOGA <> - 2018-06-26 02:40:56 ==
NOTE:
Used nvme disk as root disk here.
== Comment: #8 - Hari Krishna Bathini <> - 2018-06-26 06:06:13 ==
The dump target (/var/crash) is on NVMe device (also, the root disk).
But the kdump initrd is not being built with nvme driver modules.
Eventually, nvme disk is not found and kdump kernel is hitting the
initramfs shell. Using the default initrd, which has the nvme driver
modules included, dump was captured successfully.
Can someone from Canonical take a look at this and comment on why
nvme modules are not included in kdump initrd despite it being the
root disk..
Thanks
Hari |
|
| 2019-03-26 17:56:20 |
Thadeu Lima de Souza Cascardo |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2019-03-26 18:09:13 |
Andrew Cloke |
ubuntu-power-systems: status |
Won't Fix |
In Progress |
|
| 2019-03-31 21:00:45 |
Mathew Hodson |
bug task deleted |
makedumpfile (Ubuntu) |
|
|
| 2019-03-31 21:00:54 |
Mathew Hodson |
bug task deleted |
makedumpfile (Ubuntu Bionic) |
|
|
| 2019-03-31 21:01:02 |
Mathew Hodson |
bug task deleted |
makedumpfile (Ubuntu Cosmic) |
|
|
| 2019-03-31 21:01:11 |
Mathew Hodson |
bug task deleted |
makedumpfile (Ubuntu Disco) |
|
|
| 2019-03-31 21:04:15 |
Mathew Hodson |
initramfs-tools (Ubuntu Bionic): importance |
Undecided |
Critical |
|
| 2019-03-31 21:04:30 |
Mathew Hodson |
bug |
|
|
added subscriber Mathew Hodson |
| 2019-04-12 19:24:43 |
Steve Langasek |
initramfs-tools (Ubuntu Disco): status |
In Progress |
Fix Committed |
|
| 2019-04-12 19:25:24 |
Steve Langasek |
bug |
|
|
added subscriber Steve Langasek |
| 2019-04-12 20:29:06 |
Launchpad Janitor |
initramfs-tools (Ubuntu Disco): status |
Fix Committed |
Fix Released |
|
| 2019-04-13 19:34:28 |
Mathew Hodson |
bug task deleted |
linux (Ubuntu) |
|
|
| 2019-04-13 19:34:38 |
Mathew Hodson |
bug task deleted |
linux (Ubuntu Bionic) |
|
|
| 2019-04-13 19:34:46 |
Mathew Hodson |
bug task deleted |
linux (Ubuntu Cosmic) |
|
|
| 2019-04-13 19:34:53 |
Mathew Hodson |
bug task deleted |
linux (Ubuntu Disco) |
|
|
| 2019-04-13 19:34:59 |
Mathew Hodson |
removed subscriber Mathew Hodson |
|
|
|
| 2019-04-17 00:23:50 |
Thadeu Lima de Souza Cascardo |
attachment added |
|
fix for cosmic https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1778844/+attachment/5256423/+files/initramfs-tools_0.131ubuntu15.2.debdiff |
|
| 2019-04-17 00:24:13 |
Thadeu Lima de Souza Cascardo |
initramfs-tools (Ubuntu Cosmic): assignee |
Canonical Kernel Team (canonical-kernel-team) |
Thadeu Lima de Souza Cascardo (cascardo) |
|
| 2019-04-17 00:24:17 |
Thadeu Lima de Souza Cascardo |
initramfs-tools (Ubuntu Bionic): assignee |
|
Thadeu Lima de Souza Cascardo (cascardo) |
|
| 2019-04-17 00:27:39 |
Thadeu Lima de Souza Cascardo |
attachment added |
|
fix for bionic https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1778844/+attachment/5256424/+files/initramfs-tools_0.130ubuntu3.8.debdiff |
|
| 2019-04-20 23:11:04 |
Steve Langasek |
initramfs-tools (Ubuntu Bionic): assignee |
Thadeu Lima de Souza Cascardo (cascardo) |
Steve Langasek (vorlon) |
|
| 2019-04-22 14:33:53 |
Steve Langasek |
initramfs-tools (Ubuntu Bionic): status |
In Progress |
Incomplete |
|
| 2019-04-22 14:33:53 |
Steve Langasek |
initramfs-tools (Ubuntu Bionic): assignee |
Steve Langasek (vorlon) |
Thadeu Lima de Souza Cascardo (cascardo) |
|
| 2019-04-22 15:03:20 |
Thadeu Lima de Souza Cascardo |
description |
[Impact]
initramfs created with MODULES=dep or kdump initrd won't boot a system with root filesystem on a multipath nvme.
[Test case]
Systems with nvme multipath were able to boot with the created initramfs. Also tested on systems with non-multipath nvme, and non nvme systems.
[Regression potential]
A system could fail to boot because the generated initramfs was broken. The code should just add modules, which is safer than removing modules or doing any other changes. In any case, it was tested to boot on multipath nvme, non multipath nvme and non nvme systems.
-----------------
Problem Description:
===================
After triggering crash ,kdump is not working & system enters into initramfs state
Steps to re-create:
==================
>. woo is installed ubuntu180401 kernel
root@woo:~# uname -a
Linux woo 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 17:59:00 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
root@woo:~#
>. Crashkernel value as below
root@woo:~# free -h
total used free shared buff/cache available
Mem: 503G 2.0G 501G 13M 279M 499G
Swap: 2.0G 0B 2.0G
root@woo:~# cat /proc/cmdline
root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet crashkernel=8192M
> kdump status
root@woo:~# kdump-config status
current state : ready to kdump
root@woo:~# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.15.0-23-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.15.0-23-generic
current state: ready to kdump
kexec command:
/sbin/kexec -p --command-line="root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet nr_cpus=1 systemd.unit=kdump-tools.service irqpoll noirqdistrib nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
root@woo:~# dmesg | grep Reser
[ 0.000000] Reserving 8192MB of memory at 128MB for crashkernel (System RAM: 524288MB)
[ 0.000000] cma: Reserved 26224 MiB at 0x0000203995000000
[ 3.545490] Copyright (C) 2017-2018 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries.
> Triggered crash
root@woo:~# echo 1 > /proc/sys/kernel/sysrq
root@woo:~# echo c > /proc/sysrq-trigger
[ 73.056308] sysrq: SysRq : Trigger a crash
[ 73.056357] Unable to handle kernel paging request for data at address 0x00000000
[ 73.056459] Faulting instruction address: 0xc0000000007f24c8
[ 73.056543] Oops: Kernel access of bad area, sig: 11 [#1]
[ 73.056609] LE SMP NR_CPUS=2048 NUMA PowerNV
[ 73.056668] Modules linked in: rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc idt_89hpesx vmx_crypto crct10dif_vpmsum ofpart cmdlinepart ipmi_powernv ipmi_devintf at24 powernv_flash ipmi_msghandler ibmpowernv mtd opal_prd uio_pdrv_genirq uio nfsd auth_rpcgss nfs_acl lockd sch_fq_codel grace sunrpc knem(OE) ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq mlx5_ib(OE) ib_core(OE) nouveau lpfc ast i2c_algo_bit ttm mlx5_core(OE) drm_kms_helper mlxfw(OE) nvmet_fc devlink syscopyarea nvmet mlx_compat(OE) sysfillrect cxl nvme_fc sysimgblt fb_sys_fops nvme_fabrics nvme ahci crc32c_vpmsum drm scsi_transport_fc
[ 73.057601] tg3 libahci nvme_core pnv_php
[ 73.057652] CPU: 44 PID: 4626 Comm: bash Tainted: G OE 4.15.0-23-generic #25-Ubuntu
[ 73.057767] NIP: c0000000007f24c8 LR: c0000000007f3568 CTR: c0000000007f24a0
[ 73.057868] REGS: c000003f8269f9f0 TRAP: 0300 Tainted: G OE (4.15.0-23-generic)
[ 73.057986] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28222222 XER: 20040000
[ 73.058099] CFAR: c0000000007f3564 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1
[ 73.058099] GPR00: c0000000007f3568 c000003f8269fc70 c0000000016eaf00 0000000000000063
[ 73.058099] GPR04: c000003fef47ce18 c000003fef494368 9000000000009033 0000000031da0058
[ 73.058099] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003
[ 73.058099] GPR12: c0000000007f24a0 c000000007a2e400 00000e4fa497c900 0000000000000000
[ 73.058099] GPR16: 00000e4f79cc94b0 00000e4f79d567e0 00000e4f79d88204 00000e4f79d56818
[ 73.058099] GPR20: 00000e4f79d8d5d8 0000000000000001 0000000000000000 00007ffffefce644
[ 73.058099] GPR24: 00007ffffefce640 00000e4f79d8afb4 c0000000015e9aa8 0000000000000002
[ 73.058099] GPR28: 0000000000000063 0000000000000004 c000000001572b1c c0000000015e9e68
[ 73.059060] NIP [c0000000007f24c8] sysrq_handle_crash+0x28/0x30
[ 73.059142] LR [c0000000007f3568] __handle_sysrq+0xf8/0x2c0
[ 73.059215] Call Trace:
[ 73.059254] [c000003f8269fc70] [c0000000007f3548] __handle_sysrq+0xd8/0x2c0 (unreliable)
[ 73.059358] [c000003f8269fd10] [c0000000007f3d74] write_sysrq_trigger+0x64/0x90
[ 73.059456] [c000003f8269fd40] [c000000000481248] proc_reg_write+0x88/0xd0
[ 73.059543] [c000003f8269fd70] [c0000000003d43fc] __vfs_write+0x3c/0x70
[ 73.059627] [c000003f8269fd90] [c0000000003d4658] vfs_write+0xd8/0x220
[ 73.059716] [c000003f8269fde0] [c0000000003d4978] SyS_write+0x68/0x110
[ 73.059809] [c000003f8269fe30] [c00000000000b284] system_call+0x58/0x6c
[ 73.059896] Instruction dump:
[ 73.059940] 4bfff9f1 4bfffe50 3c4c00f0 38428a60 7c0802a6 60000000 39200001 3d42001c
[ 73.060040] 394a6db0 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f0 38428a30
[ 73.060159] ---[ end trace e116d2421d2f59a5 ]---
[ 74.067059]
[ 74.067172] Sending IPI to other CPUs
[ 75.851509[ 202.275317797,5] OPAL: Switch to big-endian OS
] IPI complet[ 207.151277658,5] OPAL: Switch to little-endian OS
[ 232.159296542,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
e
[ 78.164472] kexec: Starting switchover sequence.
[ 1.412463] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.412468] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.481335] vio vio: uevent: failed to send synthetic uevent
[ 2.534732] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.534847] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.534967] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.535144] nouveau 0035:04:00.0: unknown chipset (140000a1)
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
(initramfs)
== Comment: #1 - INDIRA P. JOGA <> - 2018-06-21 01:19:41 ==
Attached woo console logs for kdump issue
== Comment: #4 - INDIRA P. JOGA <> - 2018-06-26 01:51:47 ==
I have triggered crash & sits here
[ 1032.259696471,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
omplete
[ 823.882048] kexec: Starting switchover sequence.
[ 1.154056] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.154060] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.222719] vio vio: uevent: failed to send synthetic uevent
[ 2.212065] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.214995] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.215259] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.215408] nouveau 0035:04:00.0: unknown chipset (140000a1)
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
== Comment: #5 - INDIRA P. JOGA <> - 2018-06-26 02:40:56 ==
NOTE:
Used nvme disk as root disk here.
== Comment: #8 - Hari Krishna Bathini <> - 2018-06-26 06:06:13 ==
The dump target (/var/crash) is on NVMe device (also, the root disk).
But the kdump initrd is not being built with nvme driver modules.
Eventually, nvme disk is not found and kdump kernel is hitting the
initramfs shell. Using the default initrd, which has the nvme driver
modules included, dump was captured successfully.
Can someone from Canonical take a look at this and comment on why
nvme modules are not included in kdump initrd despite it being the
root disk..
Thanks
Hari |
[Impact]
initramfs created with MODULES=dep or kdump initrd won't boot a system with root filesystem on a multipath nvme.
[Test case]
Systems with nvme multipath were able to boot with the created initramfs. Also tested on systems with non-multipath nvme, and non nvme systems.
In order to verify the fix, one needs to change MODULES option to dep on /etc/initramfs-tools/initramfs.conf, recreate initramfs and reboot, check the system has booted fine. That should not break on systems with non nvme disks or systems with non multipath nvme systems, and that should now work on multipath nvme systems.
sed -i /MODULES=/s,=.*,=dep, /etc/initramfs-tools/initramfs.conf
update-initramfs -u -k all
reboot
[Regression potential]
A system could fail to boot because the generated initramfs was broken. The code should just add modules, which is safer than removing modules or doing any other changes. In any case, it was tested to boot on multipath nvme, non multipath nvme and non nvme systems.
-----------------
Problem Description:
===================
After triggering crash ,kdump is not working & system enters into initramfs state
Steps to re-create:
==================
>. woo is installed ubuntu180401 kernel
root@woo:~# uname -a
Linux woo 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 17:59:00 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
root@woo:~#
>. Crashkernel value as below
root@woo:~# free -h
total used free shared buff/cache available
Mem: 503G 2.0G 501G 13M 279M 499G
Swap: 2.0G 0B 2.0G
root@woo:~# cat /proc/cmdline
root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet crashkernel=8192M
> kdump status
root@woo:~# kdump-config status
current state : ready to kdump
root@woo:~# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.15.0-23-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.15.0-23-generic
current state: ready to kdump
kexec command:
/sbin/kexec -p --command-line="root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet nr_cpus=1 systemd.unit=kdump-tools.service irqpoll noirqdistrib nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
root@woo:~# dmesg | grep Reser
[ 0.000000] Reserving 8192MB of memory at 128MB for crashkernel (System RAM: 524288MB)
[ 0.000000] cma: Reserved 26224 MiB at 0x0000203995000000
[ 3.545490] Copyright (C) 2017-2018 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries.
> Triggered crash
root@woo:~# echo 1 > /proc/sys/kernel/sysrq
root@woo:~# echo c > /proc/sysrq-trigger
[ 73.056308] sysrq: SysRq : Trigger a crash
[ 73.056357] Unable to handle kernel paging request for data at address 0x00000000
[ 73.056459] Faulting instruction address: 0xc0000000007f24c8
[ 73.056543] Oops: Kernel access of bad area, sig: 11 [#1]
[ 73.056609] LE SMP NR_CPUS=2048 NUMA PowerNV
[ 73.056668] Modules linked in: rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc idt_89hpesx vmx_crypto crct10dif_vpmsum ofpart cmdlinepart ipmi_powernv ipmi_devintf at24 powernv_flash ipmi_msghandler ibmpowernv mtd opal_prd uio_pdrv_genirq uio nfsd auth_rpcgss nfs_acl lockd sch_fq_codel grace sunrpc knem(OE) ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq mlx5_ib(OE) ib_core(OE) nouveau lpfc ast i2c_algo_bit ttm mlx5_core(OE) drm_kms_helper mlxfw(OE) nvmet_fc devlink syscopyarea nvmet mlx_compat(OE) sysfillrect cxl nvme_fc sysimgblt fb_sys_fops nvme_fabrics nvme ahci crc32c_vpmsum drm scsi_transport_fc
[ 73.057601] tg3 libahci nvme_core pnv_php
[ 73.057652] CPU: 44 PID: 4626 Comm: bash Tainted: G OE 4.15.0-23-generic #25-Ubuntu
[ 73.057767] NIP: c0000000007f24c8 LR: c0000000007f3568 CTR: c0000000007f24a0
[ 73.057868] REGS: c000003f8269f9f0 TRAP: 0300 Tainted: G OE (4.15.0-23-generic)
[ 73.057986] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28222222 XER: 20040000
[ 73.058099] CFAR: c0000000007f3564 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1
[ 73.058099] GPR00: c0000000007f3568 c000003f8269fc70 c0000000016eaf00 0000000000000063
[ 73.058099] GPR04: c000003fef47ce18 c000003fef494368 9000000000009033 0000000031da0058
[ 73.058099] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003
[ 73.058099] GPR12: c0000000007f24a0 c000000007a2e400 00000e4fa497c900 0000000000000000
[ 73.058099] GPR16: 00000e4f79cc94b0 00000e4f79d567e0 00000e4f79d88204 00000e4f79d56818
[ 73.058099] GPR20: 00000e4f79d8d5d8 0000000000000001 0000000000000000 00007ffffefce644
[ 73.058099] GPR24: 00007ffffefce640 00000e4f79d8afb4 c0000000015e9aa8 0000000000000002
[ 73.058099] GPR28: 0000000000000063 0000000000000004 c000000001572b1c c0000000015e9e68
[ 73.059060] NIP [c0000000007f24c8] sysrq_handle_crash+0x28/0x30
[ 73.059142] LR [c0000000007f3568] __handle_sysrq+0xf8/0x2c0
[ 73.059215] Call Trace:
[ 73.059254] [c000003f8269fc70] [c0000000007f3548] __handle_sysrq+0xd8/0x2c0 (unreliable)
[ 73.059358] [c000003f8269fd10] [c0000000007f3d74] write_sysrq_trigger+0x64/0x90
[ 73.059456] [c000003f8269fd40] [c000000000481248] proc_reg_write+0x88/0xd0
[ 73.059543] [c000003f8269fd70] [c0000000003d43fc] __vfs_write+0x3c/0x70
[ 73.059627] [c000003f8269fd90] [c0000000003d4658] vfs_write+0xd8/0x220
[ 73.059716] [c000003f8269fde0] [c0000000003d4978] SyS_write+0x68/0x110
[ 73.059809] [c000003f8269fe30] [c00000000000b284] system_call+0x58/0x6c
[ 73.059896] Instruction dump:
[ 73.059940] 4bfff9f1 4bfffe50 3c4c00f0 38428a60 7c0802a6 60000000 39200001 3d42001c
[ 73.060040] 394a6db0 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f0 38428a30
[ 73.060159] ---[ end trace e116d2421d2f59a5 ]---
[ 74.067059]
[ 74.067172] Sending IPI to other CPUs
[ 75.851509[ 202.275317797,5] OPAL: Switch to big-endian OS
] IPI complet[ 207.151277658,5] OPAL: Switch to little-endian OS
[ 232.159296542,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
e
[ 78.164472] kexec: Starting switchover sequence.
[ 1.412463] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.412468] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.481335] vio vio: uevent: failed to send synthetic uevent
[ 2.534732] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.534847] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.534967] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.535144] nouveau 0035:04:00.0: unknown chipset (140000a1)
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
(initramfs)
== Comment: #1 - INDIRA P. JOGA <> - 2018-06-21 01:19:41 ==
Attached woo console logs for kdump issue
== Comment: #4 - INDIRA P. JOGA <> - 2018-06-26 01:51:47 ==
I have triggered crash & sits here
[ 1032.259696471,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
omplete
[ 823.882048] kexec: Starting switchover sequence.
[ 1.154056] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.154060] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.222719] vio vio: uevent: failed to send synthetic uevent
[ 2.212065] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.214995] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.215259] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.215408] nouveau 0035:04:00.0: unknown chipset (140000a1)
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
== Comment: #5 - INDIRA P. JOGA <> - 2018-06-26 02:40:56 ==
NOTE:
Used nvme disk as root disk here.
== Comment: #8 - Hari Krishna Bathini <> - 2018-06-26 06:06:13 ==
The dump target (/var/crash) is on NVMe device (also, the root disk).
But the kdump initrd is not being built with nvme driver modules.
Eventually, nvme disk is not found and kdump kernel is hitting the
initramfs shell. Using the default initrd, which has the nvme driver
modules included, dump was captured successfully.
Can someone from Canonical take a look at this and comment on why
nvme modules are not included in kdump initrd despite it being the
root disk..
Thanks
Hari |
|
| 2019-04-22 20:13:05 |
Steve Langasek |
initramfs-tools (Ubuntu Bionic): status |
Incomplete |
In Progress |
|
| 2019-05-14 21:22:49 |
Brian Murray |
initramfs-tools (Ubuntu Cosmic): status |
In Progress |
Fix Committed |
|
| 2019-05-14 21:22:53 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-05-14 21:22:54 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2019-05-14 21:22:57 |
Brian Murray |
tags |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g verification-needed verification-needed-cosmic |
|
| 2019-05-14 21:25:45 |
Brian Murray |
initramfs-tools (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2019-05-14 21:25:52 |
Brian Murray |
tags |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g verification-needed verification-needed-cosmic |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g verification-needed verification-needed-bionic verification-needed-cosmic |
|
| 2019-05-14 21:40:17 |
Brian Murray |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2019-05-15 08:48:46 |
Frank Heimes |
ubuntu-power-systems: status |
In Progress |
Fix Committed |
|
| 2019-05-21 13:43:08 |
bugproxy |
tags |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g verification-needed verification-needed-bionic verification-needed-cosmic |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g verification-done verification-done-bionic verification-needed-cosmic |
|
| 2019-05-23 05:43:08 |
bugproxy |
tags |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g verification-done verification-done-bionic verification-needed-cosmic |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g verification-done-bionic verification-needed verification-needed-cosmic |
|
| 2019-05-28 16:55:11 |
Launchpad Janitor |
initramfs-tools (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2019-05-28 16:55:20 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2019-06-25 18:03:22 |
bugproxy |
tags |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g verification-done-bionic verification-needed verification-needed-cosmic |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g verification-done verification-done-bionic verification-done-cosmic |
|
| 2019-06-25 18:03:26 |
bugproxy |
attachment added |
|
kdump success with initramfs-tools from -proposed in Cosmic https://bugs.launchpad.net/bugs/1778844/+attachment/5273233/+files/cosmic-verification.log |
|
| 2019-06-27 10:18:52 |
Launchpad Janitor |
initramfs-tools (Ubuntu Cosmic): status |
Fix Committed |
Fix Released |
|
| 2019-06-27 10:35:42 |
Andrew Cloke |
ubuntu-power-systems: status |
Fix Committed |
Fix Released |
|
| 2020-03-12 06:54:10 |
bugproxy |
tags |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin--- triage-g verification-done verification-done-bionic verification-done-cosmic |
architecture-ppc64le bugnameltc-169067 kernel-da-key p9 severity-high targetmilestone-inin18041 triage-g verification-done verification-done-bionic verification-done-cosmic |
|
