nvme multipath does not report path relationships

Bug #1778844 reported by bugproxy on 2018-06-27
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Critical
Canonical Kernel Team
initramfs-tools (Ubuntu)
Critical
Canonical Kernel Team
Bionic
Critical
Thadeu Lima de Souza Cascardo
Cosmic
Critical
Thadeu Lima de Souza Cascardo
Disco
Critical
Canonical Kernel Team

Bug Description

[Impact]
initramfs created with MODULES=dep or kdump initrd won't boot a system with root filesystem on a multipath nvme.

[Test case]
Systems with nvme multipath were able to boot with the created initramfs. Also tested on systems with non-multipath nvme, and non nvme systems.

In order to verify the fix, one needs to change MODULES option to dep on /etc/initramfs-tools/initramfs.conf, recreate initramfs and reboot, check the system has booted fine. That should not break on systems with non nvme disks or systems with non multipath nvme systems, and that should now work on multipath nvme systems.

sed -i /MODULES=/s,=.*,=dep, /etc/initramfs-tools/initramfs.conf
update-initramfs -u -k all
reboot

[Regression potential]
A system could fail to boot because the generated initramfs was broken. The code should just add modules, which is safer than removing modules or doing any other changes. In any case, it was tested to boot on multipath nvme, non multipath nvme and non nvme systems.

-----------------

Problem Description:
===================
After triggering crash ,kdump is not working & system enters into initramfs state

Steps to re-create:
==================

>. woo is installed ubuntu180401 kernel

root@woo:~# uname -a
Linux woo 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 17:59:00 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
root@woo:~#

>. Crashkernel value as below

root@woo:~# free -h
              total used free shared buff/cache available
Mem: 503G 2.0G 501G 13M 279M 499G
Swap: 2.0G 0B 2.0G

root@woo:~# cat /proc/cmdline
root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet crashkernel=8192M

> kdump status

root@woo:~# kdump-config status
current state : ready to kdump

root@woo:~# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.15.0-23-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.15.0-23-generic
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet nr_cpus=1 systemd.unit=kdump-tools.service irqpoll noirqdistrib nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

root@woo:~# dmesg | grep Reser
[ 0.000000] Reserving 8192MB of memory at 128MB for crashkernel (System RAM: 524288MB)
[ 0.000000] cma: Reserved 26224 MiB at 0x0000203995000000
[ 3.545490] Copyright (C) 2017-2018 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries.

> Triggered crash

root@woo:~# echo 1 > /proc/sys/kernel/sysrq
root@woo:~# echo c > /proc/sysrq-trigger
[ 73.056308] sysrq: SysRq : Trigger a crash
[ 73.056357] Unable to handle kernel paging request for data at address 0x00000000
[ 73.056459] Faulting instruction address: 0xc0000000007f24c8
[ 73.056543] Oops: Kernel access of bad area, sig: 11 [#1]
[ 73.056609] LE SMP NR_CPUS=2048 NUMA PowerNV
[ 73.056668] Modules linked in: rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc idt_89hpesx vmx_crypto crct10dif_vpmsum ofpart cmdlinepart ipmi_powernv ipmi_devintf at24 powernv_flash ipmi_msghandler ibmpowernv mtd opal_prd uio_pdrv_genirq uio nfsd auth_rpcgss nfs_acl lockd sch_fq_codel grace sunrpc knem(OE) ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq mlx5_ib(OE) ib_core(OE) nouveau lpfc ast i2c_algo_bit ttm mlx5_core(OE) drm_kms_helper mlxfw(OE) nvmet_fc devlink syscopyarea nvmet mlx_compat(OE) sysfillrect cxl nvme_fc sysimgblt fb_sys_fops nvme_fabrics nvme ahci crc32c_vpmsum drm scsi_transport_fc
[ 73.057601] tg3 libahci nvme_core pnv_php
[ 73.057652] CPU: 44 PID: 4626 Comm: bash Tainted: G OE 4.15.0-23-generic #25-Ubuntu
[ 73.057767] NIP: c0000000007f24c8 LR: c0000000007f3568 CTR: c0000000007f24a0
[ 73.057868] REGS: c000003f8269f9f0 TRAP: 0300 Tainted: G OE (4.15.0-23-generic)
[ 73.057986] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28222222 XER: 20040000
[ 73.058099] CFAR: c0000000007f3564 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1
[ 73.058099] GPR00: c0000000007f3568 c000003f8269fc70 c0000000016eaf00 0000000000000063
[ 73.058099] GPR04: c000003fef47ce18 c000003fef494368 9000000000009033 0000000031da0058
[ 73.058099] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003
[ 73.058099] GPR12: c0000000007f24a0 c000000007a2e400 00000e4fa497c900 0000000000000000
[ 73.058099] GPR16: 00000e4f79cc94b0 00000e4f79d567e0 00000e4f79d88204 00000e4f79d56818
[ 73.058099] GPR20: 00000e4f79d8d5d8 0000000000000001 0000000000000000 00007ffffefce644
[ 73.058099] GPR24: 00007ffffefce640 00000e4f79d8afb4 c0000000015e9aa8 0000000000000002
[ 73.058099] GPR28: 0000000000000063 0000000000000004 c000000001572b1c c0000000015e9e68
[ 73.059060] NIP [c0000000007f24c8] sysrq_handle_crash+0x28/0x30
[ 73.059142] LR [c0000000007f3568] __handle_sysrq+0xf8/0x2c0
[ 73.059215] Call Trace:
[ 73.059254] [c000003f8269fc70] [c0000000007f3548] __handle_sysrq+0xd8/0x2c0 (unreliable)
[ 73.059358] [c000003f8269fd10] [c0000000007f3d74] write_sysrq_trigger+0x64/0x90
[ 73.059456] [c000003f8269fd40] [c000000000481248] proc_reg_write+0x88/0xd0
[ 73.059543] [c000003f8269fd70] [c0000000003d43fc] __vfs_write+0x3c/0x70
[ 73.059627] [c000003f8269fd90] [c0000000003d4658] vfs_write+0xd8/0x220
[ 73.059716] [c000003f8269fde0] [c0000000003d4978] SyS_write+0x68/0x110
[ 73.059809] [c000003f8269fe30] [c00000000000b284] system_call+0x58/0x6c
[ 73.059896] Instruction dump:
[ 73.059940] 4bfff9f1 4bfffe50 3c4c00f0 38428a60 7c0802a6 60000000 39200001 3d42001c
[ 73.060040] 394a6db0 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f0 38428a30
[ 73.060159] ---[ end trace e116d2421d2f59a5 ]---
[ 74.067059]
[ 74.067172] Sending IPI to other CPUs
[ 75.851509[ 202.275317797,5] OPAL: Switch to big-endian OS
] IPI complet[ 207.151277658,5] OPAL: Switch to little-endian OS
[ 232.159296542,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
e
[ 78.164472] kexec: Starting switchover sequence.
[ 1.412463] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.412468] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.481335] vio vio: uevent: failed to send synthetic uevent
[ 2.534732] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.534847] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.534967] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.535144] nouveau 0035:04:00.0: unknown chipset (140000a1)

Gave up waiting for root file system device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!

BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)
(initramfs)

== Comment: #1 - INDIRA P. JOGA <> - 2018-06-21 01:19:41 ==
Attached woo console logs for kdump issue

== Comment: #4 - INDIRA P. JOGA <> - 2018-06-26 01:51:47 ==
I have triggered crash & sits here

[ 1032.259696471,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
omplete
[ 823.882048] kexec: Starting switchover sequence.
[ 1.154056] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.154060] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.222719] vio vio: uevent: failed to send synthetic uevent
[ 2.212065] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.214995] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.215259] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.215408] nouveau 0035:04:00.0: unknown chipset (140000a1)
Gave up waiting for root file system device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!

BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

== Comment: #5 - INDIRA P. JOGA <> - 2018-06-26 02:40:56 ==
NOTE:

Used nvme disk as root disk here.

== Comment: #8 - Hari Krishna Bathini <> - 2018-06-26 06:06:13 ==
The dump target (/var/crash) is on NVMe device (also, the root disk).
But the kdump initrd is not being built with nvme driver modules.
Eventually, nvme disk is not found and kdump kernel is hitting the
initramfs shell. Using the default initrd, which has the nvme driver
modules included, dump was captured successfully.

Can someone from Canonical take a look at this and comment on why
nvme modules are not included in kdump initrd despite it being the
root disk..

Thanks
Hari

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-169067 severity-high targetmilestone-inin---

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → linux (Ubuntu)
Changed in ubuntu-power-systems:
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
tags: added: p9 triage-g
Changed in ubuntu-power-systems:
importance: Undecided → High
Changed in ubuntu-power-systems:
status: New → Triaged
tags: added: kernel-key
tags: added: kernel-da-key
removed: kernel-key

------- Comment From <email address hidden> 2018-07-09 06:18 EDT-------
Canonical, any update on this..?

Manoj Iyer (manjo) on 2018-07-09
Changed in linux (Ubuntu):
importance: Undecided → Critical
assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Canonical Kernel Team (canonical-kernel-team)
Changed in ubuntu-power-systems:
importance: High → Critical
tags: added: triage-a
removed: triage-g

Kernel team, looks like the root cause identified by IBM (taken from description of the bug) is:

== Comment: #8 - Hari Krishna Bathini <> - 2018-06-26 06:06:13 ==
The dump target (/var/crash) is on NVMe device (also, the root disk).
But the kdump initrd is not being built with nvme driver modules.
Eventually, nvme disk is not found and kdump kernel is hitting the
initramfs shell. Using the default initrd, which has the nvme driver
modules included, dump was captured successfully.

Can someone from Canonical take a look at this and comment on why
nvme modules are not included in kdump initrd despite it being the
root disk..

Thanks
Hari

I will look at this later today. Will send updates on it.

As we use MODULES=dep for kdump initrd, that might fail to include the proper modules. It would be a bug on initramfs-tools-core.

Can you change /etc/initramfs-tools/initramfs.conf to have MODULES=dep instead of MODULES=most and run mkinitramfs -o /tmp/initrd.img `uname -r`, then send me output?

Also, I need the data usually collected for kernel bugs. /sys/ at least comes to mind, but also complete dmesg after normal boot and some other important files to detect root mountpoint, like /etc/fstab, /proc/cmdline and /proc/mounts.

Thank you.
Cascardo.

I have managed to test kdump and initramfs with MODULES=dep on an emulated nvme, using qemu. Both worked just fine. The sysfs tree seems reasonable too, in respect to how initramfs finds the necessary modules.

So, I would really need the specific details on that failure case, that is, the sysfs tree and all the rest I requested in the previous message.

Thank you.
Cascardo.

Changed in ubuntu-power-systems:
status: Triaged → Incomplete

------- Comment on attachment From <email address hidden> 2018-07-23 07:12 EDT-------

Attached woo normal boot logs & kdump console logs with modules=dep

------- Comment on attachment From <email address hidden> 2018-07-23 07:13 EDT-------

Attached dmesg logs from initramfs state from woo

------- Comment From <email address hidden> 2018-07-23 07:30 EDT-------
Indira, I think Cascardo is looking for /tmp/initrd.img-`uname -r` file
whne MODULES=dep is used. Also, sosreport?

(In reply to comment #26)
> Created attachment 128855 [details]
> woo_kdump_nvmedisk_console logs

btw, looks like PHB initialization is failing. So, whether nvme driver
is present or not in the initrd is secondary here, the device initialization
itself could be at fault. What is the f/w level. Is it latest?

Thanks
Hari

------- Comment From <email address hidden> 2018-07-23 07:35 EDT-------
(In reply to comment #28)
> Indira, I think Cascardo is looking for /tmp/initrd.img-`uname -r` file
> whne MODULES=dep is used. Also, sosreport?
>
> (In reply to comment #26)
> > Created attachment 128855 [details]
> > woo_kdump_nvmedisk_console logs
>
> btw, looks like PHB initialization is failing. So, whether nvme driver
> is present or not in the initrd is secondary here, the device initialization
> itself could be at fault. What is the f/w level. Is it latest?

also, the initial console log is different from this.. Was the f/w on the system
updated as well..

Thanks
Hari

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-07-23 08:14 EDT-------
Also both logs were placed in

banner@9.3.117.11 (don2rry) : /home/banner

banner@banner:~> ls -l sosreport-woo-20180723065534.tar.xz
-rw------- 1 banner banner 7171548 Jul 23 07:05 sosreport-woo-20180723065534.tar.xz
banner@banner:~> ls -l initrd.img
-rw-r--r-- 1 banner banner 24119847 Jul 23 07:04 initrd.img
banner@banner:~>

------- Comment on attachment From <email address hidden> 2018-07-23 08:11 EDT-------

Attached /tmpinitrd.imng file

------- Comment on attachment From <email address hidden> 2018-07-23 08:12 EDT-------

Attached sosreport when MODULES=set used

Changed in ubuntu-power-systems:
status: Incomplete → Triaged
Manoj Iyer (manjo) on 2018-07-23
Changed in makedumpfile (Ubuntu):
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Changed in initramfs-tools (Ubuntu):
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
importance: Undecided → Critical
Changed in makedumpfile (Ubuntu):
importance: Undecided → Critical
tags: added: triage-g
removed: triage-a

------- Comment From <email address hidden> 2018-07-24 06:36 EDT-------
(In reply to comment #36)
> Can you run apport-collect -p linux 1778844 ?
>
> Thank you.
> Cascardo.

HI Cascardo,

I have tried to collect , but it fails as below

root@woo:~# apport-collect -p linux 1778844
ERROR: connecting to Launchpad failed: [Errno 110] Connection timed out
You can reset the credentials by removing the file "/root/.cache/apport/launchpad.credentials"
root@woo:~# export http_proxy="http://10.33.11.31:3128"
root@woo:~# apport-collect -p linux 1778844
ERROR: connecting to Launchpad failed: [Errno 110] Connection timed out
You can reset the credentials by removing the file "/root/.cache/apport/launchpad.credentials"
root@woo:~#

Regards,
Indira

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-07-25 01:28 EDT-------
(In reply to comment #34)
> Created attachment 128858 [details]
> sosreport_moduels=set
>
> Attached sosreport when MODULES=set used

Cascardo, is the sosreport no good?

Thanks
Hari

Hi, Hari.

The sosreport should be enough. I sent that message without an updated view of the bug, so did not notice the attachments. They came on the same day I asked. Just realized now that they are present. Let me look at them.

Now, if kdump works with MODULES=most, then the PHB initialization could require a module that is not in the initramfs. I know some Intel systems require their vmd module that adds a new PCI domain. Is there some new pci host module these days on these new Power systems?

Regards.
Cascardo.

Ah, I see what you meant about the PHB. The two new attached logs are not at all similar to the original ones, maybe pointing to an unrelated problem.

As for the sosreport, the /sys/ directory is missing almost everything that I would need to debug this. So, I need any other form of collecting /sys/. Maybe just tar czf lp1778844_sys.tar.gz /sys/.

Cascardo.

Manoj Iyer (manjo) on 2018-07-30
Changed in ubuntu-power-systems:
status: Triaged → Incomplete

------- Comment From <email address hidden> 2018-09-20 07:06 EDT-------
Hi,

We lost the old environment as jmet got returned.
Got a new Witherspoon jmet and retried the kdump scenario & issue recreated on latest ubuntu180401 kernel (4.15.0-34-generic) kernel.

> Triggered kdump with default crashkernel parameter and it went to initramfs state
root@wax:~# cat /proc/cmdline
root=UUID=38d0124f-14d8-4098-a746-03f9bda5c22e ro crashkernel=8192M splash quiet crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@128M

> Collected console & dmesg logs from same state
> Rebooted system & collected sosreport, tar czf lp1778844_sys.tar.gz /sys/
> Also tried comment#23 steps & collected /tmp/initrd.img

root@wax:~# cat /etc/initramfs-tools/initramfs.conf | grep MODULES
# MODULES: [ most | netboot | dep | list ]
MODULES=dep
root@wax:~# mkinitramfs -o /tmp/initrd.img
W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast
root@wax:~# uname -r
4.15.0-34-generic
root@wax:~# ls -l /tmp/initrd.img
-rw-r--r-- 1 root root 24128581 Sep 20 04:59 /tmp/initrd.img

> Attaching all above logs & if i am unable to attach any of the logs because of size issue. Logs are placed in the path

banner.isst.aus.stglabs.ibm.com [banner/don2rry ] : /home/banner/169067_sep20

> Attaching console logs from first step till last step ( starting from system booting with latest kernel level, trigger crash , initramfs state etc...)

Thanks & Regards,
Indira

------- Comment on attachment From <email address hidden> 2018-09-20 07:31 EDT-------

Attached IPMI console logs from start of the tests till end.

------- Comment on attachment From <email address hidden> 2018-09-20 07:34 EDT-------

Attached sosreport from wax_kdump_initramfs state

------- Comment on attachment From <email address hidden> 2018-09-20 07:38 EDT-------

Attached initrd.img file from wax

------- Comment From <email address hidden> 2018-09-20 07:42 EDT-------
Unable to attach "lp1778844_sys.tar.gz" file because of size, please refer to logs path .Also test machine is taken back for other release testing.

banner.isst.aus.stglabs.ibm.com [banner/don2rry ] : /home/banner/169067_sep20

Regards,
Indira

Changed in ubuntu-power-systems:
status: Incomplete → Triaged
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-25 04:34 EDT-------
Hello Canonical

Please download the files using Anonymous FTP and download the files from following path

ftp://testcase.software.ibm.com/fromibm/linux/lp1778844_sys.tar.gz

Note:
Files will remain in /fromibm/ for 3 business days before being deleted.

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-27 02:42 EDT-------
Cascardo/Canonical, Did you get a chance to look at the logs attached?

I managed to download the lp1778844_sys.tar.gz from FTP in time. It has some huge resource files, so I have to play a little with the tarball. It looks like some symlinks were tarred as empty files, so maybe I'll miss something, but I am investigating right now. I'll let you know if I need any other information. I will look at the other files as well.

So, this is a multipath disk, whose parent is the nvme-subsys0 device, not nvme0 device. The latter is a PCI device, but the former is a virtual one. Some code to add slaves/holders relationships was added, but reverted. I will investigate any further discussion about this upstream, or look into how to relate those nodes in a way we can just fix this in initramfs-tools.

summary: - ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump
- is not working and system enters into initramfs state
+ nvme multipath does not report path relationships

So, the hidden disk was introduced by hch as commit 8ddcd653257c18a669fcb75ee42c37054908e0d6 just to support nvme multipath. This thing prevented lsblk from working when there were slaves under the multipath nvme, as lsblk would postpone processing those, and lsblk could not process the paths themselves, because they have no dev_t.

Some discussion about reverting that slaves/holders patch is under the revert submission, and they decided to go on with the revert and think about how to fix that later. A link for one of the discussion messages is below.

https://www.spinics.net/lists/stable/msg222846.html

Going forward, we could either use nvme cli, embed the assumption that kernel won't change and some of the numbers are kept (like the subsys instance is embedded in both the multipath device and the underlying devices), or introduce this new paths subdir in the kernel. Any of those solutions would require changes to initramfs-tools. The latter would require a kernel change.

The only way of not changing initramfs-tools is going back to using slaves relationships. That means we would need to get rid of the hidden disks thing. This always worked for dm-multipath, but I believe this required some use of blkdev and that is one thing hch wanted to avoid, to have a very fast path for doing multipath.

I don't like the idea of using nvme cli that much, as it would be a hard dependency on initramfs-tools. I don't like assuming some of the implementation details on the kernel numbering/naming of those devices. So, I will take a stab at providing the paths interface.

Cascardo.

Changed in makedumpfile (Ubuntu Cosmic):
status: New → Invalid
Changed in makedumpfile (Ubuntu Bionic):
status: New → Invalid
Changed in linux (Ubuntu Bionic):
status: New → Confirmed
Changed in linux (Ubuntu Cosmic):
status: New → Confirmed
Changed in initramfs-tools (Ubuntu Cosmic):
status: New → Confirmed
Changed in initramfs-tools (Ubuntu Bionic):
status: New → Confirmed

As a workaround, I suggest setting nvme-core.ko multipath=0 parameter. Can you try that, rebooting, recreating the kdump-tools initrd, then crashing?

Cascardo.

Dimitri John Ledkov (xnox) wrote :

i downloaded that, and tried to poke it.
First of all, i've never seen a 77MB tarball of /sys alone.
Also unpacking that, is dangerous as it does explode in size. However, is debugging that, please only _list_ the files present in the tarball and only extract things you need/want.

Reuploaded to http://people.canonical.com/~xnox/lp1778844_sys.tar.gz for ease of access by canonical engineers.

------- Comment From <email address hidden> 2018-09-27 22:06 EDT-------
(In reply to comment #61)
> As a workaround, I suggest setting nvme-core.ko multipath=0 parameter. Can
> you try that, rebooting, recreating the kdump-tools initrd, then crashing?

Hi Cascardo,

Where/How to set this parameters? You mean, passing them as boot parameters?

Thanks
Hari

Just create a /etc/modprobe.d/nvme.conf file with the following contents, recreate initrd, remove kdump initrds so they are recreated in the next boot, then reboot.

# cat /etc/modprobe.d/nvme.conf
options nvme-core multipath=0
# update-initramfs -u
# rm /var/lib/kdump/initrd.img*
# reboot

First piece of the puzzle submitted:
http://lists.infradead.org/pipermail/linux-nvme/2018-September/020174.html

Tested with a small change on initramfs-tools, though I will wait for comments on the first one before submitting that to initramfs-tools upstream.

Manoj Iyer (manjo) on 2018-10-01
Changed in ubuntu-power-systems:
status: Triaged → In Progress

I have been working upstream to get this fixed on the nvme driver, but it might still take some time, I already have done a back and forth with hch. Will send a v3 later this month.

Meanwhile, there are multiple workarounds we documented so far in the bug, namely:

1) Disable nvme multipath, which is done by setting the multipath module
parameter for the nvme module on modprobe.d config files, regenerate
initrd, reboot, then regenerate kdump initrd.

2) Use MODULES=most instead of MODULES=dep on
/etc/kernel/postinst.d/kdump-tools and regenerate kdump initrd.

3) Force insert nvme modules into kdump initrd. Namely, just copy the
/boot/ initrd file to replace the one in /var/lib/kdump/.

Upstream discussion is progressing. I have tested some patches from hch, will send him some feedback later today and post the links here.

Upstream is a little reluctant to accept my changes, and seems to suggest that we find out the layout of the paths some other way (which is possible, but would be such a special case on initramfs-tools, only for nvme).

Cascardo.

On 01/15/2019 09:26 AM, Thadeu Lima de Souza Cascardo wrote:
> Upstream is a little reluctant to accept my changes, and seems to
> suggest that we find out the layout of the paths some other way (which
> is possible, but would be such a special case on initramfs-tools, only
> for nvme).
>
> Cascardo.
>
Hmm, I am really sorry about that - sux to do all that work and get that
kind of response.

So making the suggested changes seems like real work to me. Do you have
a rough estimate?

Terry

I have pushed a version of initramfs-tools to my ppa that seems to fix the problem. Can IBM test it?

add-apt-repository ppa:cascardo/kdump2

It's a version for bionic. Only initramfs-tools should come from this source. kdump-tools and makedumpfile should be from bionic-updates or bionic.

After that, remove the initrd images from /var/lib/kdump/, reboot, then test the crash.

So:

add-apt-repository ppa:cascardo/kdump2
apt install initramfs-tools
rm /var/lib/kdump/initr*
reboot

After reboot:

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

Changed in ubuntu-power-systems:
status: In Progress → Incomplete

------- Comment From <email address hidden> 2019-02-06 01:54 EDT-------
Have installed below patch and retried kdump scenario with root disk as nvme disk. Now system is not dropping to initramfs state its moving further but not saving dump file (core) properly.

initramfs-tools_0.130ubuntu3.6+nvme1_all.deb
initramfs-tools-bin_0.130ubuntu3.6+nvme1_ppc64el.deb
initramfs-tools-core_0.130ubuntu3.6+nvme1_all.deb

root@woo:~# dpkg -l | grep initramfs
ii busybox-initramfs 1:1.27.2-2ubuntu3 ppc64el Standalone shell setup for initramfs
ii initramfs-tools 0.130ubuntu3.6+nvme1 all generic modular initramfs generator (automation)
ii initramfs-tools-bin 0.130ubuntu3.6+nvme1 ppc64el binaries used by initramfs-tools
ii initramfs-tools-core 0.130ubuntu3.6+nvme1 all generic modular initramfs generator (core tools)
ii libklibc 2.0.4-9ubuntu2 ppc64el minimal libc subset for use with initramfs
root@woo:~#

rm /var/lib/kdump/initr*
reboot
After reboot triggered crash

> Attached console logs

Regards,
Indira

------- Comment on attachment From <email address hidden> 2019-02-06 01:56 EDT-------

Attached woo kdump scenario console logs

Changed in ubuntu-power-systems:
status: Incomplete → Confirmed

The system has too many devices, so too many drivers are using too much memory. So, the system goes out-of-memory before makedumpfile is complete. Please, reserve more memory on crashkernel, reboot the system and repeat the crash test.

To reserve more memory on crashkernel, edit file /etc/default/grub.d/kdump-tools.cfg and change crashkernel value to crashkernel=384M, for example, then run update-grub.

Thanks.
Cascardo.

------- Comment From <email address hidden> 2019-02-06 09:27 EDT-------
(In reply to comment #82)
> The system has too many devices, so too many drivers are using too much
> memory. So, the system goes out-of-memory before makedumpfile is complete.
> Please, reserve more memory on crashkernel, reboot the system and repeat the
> crash test.
>
> To reserve more memory on crashkernel, edit file
> /etc/default/grub.d/kdump-tools.cfg and change crashkernel value to
> crashkernel=384M, for example, then run update-grub.
>
> Thanks.
> Cascardo.

Hi Cascardo

As per SUSE document recommendation used 4096M crash kernel and with this value i have seen OOM traces during kdump.

root@woo:~# free -h
total used free shared buff/cache available
Mem: 251G 7.2G 243G 16M 567M 243G
Swap: 2.0G 0B 2.0G
root@woo:~# cat /proc/cmdline
root=UUID=df4c0331-9e6c-4592-9e0c-84f431ecb1f7 ro splash quiet crashkernel=4096M

Will try to increase to some more above 4096M and retry it.

Regards,
Indira

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2019-02-06 09:45 EDT-------
Triggered crash with patch using crashkernel=8192M and did not see OOM traces and dump copied properly as below

root@woo:/var/crash/201902060928# pwd
/var/crash/201902060928
root@woo:/var/crash/201902060928# ls -l
total 469212
-rw------- 1 root root 114308 Feb 6 09:29 dmesg.201902060928
-rw------- 1 root root 480474149 Feb 6 09:29 dump.201902060928
root@woo:/var/crash/201902060928# date
Wed Feb 6 09:44:42 EST 2019
root@woo:/var/crash/201902060928#

Regards,
Indira

Changed in ubuntu-power-systems:
status: Confirmed → In Progress
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2019-02-12 01:11 EDT-------
(In reply to comment #83)
[...]
> Will try to increase to some more above 4096M and retry it.

The recommendations are based on a few assumptions that don't apply to all machines.
So, hitting OOM in this case is not surprising in that sense. As increasing crashkernel
value helped, we can consider the original issue reported here resolved..

Andrew Cloke (andrew-cloke) wrote :

Thanks. Closing out the bug.

Changed in ubuntu-power-systems:
status: In Progress → Won't Fix
Changed in initramfs-tools (Ubuntu):
status: Confirmed → Invalid
Changed in initramfs-tools (Ubuntu Bionic):
status: Confirmed → Invalid
Changed in initramfs-tools (Ubuntu Cosmic):
status: Confirmed → Invalid
Changed in linux (Ubuntu):
status: Confirmed → Invalid
Changed in linux (Ubuntu Bionic):
status: Confirmed → Invalid
Changed in linux (Ubuntu Cosmic):
status: Confirmed → Invalid

Hi, Andrew and Hari.

The original issue is only resolved by using an initramfs-tools from my ppa. We still need to get this fix included in our releases. I will start working to get this in disco.

Cascardo.

Changed in initramfs-tools (Ubuntu):
status: Invalid → In Progress
Changed in initramfs-tools (Ubuntu Cosmic):
status: Invalid → In Progress
Changed in initramfs-tools (Ubuntu Bionic):
status: Invalid → In Progress
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2019-02-12 06:15 EDT-------
(In reply to comment #87)
> Hi, Andrew and Hari.
>
> The original issue is only resolved by using an initramfs-tools from my ppa.
> We still need to get this fix included in our releases. I will start working
> to get this in disco.
>
> Cascardo.

Right, Cascardo.
Waiting for the fix to be released in an official build..

That's an OOM problem. Just increase the reserved crashkernel memory size and test again, please.

description: updated
Changed in ubuntu-power-systems:
status: Won't Fix → In Progress
no longer affects: makedumpfile (Ubuntu)
no longer affects: makedumpfile (Ubuntu Bionic)
no longer affects: makedumpfile (Ubuntu Cosmic)
no longer affects: makedumpfile (Ubuntu Disco)
Changed in initramfs-tools (Ubuntu Bionic):
importance: Undecided → Critical
Andrew Cloke (andrew-cloke) wrote :

Next step: Thadeu to propose the diff described in comment #53 into the archive.
(If this incorrect, please flag.)

Hi, Andrew, with the debdiff attached and ubuntu sponsors team subscribed, now we should wait for someone on that team to sponsor that update. With the SRU template attached, the only thing blocking that sponsorship now is that disco is now on freeze, so the sponsor is the one who would decide whether this warrants an upload during the freeze.

Andrew Cloke (andrew-cloke) wrote :

Hi Thadeu, thanks for the update.

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2019-04-12 05:41 EDT-------
Hi ,

Any update on this bug, when this fix is expected in official build.

Please update the approximate date, as issue exists with latest Ubuntu180401 kernel - 4.15.0-47-generic.

Regards,
Indira

Steve Langasek (vorlon) wrote :

initramfs-tools uploaded for disco. Thadeu, do you want to prepare a separate debdiff for bionic and cosmic?

Changed in initramfs-tools (Ubuntu Disco):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package initramfs-tools - 0.131ubuntu19

---------------
initramfs-tools (0.131ubuntu19) disco; urgency=medium

  * Add modules for nvme path components on multipath nvme. LP: #1778844

 -- Thadeu Lima de Souza Cascardo <email address hidden> Tue, 26 Mar 2019 14:49:29 -0300

Changed in initramfs-tools (Ubuntu Disco):
status: Fix Committed → Fix Released
no longer affects: linux (Ubuntu)
no longer affects: linux (Ubuntu Bionic)
no longer affects: linux (Ubuntu Cosmic)
no longer affects: linux (Ubuntu Disco)
Changed in initramfs-tools (Ubuntu Cosmic):
assignee: Canonical Kernel Team (canonical-kernel-team) → Thadeu Lima de Souza Cascardo (cascardo)
Changed in initramfs-tools (Ubuntu Bionic):
assignee: nobody → Thadeu Lima de Souza Cascardo (cascardo)
Steve Langasek (vorlon) on 2019-04-20
Changed in initramfs-tools (Ubuntu Bionic):
assignee: Thadeu Lima de Souza Cascardo (cascardo) → Steve Langasek (vorlon)
Steve Langasek (vorlon) wrote :

Thadeu, the test case for the SRU needs to describe exactly the steps that will be followed to verify the correctness of this bugfix, so that anyone (with the necessary hardware) can follow them.

Changed in initramfs-tools (Ubuntu Bionic):
assignee: Steve Langasek (vorlon) → Thadeu Lima de Souza Cascardo (cascardo)
status: In Progress → Incomplete

@vorlon, test case updated.

Thanks.
Cascardo.

description: updated
Steve Langasek (vorlon) on 2019-04-22
Changed in initramfs-tools (Ubuntu Bionic):
status: Incomplete → In Progress
Andrew Cloke (andrew-cloke) wrote :

Added to the SRU queue on 22nd April. Thanks!

Hello bugproxy, or anyone else affected,

Accepted initramfs-tools into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.131ubuntu15.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in initramfs-tools (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Changed in initramfs-tools (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Brian Murray (brian-murray) wrote :

Hello bugproxy, or anyone else affected,

Accepted initramfs-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.130ubuntu3.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ubuntu-power-systems:
status: In Progress → Fix Committed

------- Comment From <email address hidden> 2019-05-21 08:26 EDT-------
Installed latest initramfs-tools & kdump-tools packages from -proposed kernel as below

root@woo:~# dpkg -l initramfs*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================================-=====================-=====================-=======================================================================
ii initramfs-tools 0.130ubuntu3.8 all generic modular initramfs generator (automation)
ii initramfs-tools-bin 0.130ubuntu3.8 ppc64el binaries used by initramfs-tools
ii initramfs-tools-core 0.130ubuntu3.8 all generic modular initramfs generator (core tools)

root@woo:~# dpkg -l kdump-tools*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================================-=====================-=====================-=======================================================================
ii kdump-tools 1:1.6.5-1ubuntu1~18.0 ppc64el scripts and tools for automating kdump (Linux crash dumps)

Triggered crash with nvme disk as root disk ,crash triggered properly and dump file saved in /var/crash path

root@woo:~# uname -a
Linux woo 4.15.0-51-generic #55-Ubuntu SMP Wed May 15 14:26:12 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

crash logs:
***********
root@woo:~# cd /var/crash
root@woo:/var/crash# ls -l
total 580K
drwxrwxrwt 4 root root 4.0K May 21 07:51 ./
drwxr-xr-x 12 root root 4.0K May 18 12:23 ../
drwxr-xr-x 2 root root 4.0K May 20 13:12 201905201311/
drwxr-xr-x 2 root root 4.0K May 21 07:41 201905210740/
-rw-r--r-- 1 root root 228 May 21 07:51 kexec_cmd
-rw-r----- 1 root root 31K May 20 13:16 linux-image-4.15.0-51-generic-201905201311.crash
-rw-r----- 1 root root 177K May 21 07:51 linux-image-4.15.0-51-generic-201905210740.crash
-rw------- 1 root root 348K May 18 13:11 ssl-cert.0.crash

root@woo:/var/crash/201905210740# ls
total 538M
drwxr-xr-x 2 root root 4.0K May 21 07:41 ./
drwxrwxrwt 4 root root 4.0K May 21 07:51 ../
-rw------- 1 root root 994K May 21 07:41 dmesg.201905210740
-rw------- 1 root root 537M May 21 07:41 dump.201905210740
root@woo:/var/crash/201905210740# pwd
/var/crash/201905210740
root@woo:/var/crash/201905210740#

Issue is fixed with latest initramfs-tools package.

Regards,
Indira

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic

Thank you for taking the time to verify this stable release fix. We have noticed that you have used the verification-done tag for marking the bug as verified and would like to point out that due to a recent change in SRU bug verification policy fixes now have to be marked with per-release tags (i.e. verification-done-$RELEASE). Please remove the verification-done tag and add one for the release you have tested the package in. Thank you!

https://wiki.ubuntu.com/StableReleaseUpdates#Verification

Andrew Cloke (andrew-cloke) wrote :

Thanks for the verification in comment #67. From the comment, I believe this was verified with bionic. Could you also verify with Cosmic and post your results?
Thanks.

bugproxy (bugproxy) on 2019-05-23
tags: added: verification-needed
removed: verification-done
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package initramfs-tools - 0.130ubuntu3.8

---------------
initramfs-tools (0.130ubuntu3.8) bionic; urgency=medium

  * Add modules for nvme path components on multipath nvme. LP: #1778844

 -- Thadeu Lima de Souza Cascardo <email address hidden> Tue, 16 Apr 2019 21:25:56 -0300

Changed in initramfs-tools (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for initramfs-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Manoj Iyer (manjo) wrote :
Download full text (45.8 KiB)

I installed cosmic on power9 (boston) a non-nvme system, installed initramfs-tools from -proposed and install kdump-tools. Triggered a dump, but I got a kernel panic. Please let me know if I am doing something wrong here.

ubuntu@tiselius:~$ uname -a
Linux tiselius 4.18.0-21-generic #22-Ubuntu SMP Wed May 15 13:12:45 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
ubuntu@tiselius:~$

ubuntu@tiselius:~$ apt policy initramfs-tools
initramfs-tools:
  Installed: 0.131ubuntu15.2
  Candidate: 0.131ubuntu15.2
  Version table:
 *** 0.131ubuntu15.2 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic-proposed/main ppc64el Packages
ubuntu@tiselius:~$

ubuntu@tiselius:~$ apt policy kdump-tools
kdump-tools:
  Installed: 1:1.6.5-1ubuntu1~18.10.1
  Candidate: 1:1.6.5-1ubuntu1~18.10.1
  Version table:
 *** 1:1.6.5-1ubuntu1~18.10.1 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic-updates/main ppc64el Packages
        100 /var/lib/dpkg/status
     1:1.6.4-2ubuntu1 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic/main ppc64el Packages
ubuntu@tiselius:~$

ubuntu@tiselius:~$ sudo kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.18.0-21-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.18.0-21-generic
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="root=UUID=295f571b-b731-4ebb-b752-60aadc80fc1b ro console=hvc0 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll noirqdistrib nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
ubuntu@tiselius:~$

ubuntu@tiselius:~$ cat /proc/cmdline
root=UUID=295f571b-b731-4ebb-b752-60aadc80fc1b ro console=hvc0 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@128M
ubuntu@tiselius:~$

ubuntu@tiselius:~$ dmesg | grep Reser
[ 0.000000] Reserving 4096MB of memory at 128MB for crashkernel (System RAM: 131072MB)
[ 0.000000] cma: Reserved 6560 MiB at 0x0000200e62000000
ubuntu@tiselius:~$

root@tiselius:/home/ubuntu# echo 1 > /proc/sys/kernel/sysrq
root@tiselius:/home/ubuntu# echo c > /proc/sysrq-trigger

tiselius login: [ 150.167288] cloud-init[4529]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.10.1 running 'modules:final' at Mon, 10 Jun 2019 19:53:40 +0000. Up 149.80 seconds.
[ 150.167687] cloud-init[4529]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.10.1 finished at Mon, 10 Jun 2019 19:53:40 +0000. Datasource DataSourceMAAS [http://10-245-64-0--21.maas-internal:5248/MAAS/metadata/]. Up 150.11 seconds
[ 360.313029] kdump-tools[4915]: Stopping kdump-tools: * unloaded kdump kernel
[ 376.552452] kdump-tools[10449]: Starting kdump-tools: * Creating symlink /var/lib/kdump/vmlinuz
[ 376.553743] kdump-tools[10449]: * Creating symlink /var/lib/kdump/initrd.img
[ 376.585085] kdump-tools[10449]: Modified cmdline:root=UUID=295f571b-b731-4ebb-b752-60aadc80fc1b ro console=hvc0 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll noirqdistrib nousb elfcorehdr=158784K
[ 376.953223] kdump-tools[10449]: * loaded kdump kerne...

Hi, Manoj.

There is nothing wrong in what you are doing here. This seems to be an unrelated bug. I am looking at any known problems on cpuidle for power9. Can you, please, open a different bug report, and try a different system for regression testing? From the log message, it doesn't seem there is anything related to initramfs changes, but a real kernel bug, also, apparently, unrelated to kdump.

Thanks.
Cascardo.

Manoj Iyer (manjo) wrote :

@Thadeu, here is a new bug report that tracks this issue: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1832388 We dont have a system in house with NVme, we will need IBM to verify -proposed with cosmic and report back here.

To post a comment you must log in.