nvme multipath does not report path relationships

Bug #1778844 reported by bugproxy on 2018-06-27
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Critical
Canonical Kernel Team
initramfs-tools (Ubuntu)
Critical
Canonical Kernel Team
Bionic
Undecided
Unassigned
Cosmic
Critical
Canonical Kernel Team
linux (Ubuntu)
Critical
Canonical Kernel Team
Bionic
Undecided
Unassigned
Cosmic
Critical
Canonical Kernel Team
makedumpfile (Ubuntu)
Critical
Canonical Kernel Team
Bionic
Undecided
Unassigned
Cosmic
Critical
Canonical Kernel Team

Bug Description

Problem Description:
===================
After triggering crash ,kdump is not working & system enters into initramfs state

Steps to re-create:
==================

>. woo is installed ubuntu180401 kernel

root@woo:~# uname -a
Linux woo 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 17:59:00 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
root@woo:~#

>. Crashkernel value as below

root@woo:~# free -h
              total used free shared buff/cache available
Mem: 503G 2.0G 501G 13M 279M 499G
Swap: 2.0G 0B 2.0G

root@woo:~# cat /proc/cmdline
root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet crashkernel=8192M

> kdump status

root@woo:~# kdump-config status
current state : ready to kdump

root@woo:~# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.15.0-23-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.15.0-23-generic
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet nr_cpus=1 systemd.unit=kdump-tools.service irqpoll noirqdistrib nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

root@woo:~# dmesg | grep Reser
[ 0.000000] Reserving 8192MB of memory at 128MB for crashkernel (System RAM: 524288MB)
[ 0.000000] cma: Reserved 26224 MiB at 0x0000203995000000
[ 3.545490] Copyright (C) 2017-2018 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries.

> Triggered crash

root@woo:~# echo 1 > /proc/sys/kernel/sysrq
root@woo:~# echo c > /proc/sysrq-trigger
[ 73.056308] sysrq: SysRq : Trigger a crash
[ 73.056357] Unable to handle kernel paging request for data at address 0x00000000
[ 73.056459] Faulting instruction address: 0xc0000000007f24c8
[ 73.056543] Oops: Kernel access of bad area, sig: 11 [#1]
[ 73.056609] LE SMP NR_CPUS=2048 NUMA PowerNV
[ 73.056668] Modules linked in: rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc idt_89hpesx vmx_crypto crct10dif_vpmsum ofpart cmdlinepart ipmi_powernv ipmi_devintf at24 powernv_flash ipmi_msghandler ibmpowernv mtd opal_prd uio_pdrv_genirq uio nfsd auth_rpcgss nfs_acl lockd sch_fq_codel grace sunrpc knem(OE) ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq mlx5_ib(OE) ib_core(OE) nouveau lpfc ast i2c_algo_bit ttm mlx5_core(OE) drm_kms_helper mlxfw(OE) nvmet_fc devlink syscopyarea nvmet mlx_compat(OE) sysfillrect cxl nvme_fc sysimgblt fb_sys_fops nvme_fabrics nvme ahci crc32c_vpmsum drm scsi_transport_fc
[ 73.057601] tg3 libahci nvme_core pnv_php
[ 73.057652] CPU: 44 PID: 4626 Comm: bash Tainted: G OE 4.15.0-23-generic #25-Ubuntu
[ 73.057767] NIP: c0000000007f24c8 LR: c0000000007f3568 CTR: c0000000007f24a0
[ 73.057868] REGS: c000003f8269f9f0 TRAP: 0300 Tainted: G OE (4.15.0-23-generic)
[ 73.057986] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28222222 XER: 20040000
[ 73.058099] CFAR: c0000000007f3564 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1
[ 73.058099] GPR00: c0000000007f3568 c000003f8269fc70 c0000000016eaf00 0000000000000063
[ 73.058099] GPR04: c000003fef47ce18 c000003fef494368 9000000000009033 0000000031da0058
[ 73.058099] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003
[ 73.058099] GPR12: c0000000007f24a0 c000000007a2e400 00000e4fa497c900 0000000000000000
[ 73.058099] GPR16: 00000e4f79cc94b0 00000e4f79d567e0 00000e4f79d88204 00000e4f79d56818
[ 73.058099] GPR20: 00000e4f79d8d5d8 0000000000000001 0000000000000000 00007ffffefce644
[ 73.058099] GPR24: 00007ffffefce640 00000e4f79d8afb4 c0000000015e9aa8 0000000000000002
[ 73.058099] GPR28: 0000000000000063 0000000000000004 c000000001572b1c c0000000015e9e68
[ 73.059060] NIP [c0000000007f24c8] sysrq_handle_crash+0x28/0x30
[ 73.059142] LR [c0000000007f3568] __handle_sysrq+0xf8/0x2c0
[ 73.059215] Call Trace:
[ 73.059254] [c000003f8269fc70] [c0000000007f3548] __handle_sysrq+0xd8/0x2c0 (unreliable)
[ 73.059358] [c000003f8269fd10] [c0000000007f3d74] write_sysrq_trigger+0x64/0x90
[ 73.059456] [c000003f8269fd40] [c000000000481248] proc_reg_write+0x88/0xd0
[ 73.059543] [c000003f8269fd70] [c0000000003d43fc] __vfs_write+0x3c/0x70
[ 73.059627] [c000003f8269fd90] [c0000000003d4658] vfs_write+0xd8/0x220
[ 73.059716] [c000003f8269fde0] [c0000000003d4978] SyS_write+0x68/0x110
[ 73.059809] [c000003f8269fe30] [c00000000000b284] system_call+0x58/0x6c
[ 73.059896] Instruction dump:
[ 73.059940] 4bfff9f1 4bfffe50 3c4c00f0 38428a60 7c0802a6 60000000 39200001 3d42001c
[ 73.060040] 394a6db0 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f0 38428a30
[ 73.060159] ---[ end trace e116d2421d2f59a5 ]---
[ 74.067059]
[ 74.067172] Sending IPI to other CPUs
[ 75.851509[ 202.275317797,5] OPAL: Switch to big-endian OS
] IPI complet[ 207.151277658,5] OPAL: Switch to little-endian OS
[ 232.159296542,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
e
[ 78.164472] kexec: Starting switchover sequence.
[ 1.412463] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.412468] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.481335] vio vio: uevent: failed to send synthetic uevent
[ 2.534732] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.534847] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.534967] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.535144] nouveau 0035:04:00.0: unknown chipset (140000a1)

Gave up waiting for root file system device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!

BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)
(initramfs)

== Comment: #1 - INDIRA P. JOGA <> - 2018-06-21 01:19:41 ==
Attached woo console logs for kdump issue

== Comment: #4 - INDIRA P. JOGA <> - 2018-06-26 01:51:47 ==
I have triggered crash & sits here

[ 1032.259696471,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
omplete
[ 823.882048] kexec: Starting switchover sequence.
[ 1.154056] integrity: Unable to open file: /etc/keys/x509_ima.der (-2)
[ 1.154060] integrity: Unable to open file: /etc/keys/x509_evm.der (-2)
[ 1.222719] vio vio: uevent: failed to send synthetic uevent
[ 2.212065] nouveau 0004:04:00.0: unknown chipset (140000a1)
[ 2.214995] nouveau 0004:05:00.0: unknown chipset (140000a1)
[ 2.215259] nouveau 0035:03:00.0: unknown chipset (140000a1)
[ 2.215408] nouveau 0035:04:00.0: unknown chipset (140000a1)
Gave up waiting for root file system device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell!

BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

== Comment: #5 - INDIRA P. JOGA <> - 2018-06-26 02:40:56 ==
NOTE:

Used nvme disk as root disk here.

== Comment: #8 - Hari Krishna Bathini <> - 2018-06-26 06:06:13 ==
The dump target (/var/crash) is on NVMe device (also, the root disk).
But the kdump initrd is not being built with nvme driver modules.
Eventually, nvme disk is not found and kdump kernel is hitting the
initramfs shell. Using the default initrd, which has the nvme driver
modules included, dump was captured successfully.

Can someone from Canonical take a look at this and comment on why
nvme modules are not included in kdump initrd despite it being the
root disk..

Thanks
Hari

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-169067 severity-high targetmilestone-inin---

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → linux (Ubuntu)
Changed in ubuntu-power-systems:
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
tags: added: p9 triage-g
Changed in ubuntu-power-systems:
importance: Undecided → High
Changed in ubuntu-power-systems:
status: New → Triaged
tags: added: kernel-key
tags: added: kernel-da-key
removed: kernel-key

------- Comment From <email address hidden> 2018-07-09 06:18 EDT-------
Canonical, any update on this..?

Manoj Iyer (manjo) on 2018-07-09
Changed in linux (Ubuntu):
importance: Undecided → Critical
assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Canonical Kernel Team (canonical-kernel-team)
Changed in ubuntu-power-systems:
importance: High → Critical
tags: added: triage-a
removed: triage-g

Kernel team, looks like the root cause identified by IBM (taken from description of the bug) is:

== Comment: #8 - Hari Krishna Bathini <> - 2018-06-26 06:06:13 ==
The dump target (/var/crash) is on NVMe device (also, the root disk).
But the kdump initrd is not being built with nvme driver modules.
Eventually, nvme disk is not found and kdump kernel is hitting the
initramfs shell. Using the default initrd, which has the nvme driver
modules included, dump was captured successfully.

Can someone from Canonical take a look at this and comment on why
nvme modules are not included in kdump initrd despite it being the
root disk..

Thanks
Hari

I will look at this later today. Will send updates on it.

As we use MODULES=dep for kdump initrd, that might fail to include the proper modules. It would be a bug on initramfs-tools-core.

Can you change /etc/initramfs-tools/initramfs.conf to have MODULES=dep instead of MODULES=most and run mkinitramfs -o /tmp/initrd.img `uname -r`, then send me output?

Also, I need the data usually collected for kernel bugs. /sys/ at least comes to mind, but also complete dmesg after normal boot and some other important files to detect root mountpoint, like /etc/fstab, /proc/cmdline and /proc/mounts.

Thank you.
Cascardo.

I have managed to test kdump and initramfs with MODULES=dep on an emulated nvme, using qemu. Both worked just fine. The sysfs tree seems reasonable too, in respect to how initramfs finds the necessary modules.

So, I would really need the specific details on that failure case, that is, the sysfs tree and all the rest I requested in the previous message.

Thank you.
Cascardo.

Changed in ubuntu-power-systems:
status: Triaged → Incomplete

------- Comment on attachment From <email address hidden> 2018-07-23 07:12 EDT-------

Attached woo normal boot logs & kdump console logs with modules=dep

------- Comment on attachment From <email address hidden> 2018-07-23 07:13 EDT-------

Attached dmesg logs from initramfs state from woo

------- Comment From <email address hidden> 2018-07-23 07:30 EDT-------
Indira, I think Cascardo is looking for /tmp/initrd.img-`uname -r` file
whne MODULES=dep is used. Also, sosreport?

(In reply to comment #26)
> Created attachment 128855 [details]
> woo_kdump_nvmedisk_console logs

btw, looks like PHB initialization is failing. So, whether nvme driver
is present or not in the initrd is secondary here, the device initialization
itself could be at fault. What is the f/w level. Is it latest?

Thanks
Hari

------- Comment From <email address hidden> 2018-07-23 07:35 EDT-------
(In reply to comment #28)
> Indira, I think Cascardo is looking for /tmp/initrd.img-`uname -r` file
> whne MODULES=dep is used. Also, sosreport?
>
> (In reply to comment #26)
> > Created attachment 128855 [details]
> > woo_kdump_nvmedisk_console logs
>
> btw, looks like PHB initialization is failing. So, whether nvme driver
> is present or not in the initrd is secondary here, the device initialization
> itself could be at fault. What is the f/w level. Is it latest?

also, the initial console log is different from this.. Was the f/w on the system
updated as well..

Thanks
Hari

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-07-23 08:14 EDT-------
Also both logs were placed in

banner@9.3.117.11 (don2rry) : /home/banner

banner@banner:~> ls -l sosreport-woo-20180723065534.tar.xz
-rw------- 1 banner banner 7171548 Jul 23 07:05 sosreport-woo-20180723065534.tar.xz
banner@banner:~> ls -l initrd.img
-rw-r--r-- 1 banner banner 24119847 Jul 23 07:04 initrd.img
banner@banner:~>

------- Comment on attachment From <email address hidden> 2018-07-23 08:11 EDT-------

Attached /tmpinitrd.imng file

------- Comment on attachment From <email address hidden> 2018-07-23 08:12 EDT-------

Attached sosreport when MODULES=set used

Changed in ubuntu-power-systems:
status: Incomplete → Triaged
Manoj Iyer (manjo) on 2018-07-23
Changed in makedumpfile (Ubuntu):
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Changed in initramfs-tools (Ubuntu):
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
importance: Undecided → Critical
Changed in makedumpfile (Ubuntu):
importance: Undecided → Critical
tags: added: triage-g
removed: triage-a

------- Comment From <email address hidden> 2018-07-24 06:36 EDT-------
(In reply to comment #36)
> Can you run apport-collect -p linux 1778844 ?
>
> Thank you.
> Cascardo.

HI Cascardo,

I have tried to collect , but it fails as below

root@woo:~# apport-collect -p linux 1778844
ERROR: connecting to Launchpad failed: [Errno 110] Connection timed out
You can reset the credentials by removing the file "/root/.cache/apport/launchpad.credentials"
root@woo:~# export http_proxy="http://10.33.11.31:3128"
root@woo:~# apport-collect -p linux 1778844
ERROR: connecting to Launchpad failed: [Errno 110] Connection timed out
You can reset the credentials by removing the file "/root/.cache/apport/launchpad.credentials"
root@woo:~#

Regards,
Indira

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-07-25 01:28 EDT-------
(In reply to comment #34)
> Created attachment 128858 [details]
> sosreport_moduels=set
>
> Attached sosreport when MODULES=set used

Cascardo, is the sosreport no good?

Thanks
Hari

Hi, Hari.

The sosreport should be enough. I sent that message without an updated view of the bug, so did not notice the attachments. They came on the same day I asked. Just realized now that they are present. Let me look at them.

Now, if kdump works with MODULES=most, then the PHB initialization could require a module that is not in the initramfs. I know some Intel systems require their vmd module that adds a new PCI domain. Is there some new pci host module these days on these new Power systems?

Regards.
Cascardo.

Ah, I see what you meant about the PHB. The two new attached logs are not at all similar to the original ones, maybe pointing to an unrelated problem.

As for the sosreport, the /sys/ directory is missing almost everything that I would need to debug this. So, I need any other form of collecting /sys/. Maybe just tar czf lp1778844_sys.tar.gz /sys/.

Cascardo.

Manoj Iyer (manjo) on 2018-07-30
Changed in ubuntu-power-systems:
status: Triaged → Incomplete

------- Comment From <email address hidden> 2018-09-20 07:06 EDT-------
Hi,

We lost the old environment as jmet got returned.
Got a new Witherspoon jmet and retried the kdump scenario & issue recreated on latest ubuntu180401 kernel (4.15.0-34-generic) kernel.

> Triggered kdump with default crashkernel parameter and it went to initramfs state
root@wax:~# cat /proc/cmdline
root=UUID=38d0124f-14d8-4098-a746-03f9bda5c22e ro crashkernel=8192M splash quiet crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@128M

> Collected console & dmesg logs from same state
> Rebooted system & collected sosreport, tar czf lp1778844_sys.tar.gz /sys/
> Also tried comment#23 steps & collected /tmp/initrd.img

root@wax:~# cat /etc/initramfs-tools/initramfs.conf | grep MODULES
# MODULES: [ most | netboot | dep | list ]
MODULES=dep
root@wax:~# mkinitramfs -o /tmp/initrd.img
W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast
root@wax:~# uname -r
4.15.0-34-generic
root@wax:~# ls -l /tmp/initrd.img
-rw-r--r-- 1 root root 24128581 Sep 20 04:59 /tmp/initrd.img

> Attaching all above logs & if i am unable to attach any of the logs because of size issue. Logs are placed in the path

banner.isst.aus.stglabs.ibm.com [banner/don2rry ] : /home/banner/169067_sep20

> Attaching console logs from first step till last step ( starting from system booting with latest kernel level, trigger crash , initramfs state etc...)

Thanks & Regards,
Indira

------- Comment on attachment From <email address hidden> 2018-09-20 07:31 EDT-------

Attached IPMI console logs from start of the tests till end.

------- Comment on attachment From <email address hidden> 2018-09-20 07:34 EDT-------

Attached sosreport from wax_kdump_initramfs state

------- Comment on attachment From <email address hidden> 2018-09-20 07:38 EDT-------

Attached initrd.img file from wax

------- Comment From <email address hidden> 2018-09-20 07:42 EDT-------
Unable to attach "lp1778844_sys.tar.gz" file because of size, please refer to logs path .Also test machine is taken back for other release testing.

banner.isst.aus.stglabs.ibm.com [banner/don2rry ] : /home/banner/169067_sep20

Regards,
Indira

Changed in ubuntu-power-systems:
status: Incomplete → Triaged
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-25 04:34 EDT-------
Hello Canonical

Please download the files using Anonymous FTP and download the files from following path

ftp://testcase.software.ibm.com/fromibm/linux/lp1778844_sys.tar.gz

Note:
Files will remain in /fromibm/ for 3 business days before being deleted.

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-27 02:42 EDT-------
Cascardo/Canonical, Did you get a chance to look at the logs attached?

I managed to download the lp1778844_sys.tar.gz from FTP in time. It has some huge resource files, so I have to play a little with the tarball. It looks like some symlinks were tarred as empty files, so maybe I'll miss something, but I am investigating right now. I'll let you know if I need any other information. I will look at the other files as well.

So, this is a multipath disk, whose parent is the nvme-subsys0 device, not nvme0 device. The latter is a PCI device, but the former is a virtual one. Some code to add slaves/holders relationships was added, but reverted. I will investigate any further discussion about this upstream, or look into how to relate those nodes in a way we can just fix this in initramfs-tools.

summary: - ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump
- is not working and system enters into initramfs state
+ nvme multipath does not report path relationships

So, the hidden disk was introduced by hch as commit 8ddcd653257c18a669fcb75ee42c37054908e0d6 just to support nvme multipath. This thing prevented lsblk from working when there were slaves under the multipath nvme, as lsblk would postpone processing those, and lsblk could not process the paths themselves, because they have no dev_t.

Some discussion about reverting that slaves/holders patch is under the revert submission, and they decided to go on with the revert and think about how to fix that later. A link for one of the discussion messages is below.

https://www.spinics.net/lists/stable/msg222846.html

Going forward, we could either use nvme cli, embed the assumption that kernel won't change and some of the numbers are kept (like the subsys instance is embedded in both the multipath device and the underlying devices), or introduce this new paths subdir in the kernel. Any of those solutions would require changes to initramfs-tools. The latter would require a kernel change.

The only way of not changing initramfs-tools is going back to using slaves relationships. That means we would need to get rid of the hidden disks thing. This always worked for dm-multipath, but I believe this required some use of blkdev and that is one thing hch wanted to avoid, to have a very fast path for doing multipath.

I don't like the idea of using nvme cli that much, as it would be a hard dependency on initramfs-tools. I don't like assuming some of the implementation details on the kernel numbering/naming of those devices. So, I will take a stab at providing the paths interface.

Cascardo.

Changed in makedumpfile (Ubuntu Cosmic):
status: New → Invalid
Changed in makedumpfile (Ubuntu Bionic):
status: New → Invalid
Changed in linux (Ubuntu Bionic):
status: New → Confirmed
Changed in linux (Ubuntu Cosmic):
status: New → Confirmed
Changed in initramfs-tools (Ubuntu Cosmic):
status: New → Confirmed
Changed in initramfs-tools (Ubuntu Bionic):
status: New → Confirmed

As a workaround, I suggest setting nvme-core.ko multipath=0 parameter. Can you try that, rebooting, recreating the kdump-tools initrd, then crashing?

Cascardo.

Dimitri John Ledkov (xnox) wrote :

i downloaded that, and tried to poke it.
First of all, i've never seen a 77MB tarball of /sys alone.
Also unpacking that, is dangerous as it does explode in size. However, is debugging that, please only _list_ the files present in the tarball and only extract things you need/want.

Reuploaded to http://people.canonical.com/~xnox/lp1778844_sys.tar.gz for ease of access by canonical engineers.

------- Comment From <email address hidden> 2018-09-27 22:06 EDT-------
(In reply to comment #61)
> As a workaround, I suggest setting nvme-core.ko multipath=0 parameter. Can
> you try that, rebooting, recreating the kdump-tools initrd, then crashing?

Hi Cascardo,

Where/How to set this parameters? You mean, passing them as boot parameters?

Thanks
Hari

Just create a /etc/modprobe.d/nvme.conf file with the following contents, recreate initrd, remove kdump initrds so they are recreated in the next boot, then reboot.

# cat /etc/modprobe.d/nvme.conf
options nvme-core multipath=0
# update-initramfs -u
# rm /var/lib/kdump/initrd.img*
# reboot

First piece of the puzzle submitted:
http://lists.infradead.org/pipermail/linux-nvme/2018-September/020174.html

Tested with a small change on initramfs-tools, though I will wait for comments on the first one before submitting that to initramfs-tools upstream.

Manoj Iyer (manjo) on 2018-10-01
Changed in ubuntu-power-systems:
status: Triaged → In Progress

I have been working upstream to get this fixed on the nvme driver, but it might still take some time, I already have done a back and forth with hch. Will send a v3 later this month.

Meanwhile, there are multiple workarounds we documented so far in the bug, namely:

1) Disable nvme multipath, which is done by setting the multipath module
parameter for the nvme module on modprobe.d config files, regenerate
initrd, reboot, then regenerate kdump initrd.

2) Use MODULES=most instead of MODULES=dep on
/etc/kernel/postinst.d/kdump-tools and regenerate kdump initrd.

3) Force insert nvme modules into kdump initrd. Namely, just copy the
/boot/ initrd file to replace the one in /var/lib/kdump/.

Upstream discussion is progressing. I have tested some patches from hch, will send him some feedback later today and post the links here.

Upstream is a little reluctant to accept my changes, and seems to suggest that we find out the layout of the paths some other way (which is possible, but would be such a special case on initramfs-tools, only for nvme).

Cascardo.

On 01/15/2019 09:26 AM, Thadeu Lima de Souza Cascardo wrote:
> Upstream is a little reluctant to accept my changes, and seems to
> suggest that we find out the layout of the paths some other way (which
> is possible, but would be such a special case on initramfs-tools, only
> for nvme).
>
> Cascardo.
>
Hmm, I am really sorry about that - sux to do all that work and get that
kind of response.

So making the suggested changes seems like real work to me. Do you have
a rough estimate?

Terry

I have pushed a version of initramfs-tools to my ppa that seems to fix the problem. Can IBM test it?

add-apt-repository ppa:cascardo/kdump2

It's a version for bionic. Only initramfs-tools should come from this source. kdump-tools and makedumpfile should be from bionic-updates or bionic.

After that, remove the initrd images from /var/lib/kdump/, reboot, then test the crash.

So:

add-apt-repository ppa:cascardo/kdump2
apt install initramfs-tools
rm /var/lib/kdump/initr*
reboot

After reboot:

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

Changed in ubuntu-power-systems:
status: In Progress → Incomplete
To post a comment you must log in.