Failure to dump with trusty+3.16 on ppc64el

[Dave Chiluk]

Test case
# sudo apt-get install linux-crashdump

set USE_KDUMP=1 in /etc/default/kdump-tools

# sudo shutdown -r now

echo 1 | sudo tee /proc/sys/kernel/sysrq
echo c | sudo tee /proc/sysrq-trigger

It looks like there was insufficient memory devoted to the crash kernel. The defaults were used, and the kernel had 256G of ram, and only 2.6G were in use at the time of inducing the crash.

________________________Console log ________________________

[ 290.509423] SysRq : Trigger a crash
[ 290.509526] Unable to handle kernel paging request for data at address 0x00000000
[ 290.509606] Faulting instruction address: 0xc0000000005d9c94
[ 290.509672] Oops: Kernel access of bad area, sig: 11 [#1]
[ 290.509723] SMP NR_CPUS=2048 NUMA PowerNV
[ 290.509776] Modules linked in: ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_crypt rtc_generic i2c_opal powernv_rng uio_pdrv_genirq ipmi_powernv ipmi_msghandler uio mlx4_en vxlan ses enclosure mlx4_core ipr
[ 290.510178] CPU: 121 PID: 2976 Comm: tee Not tainted 3.16.0-69-generic #89~14.04.1-Ubuntu
[ 290.510254] task: c000001fdccf4a80 ti: c000001fdcd58000 task.ti: c000001fdcd58000
[ 290.510330] NIP: c0000000005d9c94 LR: c0000000005dad0c CTR: c0000000005d9c60
[ 290.510406] REGS: c000001fdcd5b9d0 TRAP: 0300 Not tainted (3.16.0-69-generic)
[ 290.510480] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28004024 XER: 20000000
[ 290.510671] CFAR: c000000000009368 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1
GPR00: c0000000005dad0c c000001fdcd5bc50 c0000000013d7d00 0000000000000063
GPR04: c000000006548540 c000000006558da8 0000000000016fa0 c000000001596218
GPR08: c000000000e37d00 0000000000000000 0000000000000001 0000000000016fa0
GPR12: c0000000005d9c60 c000000007bc4100 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 0000000000000000 00000000100094d8 00000000100094c0
GPR24: 0000000000000001 00000000100094d8 0000000000000004 0000000000000000
GPR28: c00000000130f6c0 0000000000000063 c0000000012ed7a0 c00000000130fa80
[ 290.511676] NIP [c0000000005d9c94] sysrq_handle_crash+0x34/0x50
[ 290.511742] LR [c0000000005dad0c] __handle_sysrq+0xec/0x280
[ 290.511793] Call Trace:
[ 290.511820] [c000001fdcd5bc50] [c000001fdcd5bcb0] 0xc000001fdcd5bcb0 (unreliable)
[ 290.511911] [c000001fdcd5bc70] [c0000000005dad0c] __handle_sysrq+0xec/0x280
[ 290.511988] [c000001fdcd5bd10] [c0000000005db4dc] write_sysrq_trigger+0x7c/0xa0
[ 290.512078] [c000001fdcd5bd40] [c00000000032e1d0] proc_reg_write+0xb0/0x110
[ 290.512155] [c000001fdcd5bd90] [c0000000002a47ec] vfs_write+0xdc/0x260
[ 290.512231] [c000001fdcd5bde0] [c0000000002a558c] SyS_write+0x6c/0x110
[ 290.512308] [c000001fdcd5be30] [c00000000000a1d8] system_call+0x38/0xd0
[ 290.512383] Instruction dump:
[ 290.512421] 3842e0a0 7c0802a6 f8010010 f821ffe1 60000000 60000000 3d42001b 392a006c
[ 290.512546] 39400001 91490000 7c0004ac 39200000 <99490000> 38210020 e8010010 7c0803a6
[ 290.512674] ---[ end trace ba4afa55b8a163cd ]---
[ 290.512735]
[ 290.513754] Sending IPI to other CPUs
[ 290.514897] IPI complete
[ 0.000000] OPAL V3 detected !
[ 0.000000] Using PowerNV machine description
[ 0.000000] Page sizes from device-tree:
[ 0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
[ 0.000000] base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7
[ 0.000000] base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56
[ 0.000000] base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1
[ 0.000000] base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=8
[ 0.000000] base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0
[ 0.000000] base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, tlbiel=0, penc=3
[ 0.000000] Using 1TB segments
[ 0.000000] kvm_cma: CMA: failed to reserve 128 MiB
[ 0.000000] Found initrd at 0xc000000009730000:0xc00000000afd9bd6
[ 0.000000] bootconsole [udbg0] enabled
[ 0.000000] CPU maps initialized for 8 threads per core
 -> smp_release_cpus()
spinning_secondaries = 159
 <- smp_release_cpus()
[ 0.000000] Starting Linux PPC64 #89~14.04.1-Ubuntu SMP Thu Mar 17 20:50:51 UTC 2016
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x0
[ 0.000000] physicalMemorySize = 0x19c00000
[ 0.000000] htab_address = 0xc00000000f800000
[ 0.000000] htab_hash_mask = 0xfff
[ 0.000000] physical_start = 0x8000000
[ 0.000000] -----------------------------------------------------
 <- setup_system()
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 3.16.0-69-generic (buildd@bos01-ppc64el-017) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #89~14.04.1-Ubuntu SMP Thu Mar 17 20:50:51 UTC 2016 (Ubuntu 3.16.0-69.89~14.04.1-generic 3.16.7-ckt25)
[ 0.000000] [boot]0012 Setup Arch
[ 0.000000] Initializing IODA2 OPAL PHB /pciex@3fffe40000000
[ 0.000000] PCI host bridge /pciex@3fffe40000000 (primary) ranges:
[ 0.000000] MEM 0x00003fe000000000..0x00003fe07ffeffff -> 0x0000000080000000
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x10000000000 [segment=0x100000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0x800)
[ 0.000000] Issue PHB reset ...
[ 0.000000] Initializing IODA2 OPAL PHB /pciex@3fffe40100000
[ 0.000000] PCI host bridge /pciex@3fffe40100000 ranges:
[ 0.000000] MEM 0x00003fe080000000..0x00003fe0fffeffff -> 0x0000000080000000
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x10000000000 [segment=0x100000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0x1000)
[ 0.000000] Issue PHB reset ...
[ 0.000000] Initializing IODA2 OPAL PHB /pciex@3fffe40400000
[ 0.000000] PCI host bridge /pciex@3fffe40400000 ranges:
[ 0.000000] MEM 0x00003fe200000000..0x00003fe27ffeffff -> 0x0000000080000000
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x10000000000 [segment=0x100000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0x2800)
[ 0.000000] Issue PHB reset ...
[ 0.000000] Initializing IODA2 OPAL PHB /pciex@3fffe40500000
[ 0.000000] PCI host bridge /pciex@3fffe40500000 ranges:
[ 0.000000] MEM 0x00003fe280000000..0x00003fe2fffeffff -> 0x0000000080000000
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x10000000000 [segment=0x100000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0x3000)
[ 0.000000] Issue PHB reset ...
[ 0.000000] Initializing IODA2 OPAL PHB /pciex@3fffe42000000
[ 0.000000] PCI host bridge /pciex@3fffe42000000 ranges:
[ 0.000000] MEM 0x00003ff000000000..0x00003ff07ffeffff -> 0x0000000080000000
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x10000000000 [segment=0x100000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0x20800)
[ 0.000000] Issue PHB reset ...
[ 0.000000] Initializing IODA2 OPAL PHB /pciex@3fffe42400000
[ 0.000000] PCI host bridge /pciex@3fffe42400000 ranges:
[ 0.000000] MEM 0x00003ff200000000..0x00003ff27ffeffff -> 0x0000000080000000
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x10000000000 [segment=0x100000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0x22800)
[ 0.000000] Issue PHB reset ...
[ 0.000000] Initializing IODA2 OPAL PHB /pciex@3fffe42500000
[ 0.000000] PCI host bridge /pciex@3fffe42500000 ranges:
[ 0.000000] MEM 0x00003ff280000000..0x00003ff2fffeffff -> 0x0000000080000000
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x10000000000 [segment=0x100000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0x23000)
[ 0.000000] Issue PHB reset ...
[ 0.000000] OPAL nvram setup, 1048576 bytes
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00000000-0x39bfffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00000000-0x0fffffff]
[ 0.000000] node 0: [mem 0x30000000-0x39bfffff]
[ 0.000000] Could not find start_pfn for node 1
[ 0.000000] Could not find start_pfn for node 16
[ 0.000000] Could not find start_pfn for node 17
[ 0.000000] [boot]0015 Setup Done
[ 0.000000] bootmem alloc of 41943040 bytes failed!
[ 0.000000] Kernel panic - not syncing: Out of memory
[ 0.000000] CPU: 41 PID: 0 Comm: swapper Not tainted 3.16.0-69-generic #89~14.04.1-Ubuntu
[ 0.000000] Call Trace:
[ 0.000000] [c0000000093d7b50] [c000000008017340] show_stack+0x170/0x290 (unreliable)
[ 0.000000] [c0000000093d7c30] [c0000000089eedd8] dump_stack+0xc4/0x120
[ 0.000000] [c0000000093d7c70] [c0000000089e5e1c] panic+0x104/0x2b8
[ 0.000000] [c0000000093d7d00] [c000000008d7fd58] ___alloc_bootmem_node+0x4c/0x64
[ 0.000000] [c0000000093d7d70] [c000000008d5ac60] pcpu_fc_alloc+0x50/0x64
[ 0.000000] [c0000000093d7d90] [c000000008d7db04] pcpu_embed_first_chunk+0x5e8/0x874
[ 0.000000] [c0000000093d7e80] [c000000008d5b754] setup_per_cpu_areas+0x60/0x140
[ 0.000000] [c0000000093d7f00] [c000000008d53aa0] start_kernel+0x174/0x53c
[ 0.000000] [c0000000093d7f90] [c000000008009b6c] start_here_common+0x20/0xa8
[ 0.000000] ---[ end Kernel panic - not syncing: Out of memory

[Dave Chiluk (chiluk)]

Other kernel combinations may also be affected.

[Dave Chiluk (chiluk)]

It was also suggested that
nr_cpus=1 and or kvm_cma_resv_ratio=0 should be considered for defaults for the crash kernel.

[Nish Aravamudan (nacc)]

Couple of thoughts, which we probably should get IBM's advice on:

I believe this is using: crashkernel=384M-:128M

128M reserved seems too small for so much memory, tbh.

Also, I see that all the CPUs are on in the crashkernel, when realistically only 1 CPU is needed (and there have been issues in the past with SMP in the kdump kernel, iirc).

Finally, 'kvm_cma: CMA: failed to reserve 128 MiB' probably indicates we should pass kvm_cma_resv_ratio=0 to the kdump kernel (we're never going to run guests in the kdump environment).

[Dave Chiluk (chiluk)]

According to
https://wiki.ubuntu.com/ppc64el/Recommendations#Crash_Kernel_recommendations

it looks like the following should be used.
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M

These should be made as defaults for crash on ppc64 at the very least.

[Nish Aravamudan (nacc)]

On 07.04.2016 [17:18:58 -0000], Dave Chiluk wrote:
> According to
> https://wiki.ubuntu.com/ppc64el/Recommendations#Crash_Kernel_recommendations
>
> it looks like the following should be used.
> crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
>
> These should be made as defaults for crash on ppc64 at the very least.

FYI, the default actually comes from src:kexec-tools:

cat debian/kexec-tools.grub
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT crashkernel=384M-:128M"

which, in and of itself, seems odd as if you only install kexec-tools,
you've now reserved some system memory for kdump.

And if you remove/purge kexec-tools, crashkernel= seems to stick around
:)

[Louis Bouchard (louis)]

Hello,

Regarding nr_cpus=1, the equivalent maxcpus=1 is set in the kexec command (at least on default installs) :

$ kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x2b000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-4.4.0-17-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.4.0-17-generic
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/vmlinuz-4.4.0-17-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash vt.handoff=7 irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
                                  ^^^^^^^^^^^

Maybe disabling SMP alltogether by setting maxcpus=0 could be considered but that shouldn't change much aside from not reserving any SMP data structure. To Be Tested.

Regarding kvm_cma_resv_ratio=0, this will avoid the error message but it has on bearing on the current situation. It failed to allocated it so the memory was not in use.

256Gb of RAM doesn't mean that 128Mb needs to be increased. Here is the output of free on a 128Gb system right after a kernel panic :

(initramfs) chroot /root free -h
              total used free shared buff/cache available
Mem: 99M 20M 21M 48K 57M 62M
Swap: 0B 0B 0B

And here is the memory allocation in the same context :
(initramfs) chroot /root cat /proc/meminfo
MemTotal: 102000 kB
MemFree: 22260 kB
MemAvailable: 63700 kB
Buffers: 1640 kB
Cached: 33944 kB
SwapCached: 0 kB
Active: 12400 kB
Inactive: 23584 kB
Active(anon): 416 kB
Inactive(anon): 28 kB
Active(file): 11984 kB
Inactive(file): 23556 kB
Unevictable: 0 kB ...

[Nish Aravamudan (nacc)]

On 08.04.2016 [11:04:47 -0000], Louis Bouchard wrote:
> Hello,
>
> Regarding nr_cpus=1, the equivalent maxcpus=1 is set in the kexec
> command (at least on default installs) :

Yep, you're right, sorry!

> Maybe disabling SMP alltogether by setting maxcpus=0 could be considered
> but that shouldn't change much aside from not reserving any SMP data
> structure. To Be Tested.

Agreed, shouldn't be necessary for this bug.

> Regarding kvm_cma_resv_ratio=0, this will avoid the error message but
> it has on bearing on the current situation. It failed to allocated it so
> the memory was not in use.

Right, for correctness, though.

> 256Gb of RAM doesn't mean that 128Mb needs to be increased. Here is the
> output of free on a 128Gb system right after a kernel panic :

On Power, yes it does. Were you on a Power system for the below?

Here are the "official" Ubuntu recommendations
(https://wiki.ubuntu.com/ppc64el/Recommendations):

crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M

This is based upon actual testing on Power by IBM, iirc. Power's memory
consumption pattern is very different than x86.

Based upon your meminfo output, you were on x86, which is not relevant
to this issue.

Note the above is architecture-specific -- is it possible to modify
kexec-tools to suppor a per-arch crashkernel= default?

[Louis Bouchard (louis)]

I honnestly think that this assignment should be removed from kexec-tool & moved over to kdump-tools. I'd be happy to do it, which would facilitate such hardware-specific customization.

I have another requirement for a similar tailoring on s390x so I could joint them both.

Now I'd be happy to get results on a higher crashkernel value on ppc64el, but I do still believe that it is not the amount of memory that is causing this, but where the crashkernel memory is located.

[Dave Chiluk (chiluk)]

I received another update from the end user.
"A fresh 14.04.4 DVD Ubuntu install with updates (3.16.0-69 kernel) in a
diskful setup (no NFS, no aufs, 1T swap disk) also fails to kdump, either hangs or OOMS."

My understanding is that it OOMS on too little crashkernel assignment, hangs if enough crashkernel is provided.

[Nish Aravamudan (nacc)]

On 12.04.2016 [18:51:36 -0000], Dave Chiluk wrote:
> I received another update from the end user.
> "A fresh 14.04.4 DVD Ubuntu install with updates (3.16.0-69 kernel) in a
> diskful setup (no NFS, no aufs, 1T swap disk) also fails to kdump, either hangs or OOMS."
>
> My understanding is that it OOMS on too little crashkernel assignment,
> hangs if enough crashkernel is provided.

Is the end user willing to test if relocation is sufficient to fix the
problem (appending @32M or so, iirc, so crashkernel=<whatever>@32M).

What was "enough" for their environment?

Does providing the suggested value earlier
(crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M)
succeed?

[Dave Chiluk (chiluk)]

I can confirm that the following settings for crashkernel on powernv result in the following console log.
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M

I will now test with @32M options.

modoc login: [ 1374.418348] SysRq : Trigger a crash
[ 1374.418424] Unable to handle kernel paging request for data at address 0x00000000
[ 1374.418430] Faulting instruction address: 0xc0000000005eeb94
[ 1374.418438] Oops: Kernel access of bad area, sig: 11 [#1]
[ 1374.418442] SMP NR_CPUS=2048 NUMA PowerNV
[ 1374.418448] Modules linked in: ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_crypt rtc_generic uio_pdrv_genirq uio powernv_rng dm_round_robin scsi_dh_alua ses enclosure dm_multipath scsi_dh ipr
[ 1374.418782] CPU: 57 PID: 3884 Comm: tee Not tainted 3.16.0-44-generic #59-Ubuntu
[ 1374.418851] task: c000000fd0adcf60 ti: c000000fc9558000 task.ti: c000000fc9558000
[ 1374.418919] NIP: c0000000005eeb94 LR: c0000000005efc2c CTR: c0000000005eeb60
[ 1374.418986] REGS: c000000fc955b9d0 TRAP: 0300 Not tainted (3.16.0-44-generic)
[ 1374.419053] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28004024 XER: 20000000
[ 1374.419225] CFAR: c000000000009368 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1
GPR00: c0000000005efc2c c000000fc955bc50 c0000000013e5100 0000000000000063
GPR04: c000000013e48540 c000000013e58da8 00000000000004fd c0000000015c4e40
GPR08: c000000000e45100 0000000000000001 0000000000000000 00000000000004fd
GPR12: c0000000005eeb60 c000000007ba0100 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 0000000000000000 0000000010009d08 0000000000000001
GPR24: 0000000010009d08 00003fffec7fe180 c000000001335828 0000000000000004
GPR28: 0000000000000063 c0000000013005a8 c000000001335be8 0000000000000000
[ 1374.420128] NIP [c0000000005eeb94] sysrq_handle_crash+0x34/0x50
[ 1374.420186] LR [c0000000005efc2c] __handle_sysrq+0xec/0x280
[ 1374.420231] Call Trace:
[ 1374.420256] [c000000fc955bc50] [c000000fc955bcb0] 0xc000000fc955bcb0 (unreliable)
[ 1374.420336] [c000000fc955bc70] [c0000000005efc2c] __handle_sysrq+0xec/0x280
[ 1374.420404] [c000000fc955bd10] [c0000000005f0428] write_sysrq_trigger+0x78/0xa0
[ 1374.420485] [c000000fc955bd40] [c00000000033af30] proc_reg_write+0xb0/0x110
[ 1374.420555] [c000000fc955bd90] [c0000000002aef7c] vfs_write+0xdc/0x260
[ 1374.420624] [c000000fc955bde0] [c0000000002afcec] SyS_write+0x6c/0x110
[ 1374.420694] [c000000fc955be30] [c00000000000a1d8] system_call+0x38/0xd0
[ 1374.420761] Instruction dump:
[ 1374.420795] 384265a0 7c0802a6 f8010010 f821ffe1 60000000 60000000 3d22001b 39492c6c
[ 1374.420909] 39200001 912a0000 7c0004ac 39400000 <992a0000> 38210020 e8010010 7c0803a6
[ 1374.421027] ---[ end trace 039edbe0a3f4d32b ]---
[ 1374.421073]
[ 1376.421463] Kernel panic - not syncing: Fatal exception
[ 1376.421618] ---[ end Kernel panic - not syncing: Fatal exception

[Nish Aravamudan (nacc)]

On 12.04.2016 [19:53:57 -0000], Dave Chiluk wrote:
> I can confirm that the following settings for crashkernel on powernv result in the following console log.
> crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M

If I read that correctly, it failed to kdump at all? Can you provide the
dmesg before you trigger the crash?

[Dave Chiluk (chiluk)]

It turns out I was hitting issues related to the following bug.
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1569567

I will rerun my test

[Dave Chiluk (chiluk)]

With
ubuntu@modoc:~$ cat /proc/cmdline
root=/dev/mapper/mpath0-part2 ro console=hvc0 crashkernel=2G-4G:320M@32M,4G-32G:512M@32M,32G-64G:1024M@32M,64G-128G:2048M@32M,128G-:4096M@32M

I get the following console log
[ 191.833046] SysRq : Trigger a crash
[ 191.833117] Unable to handle kernel paging request for data at address 0x00000000
[ 191.833123] Faulting instruction address: 0xc0000000005eeb94
[ 191.83

That is it. I will continue testing.

[Nish Aravamudan (nacc)]

On 12.04.2016 [21:13:40 -0000], Dave Chiluk wrote:
> With
> ubuntu@modoc:~$ cat /proc/cmdline
> root=/dev/mapper/mpath0-part2 ro console=hvc0 crashkernel=2G-4G:320M@32M,4G-32G:512M@32M,32G-64G:1024M@32M,64G-128G:2048M@32M,128G-:4096M@32M
>
> I get the following console log
> [ 191.833046] SysRq : Trigger a crash
> [ 191.833117] Unable to handle kernel paging request for data at address 0x00000000
> [ 191.833123] Faulting instruction address: 0xc0000000005eeb94
> [ 191.83
>
> That is it. I will continue testing.

This is over the IPMI console? Sorry I should have been more explicit,
but is it possible to provide the full logs from before the crash is
attempted? (dmesg is fine).

[Dave Chiluk (chiluk)]

So it looks like this crashkernel argument is not resulting in any reserved memory.
The correct crashkernel argument should look like this.
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@32M

Additionally my earlier failures here were the result of no reserved memory being allocated due to the way crashkernel parses the @32M.

So to recap, you definitely need the @32M. And there seems to be a parsing issue if multiple @##M offsets are desired.

[Nish Aravamudan (nacc)]

On 13.04.2016 [23:44:39 -0000], Dave Chiluk wrote:
> So it looks like this crashkernel argument is not resulting in any reserved memory.
> The correct crashkernel argument should look like this.
> crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@32M
>
> Additionally my earlier failures here were the result of no reserved
> memory being allocated due to the way crashkernel parses the @32M.

Totally my fault, I glossed that on reading, there is only one offset
allowed, aiui, as only one of the above cases ever applies to a given
execution. And the offset should not need to be different for differnt
values (as the pre-allocated memory you are presumably bumping into is
at the same location in all cases).

[Launchpad Janitor (janitor)]

This bug was fixed in the package makedumpfile - 1:1.5.9-7

---------------
makedumpfile (1:1.5.9-7) sid; urgency=medium

  * [d/rules] Lower kexec-tools dependency to -2
      The ubuntu merge will happen on an 1:2.0.10-2 version so it cannot
      depends on -3.

 -- Louis Bouchard <email address hidden> Tue, 31 May 2016 14:36:07 +0200