------- Comment From <email address hidden> 2017-09-13 13:18 EDT------- Hi
Today I tested kdump with 16.10 on talclp3 Access info : HMC: hmc-lte2.isst.aus.stglabs.ibm.com (hscroot/abc123)
Console Access: rmvterm -m talc -p talclp3;mkvterm -m talc -p talclp3;
Logs:
root@talclp3:~# echo c > /proc/sysrq-trigger [ 424.180480] sysrq: SysRq : Trigger a crash [ 424.180497] Unable to handle kernel paging request for data at address 0x00000000 [ 424.180500] Faulting instruction address: 0xc0000000006a2428 [ 424.180504] Oops: Kernel access of bad area, sig: 11 [#1] [ 424.180506] SMP NR_CPUS=2048 NUMA pSeries [ 424.180509] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) configfs ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_ib(OE) pseries_rng ib_core(OE) vmx_crypto binfmt_misc dm_round_robin sunrpc dm_multipath knem(OE) ip_tables x_tables autofs4 btrfs xor raid6_pq mlx4_en(OE) ibmvfc scsi_transport_fc ibmvscsi bnx2x mlx4_core(OE) devlink mlx_compat(OE) mdio libcrc32c be2net crc32c_vpmsum [ 424.180541] CPU: 0 PID: 2733 Comm: bash Tainted: G OE 4.8.0-59-generic #64-Ubuntu [ 424.180545] task: c0000000b3d78600 task.stack: c0000000a2104000 [ 424.180547] NIP: c0000000006a2428 LR: c0000000006a3478 CTR: c0000000006a2400 [ 424.180550] REGS: c0000000a21079f0 TRAP: 0300 Tainted: G OE (4.8.0-59-generic) [ 424.180553] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 28222222 XER: 00000001 [ 424.180560] CFAR: c000000000008750 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1 GPR00: c0000000006a3478 c0000000a2107c70 c000000001467500 0000000000000063 GPR04: c0000000bd00aca0 c0000000bd01fb40 c00000017fd2e300 000000000000b240 GPR08: 0000000000000007 0000000000000001 0000000000000000 0000000000000001 GPR12: c0000000006a2400 c000000007b30000 0000000000000000 0000000022000000 GPR16: 0000000010170dc8 000001000df90258 0000000010140528 00000000100c6f60 GPR20: 0000000000000000 000000001017dd58 0000000010152bf0 000000001017b608 GPR24: 00003ffff97be144 00003ffff97be140 c00000000137e6e0 0000000000000004 GPR28: c00000000137eaa0 0000000000000063 c000000001332590 0000000000000000 [ 424.180599] NIP [c0000000006a2428] sysrq_handle_crash+0x28/0x30 [ 424.180602] LR [c0000000006a3478] __handle_sysrq+0xe8/0x280 [ 424.180604] Call Trace: [ 424.180606] [c0000000a2107c70] [c0000000006a3458] __handle_sysrq+0xc8/0x280 (unreliable) [ 424.180610] [c0000000a2107d10] [c0000000006a3bcc] write_sysrq_trigger+0x6c/0x90 [ 424.180614] [c0000000a2107d40] [c0000000003adb48] proc_reg_write+0x88/0xd0 [ 424.180619] [c0000000a2107d70] [c0000000003105ac] __vfs_write+0x3c/0x70 [ 424.180622] [c0000000a2107d90] [c000000000311814] vfs_write+0xd4/0x240 [ 424.180625] [c0000000a2107de0] [c000000000313368] SyS_write+0x68/0x110 [ 424.180629] [c0000000a2107e30] [c000000000009584] system_call+0x38/0xec [ 424.180631] Instruction dump: [ 424.180633] 60000000 60000000 3c4c00dc 38425100 7c0802a6 60000000 3d22001a 3949bc60 [ 424.180639] 39200001 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00dc 384250d0 [ 424.180645] ---[ end trace 8fd1cd00c31ebdd4 ]--- [ 424.183431] [ 424.183450] Sending IPI to other CPUs [ 424.183452] IPI complete I'm in purgatory -> smp_release_cpus() spinning_secondaries = 47 <- smp_release_cpus() [ 0.184530] pci 002b:50:00.0: of_irq_parse_pci() failed with rc=-22 [ 0.569039] Kernel panic - not syncing: Out of memory and no killable processes... [ 0.569039] [ 0.569066] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.8.0-59-generic #64-Ubuntu [ 0.569069] Call Trace: [ 0.569071] [c00000000d10b220] [c000000008b0fe4c] dump_stack+0xb0/0xf0 (unreliable) [ 0.569075] [c00000000d10b260] [c000000008b0bf58] panic+0x144/0x308 [ 0.569078] [c00000000d10b2f0] [c000000008249c2c] out_of_memory+0x48c/0x570 [ 0.569082] [c00000000d10b3a0] [c000000008250ad8] __alloc_pages_nodemask+0xdf8/0xe20 [ 0.569086] [c00000000d10b560] [c0000000082c6da8] alloc_page_interleave+0x58/0xc0 [ 0.569089] [c00000000d10b5a0] [c0000000082c7678] alloc_pages_current+0x168/0x1d0 [ 0.569093] [c00000000d10b600] [c0000000082435e8] __page_cache_alloc+0x118/0x160 [ 0.569096] [c00000000d10b640] [c0000000082437b4] pagecache_get_page+0x184/0x3c0 [ 0.569100] [c00000000d10b6b0] [c000000008243a34] grab_cache_page_write_begin+0x44/0x70 [ 0.569103] [c00000000d10b6e0] [c00000000834bf6c] simple_write_begin+0x4c/0x1b0 [ 0.569107] [c00000000d10b730] [c000000008243264] generic_perform_write+0x104/0x280 [ 0.569111] [c00000000d10b7d0] [c000000008245540] __generic_file_write_iter+0x1e0/0x230 [ 0.569114] [c00000000d10b830] [c00000000824567c] generic_file_write_iter+0xec/0x250 [ 0.569118] [c00000000d10b870] [c00000000831050c] new_sync_write+0xec/0x150 [ 0.569121] [c00000000d10b900] [c000000008311814] vfs_write+0xd4/0x240 [ 0.569124] [c00000000d10b950] [c000000008313368] SyS_write+0x68/0x110 [ 0.569127] [c00000000d10b9a0] [c000000008ea5d0c] xwrite+0x4c/0xb0 [ 0.569130] [c00000000d10b9e0] [c000000008ea5e60] do_copy+0xf0/0x170 [ 0.569133] [c00000000d10ba10] [c000000008ea59c4] write_buffer+0x5c/0x88 [ 0.569136] [c00000000d10ba40] [c000000008ea5a50] flush_buffer+0x60/0xec [ 0.569140] [c00000000d10ba90] [c000000008eec4c8] __gunzip+0x378/0x47c [ 0.569142] [c00000000d10bb10] [c000000008ea650c] unpack_to_rootfs+0x1c8/0x338 [ 0.569146] [c00000000d10bbc0] [c000000008ea688c] populate_rootfs+0x94/0x17c [ 0.569149] [c00000000d10bc40] [c00000000800b948] do_one_initcall+0x68/0x1d0 [ 0.569152] [c00000000d10bd00] [c000000008ea42e8] kernel_init_freeable+0x278/0x360 [ 0.569156] [c00000000d10bdc0] [c00000000800c1b4] kernel_init+0x24/0x170 [ 0.569159] [c00000000d10be30] [c0000000080098f0] ret_from_kernel_thread+0x5c/0x6c [ 0.571060] ---[ end Kernel panic - not syncing: Out of memory and no killable processes... [ 0.571060]
root@talclp3:~# service kdump-tools status * kdump-tools.service - Kernel crash dump capture service Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; vendor pres Active: active (exited) since Wed 2017-09-13 12:02:16 CDT; 3min 28s ago Main PID: 2281 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 9830) CGroup: /system.slice/kdump-tools.service
Sep 13 12:02:14 talclp3 systemd[1]: Starting Kernel crash dump capture service.. Sep 13 12:02:15 talclp3 kdump-tools[2281]: Starting kdump-tools: Modified cmdlin Sep 13 12:02:16 talclp3 kdump-tools[2281]: * loaded kdump kernel Sep 13 12:02:16 talclp3 kdump-tools[2581]: /sbin/kexec -p --command-line="BOOT_I Sep 13 12:02:16 talclp3 kdump-tools[2582]: loaded kdump kernel Sep 13 12:02:16 talclp3 systemd[1]: Started Kernel crash dump capture service. root@talclp3:~# root@talclp3:~# uname -a Linux talclp3 4.8.0-59-generic #64-Ubuntu SMP Thu Jun 29 19:36:04 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux root@talclp3:~# uname -r 4.8.0-59-generic root@talclp3:~# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.10 DISTRIB_CODENAME=yakkety DISTRIB_DESCRIPTION="Ubuntu 16.10" root@talclp3:~# cat /proc/cmdline BOOT_IMAGE=/boot/vmlinux-4.8.0-59-generic root=UUID=30629c5d-7ff0-48db-b2ca-7c2255d0fa18 ro splash quiet crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@32M maxcpus=1 crashkernel=384M-:128M root@talclp3:~# kdump-config show DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_SYSCTL: kernel.panic_on_oops=1 KDUMP_COREDIR: /var/crash crashkernel addr: /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.8.0-59-generic kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.8.0-59-generic current state: ready to kdump
kexec command: /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinux-4.8.0-59-generic root=UUID=30629c5d-7ff0-48db-b2ca-7c2255d0fa18 ro splash quiet maxcpus=1 irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz root@talclp3:~# apt list --installed|grep makedumpfile
makedumpfile/yakkety-updates,now 1:1.6.0-2ubuntu1.2 ppc64el [installed,automatic]
Thanks Lekshmi
------- Comment From <email address hidden> 2017-09-13 13:18 EDT-------
Hi
Today I tested kdump with 16.10 on talclp3 isst.aus. stglabs. ibm.com (hscroot/abc123)
Access info :
HMC: hmc-lte2.
Console Access: rmvterm -m talc -p talclp3;mkvterm -m talc -p talclp3;
Logs:
root@talclp3:~# echo c > /proc/sysrq-trigger ME,IR,DR, RI,LE> CR: 28222222 XER: 00000001 crash+0x28/ 0x30 sysrq+0xe8/ 0x280 sysrq+0xc8/ 0x280 (unreliable) trigger+ 0x6c/0x90 write+0x88/ 0xd0 0x3c/0x70 0xd4/0x240 0x68/0x110 call+0x38/ 0xec secondaries = 47 0xb0/0xf0 (unreliable) memory+ 0x48c/0x570 pages_nodemask+ 0xdf8/0xe20 interleave+ 0x58/0xc0 current+ 0x168/0x1d0 cache_alloc+ 0x118/0x160 get_page+ 0x184/0x3c0 page_write_ begin+0x44/ 0x70 write_begin+ 0x4c/0x1b0 perform_ write+0x104/ 0x280 file_write_ iter+0x1e0/ 0x230 file_write_ iter+0xec/ 0x250 write+0xec/ 0x150 0xd4/0x240 0x68/0x110 0x5c/0x88 0x60/0xec 0x378/0x47c to_rootfs+ 0x1c8/0x338 rootfs+ 0x94/0x17c initcall+ 0x68/0x1d0 init_freeable+ 0x278/0x360 init+0x24/ 0x170 kernel_ thread+ 0x5c/0x6c
[ 424.180480] sysrq: SysRq : Trigger a crash
[ 424.180497] Unable to handle kernel paging request for data at address 0x00000000
[ 424.180500] Faulting instruction address: 0xc0000000006a2428
[ 424.180504] Oops: Kernel access of bad area, sig: 11 [#1]
[ 424.180506] SMP NR_CPUS=2048 NUMA pSeries
[ 424.180509] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) configfs ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_ib(OE) pseries_rng ib_core(OE) vmx_crypto binfmt_misc dm_round_robin sunrpc dm_multipath knem(OE) ip_tables x_tables autofs4 btrfs xor raid6_pq mlx4_en(OE) ibmvfc scsi_transport_fc ibmvscsi bnx2x mlx4_core(OE) devlink mlx_compat(OE)
mdio libcrc32c be2net crc32c_vpmsum
[ 424.180541] CPU: 0 PID: 2733 Comm: bash Tainted: G OE 4.8.0-59-generic #64-Ubuntu
[ 424.180545] task: c0000000b3d78600 task.stack: c0000000a2104000
[ 424.180547] NIP: c0000000006a2428 LR: c0000000006a3478 CTR: c0000000006a2400
[ 424.180550] REGS: c0000000a21079f0 TRAP: 0300 Tainted: G OE (4.8.0-59-generic)
[ 424.180553] MSR: 8000000000009033 <SF,EE,
[ 424.180560] CFAR: c000000000008750 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1
GPR00: c0000000006a3478 c0000000a2107c70 c000000001467500 0000000000000063
GPR04: c0000000bd00aca0 c0000000bd01fb40 c00000017fd2e300 000000000000b240
GPR08: 0000000000000007 0000000000000001 0000000000000000 0000000000000001
GPR12: c0000000006a2400 c000000007b30000 0000000000000000 0000000022000000
GPR16: 0000000010170dc8 000001000df90258 0000000010140528 00000000100c6f60
GPR20: 0000000000000000 000000001017dd58 0000000010152bf0 000000001017b608
GPR24: 00003ffff97be144 00003ffff97be140 c00000000137e6e0 0000000000000004
GPR28: c00000000137eaa0 0000000000000063 c000000001332590 0000000000000000
[ 424.180599] NIP [c0000000006a2428] sysrq_handle_
[ 424.180602] LR [c0000000006a3478] __handle_
[ 424.180604] Call Trace:
[ 424.180606] [c0000000a2107c70] [c0000000006a3458] __handle_
[ 424.180610] [c0000000a2107d10] [c0000000006a3bcc] write_sysrq_
[ 424.180614] [c0000000a2107d40] [c0000000003adb48] proc_reg_
[ 424.180619] [c0000000a2107d70] [c0000000003105ac] __vfs_write+
[ 424.180622] [c0000000a2107d90] [c000000000311814] vfs_write+
[ 424.180625] [c0000000a2107de0] [c000000000313368] SyS_write+
[ 424.180629] [c0000000a2107e30] [c000000000009584] system_
[ 424.180631] Instruction dump:
[ 424.180633] 60000000 60000000 3c4c00dc 38425100 7c0802a6 60000000 3d22001a 3949bc60
[ 424.180639] 39200001 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00dc 384250d0
[ 424.180645] ---[ end trace 8fd1cd00c31ebdd4 ]---
[ 424.183431]
[ 424.183450] Sending IPI to other CPUs
[ 424.183452] IPI complete
I'm in purgatory
-> smp_release_cpus()
spinning_
<- smp_release_cpus()
[ 0.184530] pci 002b:50:00.0: of_irq_parse_pci() failed with rc=-22
[ 0.569039] Kernel panic - not syncing: Out of memory and no killable processes...
[ 0.569039]
[ 0.569066] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.8.0-59-generic #64-Ubuntu
[ 0.569069] Call Trace:
[ 0.569071] [c00000000d10b220] [c000000008b0fe4c] dump_stack+
[ 0.569075] [c00000000d10b260] [c000000008b0bf58] panic+0x144/0x308
[ 0.569078] [c00000000d10b2f0] [c000000008249c2c] out_of_
[ 0.569082] [c00000000d10b3a0] [c000000008250ad8] __alloc_
[ 0.569086] [c00000000d10b560] [c0000000082c6da8] alloc_page_
[ 0.569089] [c00000000d10b5a0] [c0000000082c7678] alloc_pages_
[ 0.569093] [c00000000d10b600] [c0000000082435e8] __page_
[ 0.569096] [c00000000d10b640] [c0000000082437b4] pagecache_
[ 0.569100] [c00000000d10b6b0] [c000000008243a34] grab_cache_
[ 0.569103] [c00000000d10b6e0] [c00000000834bf6c] simple_
[ 0.569107] [c00000000d10b730] [c000000008243264] generic_
[ 0.569111] [c00000000d10b7d0] [c000000008245540] __generic_
[ 0.569114] [c00000000d10b830] [c00000000824567c] generic_
[ 0.569118] [c00000000d10b870] [c00000000831050c] new_sync_
[ 0.569121] [c00000000d10b900] [c000000008311814] vfs_write+
[ 0.569124] [c00000000d10b950] [c000000008313368] SyS_write+
[ 0.569127] [c00000000d10b9a0] [c000000008ea5d0c] xwrite+0x4c/0xb0
[ 0.569130] [c00000000d10b9e0] [c000000008ea5e60] do_copy+0xf0/0x170
[ 0.569133] [c00000000d10ba10] [c000000008ea59c4] write_buffer+
[ 0.569136] [c00000000d10ba40] [c000000008ea5a50] flush_buffer+
[ 0.569140] [c00000000d10ba90] [c000000008eec4c8] __gunzip+
[ 0.569142] [c00000000d10bb10] [c000000008ea650c] unpack_
[ 0.569146] [c00000000d10bbc0] [c000000008ea688c] populate_
[ 0.569149] [c00000000d10bc40] [c00000000800b948] do_one_
[ 0.569152] [c00000000d10bd00] [c000000008ea42e8] kernel_
[ 0.569156] [c00000000d10bdc0] [c00000000800c1b4] kernel_
[ 0.569159] [c00000000d10be30] [c0000000080098f0] ret_from_
[ 0.571060] ---[ end Kernel panic - not syncing: Out of memory and no killable processes...
[ 0.571060]
root@talclp3:~# service kdump-tools status system/ kdump-tools. service; enabled; vendor pres slice/kdump- tools.service
* kdump-tools.service - Kernel crash dump capture service
Loaded: loaded (/lib/systemd/
Active: active (exited) since Wed 2017-09-13 12:02:16 CDT; 3min 28s ago
Main PID: 2281 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 9830)
CGroup: /system.
Sep 13 12:02:14 talclp3 systemd[1]: Starting Kernel crash dump capture service.. line="BOOT_ I RELEASE= 16.10 CODENAME= yakkety DESCRIPTION= "Ubuntu 16.10" /boot/vmlinux- 4.8.0-59- generic root=UUID= 30629c5d- 7ff0-48db- b2ca-7c2255d0fa 18 ro splash quiet crashkernel= 2G-4G:320M, 4G-32G: 512M,32G- 64G:1024M, 64G-128G: 2048M,128G- :4096M@ 32M maxcpus=1 crashkernel= 384M-:128M panic_on_ oops=1 kdump/vmlinuz: symbolic link to /boot/vmlinux- 4.8.0-59- generic kdump/initrd. img: symbolic link to /var/lib/ kdump/initrd. img-4.8. 0-59-generic
Sep 13 12:02:15 talclp3 kdump-tools[2281]: Starting kdump-tools: Modified cmdlin
Sep 13 12:02:16 talclp3 kdump-tools[2281]: * loaded kdump kernel
Sep 13 12:02:16 talclp3 kdump-tools[2581]: /sbin/kexec -p --command-
Sep 13 12:02:16 talclp3 kdump-tools[2582]: loaded kdump kernel
Sep 13 12:02:16 talclp3 systemd[1]: Started Kernel crash dump capture service.
root@talclp3:~#
root@talclp3:~# uname -a
Linux talclp3 4.8.0-59-generic #64-Ubuntu SMP Thu Jun 29 19:36:04 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
root@talclp3:~# uname -r
4.8.0-59-generic
root@talclp3:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_
DISTRIB_
DISTRIB_
root@talclp3:~# cat /proc/cmdline
BOOT_IMAGE=
root@talclp3:~# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.
KDUMP_COREDIR: /var/crash
crashkernel addr:
/var/lib/
kdump initrd:
/var/lib/
current state: ready to kdump
kexec command: line="BOOT_ IMAGE=/ boot/vmlinux- 4.8.0-59- generic root=UUID= 30629c5d- 7ff0-48db- b2ca-7c2255d0fa 18 ro splash quiet maxcpus=1 irqpoll nr_cpus=1 nousb systemd. unit=kdump- tools.service" --initrd= /var/lib/ kdump/initrd. img /var/lib/ kdump/vmlinuz
/sbin/kexec -p --command-
root@talclp3:~# apt list --installed|grep makedumpfile
makedumpfile/ yakkety- updates, now 1:1.6.0-2ubuntu1.2 ppc64el [installed, automatic]
Thanks
Lekshmi