ISST:LTE: LPAR roselp1 kexec_core from reboot command

[bugproxy]

== Comment: #0 2016-02-08 18:18:35 ==
---Problem Description---
reboot from command line.
kernel cores on booting.

---uname output---
4.4.0-2-generic

Machine Type = 8286-42A

---System Hang---
 system hung.
need to reboot via hmc

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 install machine with ubuntu
setup generl post install scripts
reboot command line
fails on boot up

Stack trace output:
 no

Oops output:
 no

System Dump Info:
  The system was configured to capture a dump, however a dump was not produced.

== Comment: #1 -2016-02-08 18:19:27 ==

root@roselp1:/kte/tools# reboot
[ 878.305097] kdump-tools[63102]: Stopping kdump-tools: * unloaded kdump kernel

Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .Ubuntu 16.04. . . .[ 939.516426] kexec_core: Starting new kernel

== Comment: #3 - 2016-02-08 18:28:40 ==

> I am going to try to dump via hmc to capture kernel logs.
hmc dump not working. I am going to change the kernel command line from crash kernel to xmon and reproduce

I can't even get it to dump to xmon.

== Comment: #6 - 2016-02-09 05:25:54 ==
(In reply to comment #5)

> Please share kernel cmdline params.

root@roselp1:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-4.4.0-2-generic root=UUID=b0dee2d0-a2c9-43e2-a43b-70fec2cf6180 ro splash quiet xmon=on

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1544438/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[bugproxy (bugproxy)]

Canonical,

We are still debugging on the issue.

Thanks,

[bugproxy (bugproxy)]

Externalizing comment for canonical to take a look and advise:

root@roselp1:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-4.4.0-2-generic root=UUID=b0dee2d0-a2c9-43e2-a43b-70fec2cf6180 ro splash quiet smt-enabled=off xmon=on crashkernel=384M-:128M
root@roselp1:~#
root@roselp1:~# ls /var/crash/
.lock _usr_bin_host.0.crash
root@roselp1:~# service kdump-tools status
? kdump-tools.service - Kernel crash dump capture service
Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; vendor preset: enabled)
Active: active (exited) since Fri 2016-02-12 03:47:59 CST; 8min ago
Process: 4359 ExecStart=/etc/init.d/kdump-tools start (code=exited, status=0/SUCCESS)
Main PID: 4359 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 512)
CGroup: /system.slice/kdump-tools.service

Feb 12 03:47:59 roselp1 systemd[1]: Starting Kernel crash dump capture serv.....
Feb 12 03:47:59 roselp1 kdump-tools[4359]: Starting kdump-tools: * Broken s...c
Feb 12 03:47:59 roselp1 kdump-tools[4359]: * Creating symlink /var/lib/kdump...z
Feb 12 03:47:59 roselp1 kdump-tools[4359]: * /etc/default/kdump-tools: KDUMP...z
Feb 12 03:47:59 roselp1 systemd[1]: Started Kernel crash dump capture service.
Hint: Some lines were ellipsized, use -l to show in full.
root@roselp1:~#

root@roselp1:~# echo l > /proc/sysrq-trigger
[ 1034.134755] sysrq: SysRq : Show backtrace of all active CPUs
root@roselp1:~#
root@roselp1:~# echo 8 > /proc/sysrq-trigger
[ 1246.306981] sysrq: SysRq : Changing Loglevel
[ 1246.306989] sysrq: Loglevel set to 8

root@roselp1:~# poweroff --reboot

[-1;-1f[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[ 1441.837792] kexec_core: Starting new kernel
[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .[-1;-1fUbuntu 16.04[-1;-1f. . . .

------>> HUNG <<-----------

-----------------------------------

kdump and xmon are configured, however the lpar neither have kernel core dump nor did the lpar drop to xmon.

From console, while reboot
[ 1441.837792] kexec_core: Starting new kernel

Need to check kernel code
kernel/kexec_core.c:kernel_kexec()

------------------8<----------------
{
kexec_in_progress = true;
kernel_restart_prepare(NULL);
migrate_to_reboot_cpu();

/*
* migrate_to_reboot_cpu() disables CPU hotplug assuming that
* no further code needs to use CPU hotplug (which is true in
* the reboot case). However, the kexec path depends on using
* CPU hotplug again; so re-enable it here.
*/
cpu_hotplug_enable();
pr_emerg("Starting new kernel\n");
machine_shutdown();
}

machine_kexec(kexec_image);
-------------------8<--------------

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Hi team canonical,

Can you please take look on the issue and let us know, if you require any additional info or test to debug/solve the issue.

Thanks,
Chandan

[bugproxy (bugproxy)]

Alton,

Can you re-create with dmesg log level raised and logs moved to console instead of k-buffer (I am hoping to get some kernel debug log on console while reboot).

Step 1. Back_up the dmesg log

step 2. Raise dmesg console level
[root@dev-fedora ~]# dmesg --console-level=7
[root@dev-fedora ~]# sysctl kernel.printk
kernel.printk = 7 4 1 7

(another option to try) Make dmesg to print on console
[root@dev-fedora ~]# dmesg --console-on

Note: --console-level and --console-on are mutually exclusive option so cannot be used together.

step 3.
[root@dev-fedora ~]# poweroff --reboot

Please share your observation.

Thanks,
Chandan.

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

For the benefit of Canonical, this bug is likely related to LP bug 1546260 which is generally covering the issue with kexec issues causing kdump to not work.

[Luciano Chavez (lnx1138)]

Marking as a dup of bug 1546260 as the kexec fix also took care of this issue.