Customize 'crashkernel' parameter is not properly working

Bug #1790788 reported by Frank Heimes on 2018-09-05
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
High
Unassigned
makedumpfile (Ubuntu)
High
Thadeu Lima de Souza Cascardo
Xenial
Undecided
Unassigned
Bionic
High
Thadeu Lima de Souza Cascardo
Cosmic
High
Thadeu Lima de Souza Cascardo
Disco
High
Thadeu Lima de Souza Cascardo

Bug Description

SRU Justification:
------------------

[Impact]

 * While installing makedumpfile the "crashkernel=" argument is not properly set, hence dump is not triggered on reboot.

 * Means the triggering of dumpfiles is currently not possible using makedumpfile.

 * Dumpfiles are obviously only needed in rare cases, but if they are needed (e.g. in production environments) the situation is usually critical.

 * Hence fixing this is needed to allow post-mortem analysis of a dumps.

 * The provided shell code snippet provides a fixed sed statement that makes sure that the kernel parameter is propoerly set.

[Test Case]

 * Create and boot a s390x (KVM virtual) machine

 * Install kdump-tools and makedumpfile
   Select 'yes' on question 'Should kdump-tools be enabled by default?' during installation

 * [ Reboot system ]

 * Look for crashkernel line in zipl boot-loader
   grep crashkernel /etc/zipl.conf
   crashkernel line is missing in case this bug still exists
   one or more lines like this should be given:
   parameters = root=UUID=5ed8f208-adce-4fad-b1a6-feb5e8732d89 crashkernel=196M

 * One may further trigger a crash (for a full positiv test)
   sudo -s
   sysctl -w kernel.sysrq=1
   echo c > /proc/sysrq-trigger
   (in case this bug still exists the system will not come up again - check console in parallel)
   Finally dump files should be visible in /var/crash

[Regression Potential]

 * The regression potential is very low, since:

 * it's limited to the zipl boot loader configuration file only
   and this means again it's on the s390x platform only (IBM Z)

 * kdump-tools and makedumpfile are not installed by default and only used in debug situations
   hence only system where the package(s) got manually installed get updated

 * The function is today broken anyway, hence it can actually only get better

 * I successfully verified this in disco.
_________________________

Trying to use crashdump especially in a KVM machine.
Installation looks fine and the reboot is triggered.
But it does not work because the kernel does not have a 'crashkernel=' parameter.
Nothing in /proc/cmdline:
$ cat /proc/cmdline
root=LABEL=cloudimg-rootfs

Issue seems to be in adding the crashkernel line in this snippet:
# Customize crashkernel= value according to architecture
ARCH="$(arch)"
DEF_PRESET="384M-:128M"
case "$ARCH" in
   s390x)
      HAS_CRASHKERNEL="$(grep crashkernel /etc/zipl.conf)" || true
      if test -z "$HAS_CRASHKERNEL"; then
         sed -i "/parameters/{s|\"$| crashkernel=${DEF_PRESET}\"|}" /etc/zipl.conf
         zipl
      fi
         CIO_IGNORE="$(cio_ignore -u -k)"
         sed -i "s/\#KDUMP_CMDLINE_APPEND/KDUMP_CMDLINE_APPEND/" $INITCONFFILE
         sed -i "/KDUMP_CMDLINE_APPEND/{s|\"$| ${CIO_IGNORE}\"|}" $INITCONFFILE
    ;;
esac

(especially 1st sed stmt)

Frank Heimes (fheimes) on 2018-09-05
affects: ubuntu-z-systems → makedumpfile (Ubuntu)

Ack to the report.
While I had some systems (lpar) which worked I was trivially able to confirm that it doesn't work on KVM guests based on cloud images.

I see this on install of kdump-tools (default answers to debconf)
Setting up kdump-tools (1:1.6.4-2) ...
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: vda (0000).
Done.

Creating config file /etc/default/kdump-tools with new version

But that leaves /etc/zipl.conf unmodified.
Frank already copied the snippet in the report.
This line isn't working:

$ sed -i "/parameters/{s|\"$| crashkernel=${DEF_PRESET}\"|}" /etc/zipl.conf

At least in my guest and my LPARs the following seemed better, so for your consideration:

$ sed -i "s/^parameters\s*=/& crashkernel=${DEF_PRESET} /" /etc/zipl.conf

Note: at this point it already checked there is no crashkernel statement yet, so we can unconditionally insert it here.

For your testing here the default content of zipl.conf on a cloud-img KVM guest

$ cat /etc/zipl.conf
# This has been modified by the cloud image build process
[defaultboot]
default=ubuntu

[ubuntu]
target = /boot
image = /boot/vmlinuz
parameters = root=LABEL=cloudimg-rootfs
ramdisk = /boot/initrd.img

This is also the reason for all the fails in [1] which actually flags a real issue as reported in this bug

[1]: http://autopkgtest.ubuntu.com/packages/m/makedumpfile/cosmic/s390x

Frank Heimes (fheimes) wrote :

Note: this also affects the makedumpfile autopkgtest

Assigned the task on request by JFH

Changed in makedumpfile (Ubuntu):
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Changed in makedumpfile (Ubuntu):
assignee: Canonical Kernel Team (canonical-kernel-team) → Thadeu Lima de Souza Cascardo (cascardo)
status: New → In Progress
importance: Undecided → High
Frank Heimes (fheimes) wrote :

Adding a sample zipl.conf of an LPAR installation - just for reference:

$ cat /etc/zipl.conf
[defaultboot]
defaultmenu = menu

:menu
target = /boot
1 = ubuntu
2 = old
default = 1
prompt = 1
timeout = 10

[ubuntu]
target = /boot
image = /boot/vmlinuz
ramdisk = /boot/initrd.img
parameters = root=UUID=83c2f8ac-7d56-4085-9d20-1e6e01175326 crashkernel=196M

[old]
target = /boot
image = /boot/vmlinuz.old
ramdisk = /boot/initrd.img.old
parameters = root=UUID=83c2f8ac-7d56-4085-9d20-1e6e01175326 crashkernel=196M
optional = 1

tags: added: patch

This fix has been uploaded to Debian, and will be synced to 19.04 when it opens. After that, it will be backported to cosmic, bionic and xenial.

Thanks.
Cascardo.

I am working on the merge of Debian and Ubuntu changes to Disco.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package makedumpfile - 1:1.6.5-1ubuntu1

---------------
makedumpfile (1:1.6.5-1ubuntu1) disco; urgency=low

  [ Ubuntu Merge-o-Matic ]
  * Merge from Debian unstable. Remaining changes:
    - Bump amd64 crashkernel from 384M-:128M to 512M-:192M.

  [ Thadeu Lima de Souza Cascardo ]
  * Use a different service for vmcore dump. (LP: #1811692)

makedumpfile (1:1.6.5-1) unstable; urgency=medium

  * Update to new upstream version 1.6.5.
  * debian: remove debian/source/local-options
  * [i18n] Move PT debconf translation (Closes: #910465)

makedumpfile (1:1.6.4-3) unstable; urgency=medium

  * Reload kdump after memory/CPU hotplug. (LP: #1655280)
  * Fix adding crashkernel to zipl.conf when no quotation mark is used.
    (LP: #1790788)

 -- Thadeu Lima de Souza Cascardo <email address hidden> Mon, 14 Jan 2019 15:42:44 -0200

Changed in makedumpfile (Ubuntu):
status: In Progress → Fix Released
Timo Aaltonen (tjaalton) wrote :

needs SRU template, test case etc

Frank Heimes (fheimes) wrote :

SRU justification added - see bug description

description: updated
Frank Heimes (fheimes) on 2019-03-08
description: updated
Changed in ubuntu-z-systems:
status: New → In Progress
Changed in makedumpfile (Ubuntu Bionic):
status: New → In Progress
assignee: nobody → Thadeu Lima de Souza Cascardo (cascardo)
importance: Undecided → High

Our makedumpfile updates are waiting for us to get approval for micro release updates. I am working on the justification so we can get exceptions for this and future updates, as makedumpfile is closely tied to the kernel version.

Hello Frank, or anyone else affected,

Accepted makedumpfile into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1~18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in makedumpfile (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Łukasz Zemczak (sil2100) wrote :

Hello Frank, or anyone else affected,

Accepted makedumpfile into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in makedumpfile (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Frank Heimes (fheimes) wrote :

Adjusting the cosmic-verification tag, since I was able to successfully verify the package from cosmic-proposed.

tags: added: verification-done-cosmic
removed: verification-needed-cosmic
Frank Heimes (fheimes) wrote :

Adjusting the bionic-verification tag, since I was able to successfully verify the package from bionic-proposed, too.

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package makedumpfile - 1:1.6.5-1ubuntu1~18.10.1

---------------
makedumpfile (1:1.6.5-1ubuntu1~18.10.1) cosmic; urgency=low

  * Backport back to cosmic. (LP: #1655280) (LP: #1790788)

makedumpfile (1:1.6.5-1ubuntu1) disco; urgency=low

  [ Ubuntu Merge-o-Matic ]
  * Merge from Debian unstable. Remaining changes:
    - Bump amd64 crashkernel from 384M-:128M to 512M-:192M.

  [ Thadeu Lima de Souza Cascardo ]
  * Use a different service for vmcore dump. (LP: #1811692)

makedumpfile (1:1.6.5-1) unstable; urgency=medium

  * Update to new upstream version 1.6.5.
  * debian: remove debian/source/local-options
  * [i18n] Move PT debconf translation (Closes: #910465)

makedumpfile (1:1.6.4-3) unstable; urgency=medium

  * Reload kdump after memory/CPU hotplug. (LP: #1655280)
  * Fix adding crashkernel to zipl.conf when no quotation mark is used.
    (LP: #1790788)

 -- Thadeu Lima de Souza Cascardo <email address hidden> Thu, 07 Feb 2019 09:22:23 -0200

Changed in makedumpfile (Ubuntu Cosmic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for makedumpfile has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Frank Heimes (fheimes) on 2019-05-20
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package makedumpfile - 1:1.6.5-1ubuntu1~18.04.1

---------------
makedumpfile (1:1.6.5-1ubuntu1~18.04.1) bionic; urgency=low

  * Backport back to bionic. (LP: #1655280) (LP: #1790788)

makedumpfile (1:1.6.5-1ubuntu1) disco; urgency=low

  [ Ubuntu Merge-o-Matic ]
  * Merge from Debian unstable. Remaining changes:
    - Bump amd64 crashkernel from 384M-:128M to 512M-:192M.

  [ Thadeu Lima de Souza Cascardo ]
  * Use a different service for vmcore dump. (LP: #1811692)

makedumpfile (1:1.6.5-1) unstable; urgency=medium

  * Update to new upstream version 1.6.5.
  * debian: remove debian/source/local-options
  * [i18n] Move PT debconf translation (Closes: #910465)

makedumpfile (1:1.6.4-3) unstable; urgency=medium

  * Reload kdump after memory/CPU hotplug. (LP: #1655280)
  * Fix adding crashkernel to zipl.conf when no quotation mark is used.
    (LP: #1790788)

 -- Thadeu Lima de Souza Cascardo <email address hidden> Thu, 07 Feb 2019 09:22:23 -0200

Changed in makedumpfile (Ubuntu Bionic):
status: Fix Committed → Fix Released
Frank Heimes (fheimes) on 2019-05-24
Changed in ubuntu-z-systems:
importance: Undecided → High
Frank Heimes (fheimes) wrote :

With the changed focus from xenial to bionic - the xenial entry is now set to Won't Fix.

Frank Heimes (fheimes) on 2019-06-25
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers