Use the kernel default for crashkernel offset

Bug #1741860 reported by bugproxy on 2018-01-08
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
High
Canonical Kernel Team
makedumpfile (Ubuntu)
High
Connor Kuehl
Xenial
High
Connor Kuehl
Bionic
High
Connor Kuehl
Disco
High
Connor Kuehl
Eoan
High
Connor Kuehl

Bug Description

[Impact]

 * The value chosen for ppc64el of 128MB aligns with that of the kernel default. That may change some day, so it would be best to let the kernel decide what value it should use. If the value does change and the kernel is not allowed to choose a value, this may stop the kernel from booting on a production system.

[Test Case]

 Run `cat /etc/default/grub.d/kdump-tools.cfg`

 * Expected result: there is no offset specified at the end of the line (.e.g, @128M).

 * Actual result: For Xenial, Bionic, Disco, and Eoan there are offsets specified.

[Regression Potential]

 * Right now, the offset described in the kdump-tools.cfg aligns with that of what the kernel would select, so since they are the same we would expect no change in operation.

Original bug description follows:
-----------------------------------------

== Comment: #0 - Hari Krishna Bathini <email address hidden> - 2018-01-08 01:06:41 ==
---Problem Description---
A default offset of 128MB is enforced for crashkernel by kdump-tools utility
overriding the kernel default.

While the kernel default offset for crashkernel is also 128MB, that may change
and the right thing to do would be to let the kernel decide on the offset of
crashkernel in the default scenario..

Get rid of "@128M" in kdump-tools.cfg file

Contact Information = <email address hidden>

---uname output---
na

Machine Type = na

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 # cat /etc/default/grub.d/kdump-tools.cfg
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@128M"
---

The offset is specified via kdump-tools where as the kernel may be the right place to
set an offset by default..

Userspace tool common name: kdump-tools

The userspace tool has the following bit modes: 64-bit

Userspace rpm: kdump-tools

Userspace tool obtained from project website: na

*Additional Instructions for <email address hidden>:
-Attach ltrace and strace of userspace application.

== Comment: #3 - MAMATHA INAMDAR <email address hidden> - 2018-01-08 03:05:05 ==
This bug is opened to follow-up other bug based on the comment 19
https://bugzilla.linux.ibm.com/show_bug.cgi?id=152905#c19 (Canonical Launchpad 1676884 )

bugproxy (bugproxy) on 2018-01-08
tags: added: architecture-ppc64le bugnameltc-163190 severity-medium targetmilestone-inin---
Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → makedumpfile (Ubuntu)
Frank Heimes (fheimes) on 2018-01-08
Changed in ubuntu-power-systems:
importance: Undecided → Medium
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
status: New → Triaged
tags: added: ppc64el-kdump triage-g

------- Comment From <email address hidden> 2018-01-22 13:19 EDT-------
(In reply to comment #0)
> ---Problem Description---
> A default offset of 128MB is enforced for crashkernel by kdump-tools utility
> overriding the kernel default.
>
> While the kernel default offset for crashkernel is also 128MB, that may
> change
> and the right thing to do would be to let the kernel decide on the offset of
> crashkernel in the default scenario..
>
> Get rid of "@128M" in kdump-tools.cfg file
>
>
> Contact Information = <email address hidden>
>
> ---uname output---
> na
>
> Machine Type = na
>
> ---Debugger---
> A debugger is not configured
>
> ---Steps to Reproduce---
> # cat /etc/default/grub.d/kdump-tools.cfg
> GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT
> crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:
> 4096M@128M"

While the offset is changed to 128MB in some releases, it is still 32M on some.
Having the default crashkernel offset as 32MB is NOT a good idea. It would leave
the production kernel un-bootable. Please drop the offset of 32M/128M and
let the kernel decide on the offset in default case to ensure we don't end up
in a scenario where production kernel is unbootable. This applies to all
Canonical releases..

Thanks
Hari

tags: added: severity-critical
removed: severity-medium
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-02-28 01:27 EDT-------
*** Bug 163566 has been marked as a duplicate of this bug. ***

------- Comment From <email address hidden> 2018-02-28 01:28 EDT-------
Issue is seen on Ubuntu 17.10:
Ubuntu 17.10 [4.13.0-26-generic #30~lp1741497.1]: Machine fails to boot after installing linux-crashdump package.
I see that the after installing the linux-crashdump package, the kernel cmdline has
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@32M as the crashkernel value set. This is leaving the production kernel unbootable.

Thank you.

Can you verify using makedumpfile + kdump-tools from artful-proposed?

Then, we can push it to -updates.

Thank you very much.
Cascardo.

Frank Heimes (fheimes) on 2018-02-28
Changed in ubuntu-power-systems:
status: Triaged → In Progress
Manoj Iyer (manjo) on 2018-03-05
Changed in makedumpfile (Ubuntu):
importance: Undecided → Medium
assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Canonical Kernel Team (canonical-kernel-team)
Changed in makedumpfile (Ubuntu):
status: New → In Progress

It has already been changed to @128M on bionic. For 18.10, I will drop, and consider backporting it to previous versions if necessary. I will backport the @128M change to xenial and artful as soon as the versions in -proposed move to -updates or are dropped.

Cascardo.

Changed in makedumpfile (Ubuntu):
assignee: Canonical Kernel Team (canonical-kernel-team) → Thadeu Lima de Souza Cascardo (cascardo)
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-08-07 02:01 EDT-------
(In reply to comment #10)
> It has already been changed to @128M on bionic. For 18.10, I will drop, and
> consider backporting it to previous versions if necessary. I will backport
> the @128M change to xenial and artful as soon as the versions in -proposed
> move to -updates or are dropped.
>
> Cascardo.
Sure I will validate and update the bug ASAP..

Andrew Cloke (andrew-cloke) wrote :

Were you able to perform the validation described in comment #5?

Frank Heimes (fheimes) on 2018-11-12
Changed in ubuntu-power-systems:
status: In Progress → Incomplete
Changed in makedumpfile (Ubuntu):
status: In Progress → Incomplete
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2019-04-04 07:18 EDT-------
Any update on this? is this fix is integrated?

Frank Heimes (fheimes) wrote :

Hi Mamatha, well, as you can see from Launchpad comment #6:
https://bugs.launchpad.net/ubuntu/+source/makedumpfile/+bug/1741860/comments/6
we are still waiting for the feedback from IBM about the validation mentioned in comment #5:
https://bugs.launchpad.net/ubuntu/+source/makedumpfile/+bug/1741860/comments/5

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2019-06-24 07:52 EDT-------
Sorry. Couldn't validate this on 18.10 -proposed. Can we have it on
eoan-proposed for validation and SRU to stable release..

Thanks
Hari

Manoj Iyer (manjo) wrote :

Is this issue even reproducible on Disco or Eoan? Looks like there are fixes that could have landed for this issue in 18.10. Can IBM please confirm this is an issue in Disco and Eoan?

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2019-06-24 10:49 EDT-------
Yes. kdump-tools.cfg file still carries the offset of "@128M" in it while the request
here is to drop this so as to fallback to kernel default. Note that the kernel
default right now is 128M but that may change

Manoj Iyer (manjo) on 2019-07-01
Changed in ubuntu-power-systems:
status: Incomplete → Confirmed
Changed in makedumpfile (Ubuntu):
status: Incomplete → Confirmed
importance: Medium → High
Changed in ubuntu-power-systems:
importance: Medium → High
Connor Kuehl (connork) on 2019-07-24
Changed in makedumpfile (Ubuntu Disco):
status: New → In Progress
Changed in makedumpfile (Ubuntu Eoan):
status: Confirmed → In Progress
Changed in makedumpfile (Ubuntu Bionic):
status: New → In Progress
Changed in makedumpfile (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Connor Kuehl (connork)
Changed in makedumpfile (Ubuntu Bionic):
assignee: nobody → Connor Kuehl (connork)
Changed in makedumpfile (Ubuntu Disco):
assignee: nobody → Connor Kuehl (connork)
Changed in makedumpfile (Ubuntu Eoan):
assignee: Thadeu Lima de Souza Cascardo (cascardo) → Connor Kuehl (connork)
Changed in makedumpfile (Ubuntu Disco):
importance: Undecided → High
Changed in makedumpfile (Ubuntu Bionic):
importance: Undecided → High
Changed in makedumpfile (Ubuntu Xenial):
importance: Undecided → High
Connor Kuehl (connork) wrote :

Hi Hari and Mamatha,

I made a test package for kdump-tools with "@128M" removed from the end of kdump-tools.cfg. This test package is for Eoan as requested in Launchpad Bug Comment #9 (although this package is not in the proposed repository, it will have to be installed manually with `dpkg -i kdump-tools_1.6.5-1ubuntu3.1_ppc64el.deb`)

The test package is located here: https://kernel.ubuntu.com/~connor/LP1741860-crashkernel/eoan/

If everything looks okay on your end, I will proceed with SRU'ing this fix to the other Ubuntu releases.

Thanks,

Connor

Frank Heimes (fheimes) on 2019-07-25
Changed in ubuntu-power-systems:
status: Confirmed → In Progress
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2019-07-25 02:21 EDT-------
(In reply to comment #20)
[...]
> If everything looks okay on your end, I will proceed with SRU'ing this fix
> to the other Ubuntu releases.

The change works as intended. Please proceed with SRU'ing.

tags: added: targetmilestone-inin1804
removed: targetmilestone-inin---
Connor Kuehl (connork) wrote :

Attaching .debdiff to this comment for SRU sponsorship consideration into Xenial.

Connor Kuehl (connork) wrote :

Attaching .debdiff to this comment for SRU sponsorship consideration into Bionic.

Connor Kuehl (connork) wrote :

Attaching .debdiff to this comment for SRU sponsorship consideration into Disco.

Connor Kuehl (connork) wrote :

Attaching .debdiff to this comment for SRU sponsorship consideration into Eoan.

description: updated

The attachment "makedumpfile_1.6.3-2_16.04.1ubuntu1.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package makedumpfile - 1:1.6.6-2ubuntu1

---------------
makedumpfile (1:1.6.6-2ubuntu1) eoan; urgency=medium

  [ Thadeu Lima de Souza Cascardo ]
  * Merge from Debian unstable. Remaining changes:
    - Bump amd64 crashkernel from 384M-:128M to 512M-:192M.
  * Add kdump retry/delay mechanism when dumping over network (LP: #1681909)
  * Allow proper reload of kdump after multiple hotplug events. (LP: #1828596)

  [ Connor Kuehl ]
  * Let the kernel decide the crashkernel offset for ppc64el (LP: #1741860)

makedumpfile (1:1.6.6-2) unstable; urgency=medium

  [ Guilherme G. Piccoli ]
  * Add kdump retry/delay mechanism when dumping over network

  [ Thadeu Lima de Souza Cascardo ]
  * Use a different service for vmcore dump.
  * Use maxcpus instead of nr_cpus on ppc64el.
  * Reload kdump when CPU is brought online.
  * Allow proper reload of kdump after multiple hotplug events.

makedumpfile (1:1.6.6-1) unstable; urgency=medium

  * Update to new upstream version 1.6.6.

 -- Thadeu Lima de Souza Cascardo <email address hidden> Tue, 06 Aug 2019 12:18:15 -0300

Changed in makedumpfile (Ubuntu Eoan):
status: In Progress → Fix Released

Hello bugproxy, or anyone else affected,

Accepted makedumpfile into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in makedumpfile (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco
Andy Whitcroft (apw) wrote :

Hello bugproxy, or anyone else affected,

Accepted makedumpfile into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in makedumpfile (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic

------- Comment From <email address hidden> 2019-08-29 07:43 EDT-------
I verified that the offset (looking at /etc/default/grub.d/kdump-tools.cfg file) is removed
on -bionic with kdump-tools package version 1.6.5-1ubuntu1~18.04.2

tags: added: verification-done-bionic
removed: verification-needed-bionic

All autopkgtests for the newly accepted makedumpfile (1:1.6.5-1ubuntu1.1) for disco have finished running.
The following regressions have been reported in tests triggered by the package:

makedumpfile/1:1.6.5-1ubuntu1.1 (s390x, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/disco/update_excuses.html#makedumpfile

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

------- Comment From <email address hidden> 2019-08-30 02:18 EDT-------
Resolved with makedumpfile/kdump-tools version 1.6.5-1ubuntu1.1 on Disco

bugproxy (bugproxy) on 2019-08-30
tags: added: verification-done verification-done-disco
removed: verification-needed verification-needed-disco

The version of makedumpfile in the proposed pocket of Bionic that was purported to fix this bug report has been removed because one or more bugs that were to be fixed by the upload have failed verification and been in this state for more than 10 days.

Changed in makedumpfile (Ubuntu Bionic):
status: Fix Committed → Won't Fix
Andrew Cloke (andrew-cloke) wrote :

Following f2f discussions, next step is to re-upload new version of makedumpfile.

On 9/30/19 7:47 AM, Andrew Cloke wrote:
> Following f2f discussions, next step is to re-upload new version of
> makedumpfile.
>

Pushing on this one too.

Hello bugproxy, or anyone else affected,

Accepted makedumpfile into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-disco
removed: verification-done verification-done-disco
Changed in makedumpfile (Ubuntu Bionic):
status: Won't Fix → Fix Committed
tags: added: verification-needed-bionic
removed: verification-done-bionic
Andy Whitcroft (apw) wrote :

Hello bugproxy, or anyone else affected,

Accepted makedumpfile into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1~18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted makedumpfile (1:1.6.5-1ubuntu1~18.04.3) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

makedumpfile/1:1.6.5-1ubuntu1~18.04.3 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#makedumpfile

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

------- Comment From <email address hidden> 2019-10-25 06:56 EDT-------
On disco, with kdump-tools/makedumpfile package version 1:1.6.5-1ubuntu1.3
kernel offset is being used by default as expected.

On bionic, with kdump-tools/makedumpfile package version 1:1.6.5-1ubuntu1~18.04.3
kernel offset is being used by default as expected.

tags: added: verification-done verification-done-bionic verification-done-disco
removed: verification-needed verification-needed-bionic verification-needed-disco

All autopkgtests for the newly accepted makedumpfile (1:1.6.5-1ubuntu1.3) for disco have finished running.
The following regressions have been reported in tests triggered by the package:

makedumpfile/1:1.6.5-1ubuntu1.3 (s390x, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/disco/update_excuses.html#makedumpfile

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package makedumpfile - 1:1.6.5-1ubuntu1.3

---------------
makedumpfile (1:1.6.5-1ubuntu1.3) disco; urgency=medium

  [ Guilherme G. Piccoli ]
  * Add kdump retry/delay mechanism when dumping over network (LP: #1681909)

  [ Thadeu Lima de Souza Cascardo ]
  * Use maxcpus instead of nr_cpus on ppc64el. (LP: #1828597)
  * ppc64: increase MAX_PHYSMEM_BITS to 2PB (LP: #1841288)

  [ Connor Kuehl ]
  * Let the kernel decide the crashkernel offset for ppc64el (LP: #1741860)

 -- Thadeu Lima de Souza Cascardo <email address hidden> Wed, 09 Oct 2019 15:33:57 -0300

Changed in makedumpfile (Ubuntu Disco):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for makedumpfile has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package makedumpfile - 1:1.6.5-1ubuntu1~18.04.3

---------------
makedumpfile (1:1.6.5-1ubuntu1~18.04.3) bionic; urgency=medium

  [ Guilherme G. Piccoli ]
  * Add kdump retry/delay mechanism when dumping over network (LP: #1681909)

  [ Thadeu Lima de Souza Cascardo ]
  * Use maxcpus instead of nr_cpus on ppc64el. (LP: #1828597)
  * ppc64: increase MAX_PHYSMEM_BITS to 2PB (LP: #1841288)

  [ Connor Kuehl ]
  * Let the kernel decide the crashkernel offset for ppc64el (LP: #1741860)

 -- Thadeu Lima de Souza Cascardo <email address hidden> Wed, 09 Oct 2019 15:38:08 -0300

Changed in makedumpfile (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers