makedumpfile falls back to cp on focal with 5.11 kernel

[Ioanna Alifieraki]

[IMPACT]

When running Focal with the hwe 5.11 kernel makedumpfile is not working as it should.
Instead of compressing the dump makedumpfile falls back to cp.
This can create problems on systems with very large RAM and result into exhausting all storage on the host.

[FIX]

This can be resolved with upstream commit f1197543d1c4 ([PATCH] Retrieve MAX_PHYSMEM_BITS from vmcoreinfo).
The reason for falling back to cp is that without this patch makedumpfile calculates wrongly the memory layout.

This happens after the kernel commit 1d50e5d0c505 ("crash_core, vmcoreinfo: Append 'MAX_PHYSMEM_BITS'to vmcoreinfo").
With this commit applied, reading MAX_PHYSMEM_BITS from vmcoreinfo instead of inferring lead to correctly detecting the memory layout and creating the dump file properly.

[TEST CASE]

For testing a machine running focal with the 5.11 kernel is required.
Then trigger a crash.
On the console we can see that makedumpfile fails and `falls back to cp`.
In /var/crash we can see that no dmesg file is created and instead of the dump file
the uncompressed vmcoreinfo is created.

[WHERE PROBLEMS COULD OCCUR]

Any regression potential would involve the crash dump not being created after a crash in case kdump is configured.

[OTHER]

Only Focal with 5.11 kernel is affected.

[Ioanna Alifieraki (joalif)]

Debdiff for Focal.

[Thadeu Lima de Souza Cascardo (cascardo)]

Hey, @joalif.

Has this been tested with focal 5.4 kernel too?

Thanks.
Cascardo.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1942784_focal.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Ioanna Alifieraki (joalif)]

@cascardo,

Good point, just tested it with 5.4 kernel and works fine with it too.

Thanks,
Jo

[Mauricio Faria de Oliveira (mfo)]

Hey Jo,

Thanks for the debdiff!

...

So, it looks like the 2 commits for dmesg/printk are actually LP#1921403 that you handled previously?

For better tracking/documentation purposes, the right thing to do would be to keep these 2 patches
associated with that bug (well, it requires an SRU template now, but you already have it on here.)

Thus I added a Focal task there, and it can be properly referenced in the changelog.

...

And the first commit, for uts_namespace.name should go into Hirsute too, right? I see it's not in
the source package. Since we're adding it back to Focal, it should get into the later releases too.

Adding a Hirsute task here. Could you please provide a debdiff for Hirsute as well?

...

BTW, the debdiff has a `makedumpdife` typo on changelog (that was funny. :)

And despite the patches being DEP3 tagged as 'upstream', at least one of them
needs changes to apply (uts_namespace.name, hunk 5), so it should be tagged
'backport' (or just leave the first, optional field out.)

...

And last but not least, since the patches are architecture independent,
would you mind testing on the supported architectures? I know it seems
overkill per the changes on these patches, but we're better safe than
sorry, particularly on the kernel crash dump path.

If some arch isn't available on canonistack, please see some old notes [1]
that I used to get them running in QEMU to verify installer changes
(s390x emulation was surprisingly fast, and arm64 worked well enough;
I guess it should be ok to skip armhf, and of course, no more i386 on F/H.)

Uh, on both kernels :/ But that should be the least of the suggested
work steps, once the VM is already installed w/ the GA kernel. :)

...

Thanks again!
Mauricio

[1] https://pastebin.ubuntu.com/p/YXBF9dFFp3/

[Ioanna Alifieraki (joalif)]

New debdiff for focal.

[Ioanna Alifieraki (joalif)]

Tested new debdiff on Focal with 5.4 and 5.11 kernels, on amd64, arm64, ppc64el and s390x and
works as expected.
Making Hirsute 'Invalid' as all patches in new debdiff are already in H.

[Mauricio Faria de Oliveira (mfo)]

Hey Jo,

Thanks for finding out the real patch/fix,
and testing on the supported architectures!

The debdiff looks good; I'll review/sponsor later!

[Mauricio Faria de Oliveira (mfo)]

Reviewed and uploaded to Focal. Thanks @joalif!

[Brian Murray (brian-murray)]

Hello Ioanna, or anyone else affected,

Accepted makedumpfile into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.7-1ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ioanna Alifieraki (joalif)]

VERIFICATION ON FOCAL

Installed makedumpfile from proposed.
Testing on a Focal machine running 5.11 kernel.

# uname -rv
5.11.0-40-generic #44~20.04.1-Ubuntu SMP Wed Oct 20 19:04:34 UTC 2021

Trigger crash:
# echo 1 > /proc/sys/kernel/sysrq
# echo c > /proc/sysrq-trigger

From console output :

[ 57.418037] kdump-tools[548]: Starting kdump-tools:
[ 57.419099] kdump-tools[554]: * running makedumpfile -c -d 31 /proc/vmcore /var/crash/202110271414/dump-incomplete
Copying data : [100.0 %] / eta: 0s
[ 61.481957] kdump-tools[572]: The kernel version is not supported.
[ 61.494134] kdump-tools[572]: The makedumpfile operation may be incomplete.
[ 61.501242] kdump-tools[572]: The dumpfile is saved to /var/crash/202110271414/dump-incomplete.
[ 61.502711] kdump-tools[572]: makedumpfile Completed.
[ 61.503885] kdump-tools[554]: * kdump-tools: saved vmcore in /var/crash/202110271414
[ 61.851266] kdump-tools[554]: * running makedumpfile --dump-dmesg /proc/vmcore /var/crash/202110271414/dmesg.202110271414
[ 61.867626] kdump-tools[576]: The kernel version is not supported.
[ 61.871305] kdump-tools[576]: The makedumpfile operation may be incomplete.
[ 61.877902] kdump-tools[576]: The dmesg log is saved to /var/crash/202110271414/dmesg.202110271414.
[ 61.881797] kdump-tools[576]: makedumpfile Completed.
[ 61.885645] kdump-tools[554]: * kdump-tools: saved dmesg content in /var/crash/202110271414
[ 61.989595] kdump-tools[579]: Wed, 27 Oct 2021 14:14:21 +0000
[ 62.123243] kdump-tools[580]: Rebooting.
[ 62.561341] reboot: Restarting system

Makedumpfile compress the dump and does not fall back to cp.

[Launchpad Janitor (janitor)]

This bug was fixed in the package makedumpfile - 1:1.6.7-1ubuntu2.3

---------------
makedumpfile (1:1.6.7-1ubuntu2.3) focal; urgency=medium

  * makedumpfile falls back to cp with the 5.11 kernel (LP: #1942784)
    -d/p/0008-PATCH-Retrieve-MAX_PHYSMEM_BITS-from-vmcoreinfo.patch
  * makedumpfile doesn't create dmesg file on 5.10+ kernels (LP: #1921403)
    -d/p/0009-PATCH-1-2-printk-add-support-for-lockless-ringbuffer.patch
    -d/p/0010-PATCH-2-2-printk-use-committed-finalized-state-value.patch

 -- Ioanna Alifieraki <email address hidden> Mon, 18 Oct 2021 16:43:56 +0000

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for makedumpfile has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.