[20.10 FEAT] Increase the crashkernel setting if the root volume is luks2-encrypted

[bugproxy]

Description:
In case the volume containing the root filesystem is encrypted using LUKS2 the memory used while unlocking the volume may exceed the size allocated to the kdump kernel. This will lead to a failure while processing kdump and the dump file will not be stored. Unfortunately, this condition may not be detected by a client before a problem occurs.
The request is to have the kdump package installation script check for LUKS2 encryption (more precisely for Argon2i PBKDF, which is the root cause of the high memory usage). If the condition is met, the installation procedure should increase the crashkernel parameter to a higher value (>=512M)or issue a warning, if the system memory is insufficient to reserve enough crashkernel memory.

Business Case:
Pervasive Encryption and Secure Execution require encryption of the filesystems in order to keep customer data secure at all times. With the increasing usage of these technologies, the number of kdump will rise too, typically at inconvenient times, when the kdump is triggered due to a real customer issue.
With the suggested change, the number of customer complaints and effort to handle them will be reduced.

[Frank Heimes (fheimes)]

After discussing this it turned out that the crashkernel increase is not only be needed for 20.10, but also for 20.04 (because both support secure execution) - hence I added the target series for focal and gorilla.

The question is whether it should really be checked if LUKS2 encryption (or even Argon2i PBKDF) is in use, or if instead the setting should generally be increased to 512M?

[Dimitri John Ledkov (xnox)]

I am slightly concerned why Argon2i is mentioned, and if the drives are not correctly setup for the size of the machine.

Normally, the encrypted drives are created on the same target machine spec as the intended runtime, benchmark is executed and appropriate Argon2i parameters are picked for the size of the machine and used. If the drives are encrypted on a big machine, and then later attached / operated with lower CPU/Storage (RAM), then Argon2i settings on the drive might be inappropriate, and should be reduced. One can manually specify the benchmark rounds when encrypting the drives (usually this is done on the big LPAR, when preparing drives to be used by smaller LPARs).

After the initial unlocking of the drives and memory pressure during that time, there should not be elevated RAM requirements from the kernel, or for its crashdumps, unless one is dumping the kernel whilst unlocking the drives.

On the other hand, we have noticed that the new kernel is using more ram across all architectures, and maybe simply any crashdump (with or without luks2 argon2i encryption) are bigger than before and the baseline crashdump setting should be bumped up anyway.

[Thadeu Lima de Souza Cascardo (cascardo)]

Setting as Invalid for linux, as this is not a linux bug. Marking as Confirmed for makedumpfile, where there is a chance this might be fixed.

However, I disagree with checking for a specific module on a specific architecture to decide crashkernel size.

crashkernel size is a suggestion. Users are supposed to test that kdump works with the size they are supposed to configure. We may need to document that.

[Frank Heimes (fheimes)]

Hi Thadeu, I was also a bit concerned about the check and I think that bumping it generally to for example 512 (in addition with some doc) would be a feasible compromise.
According to comment #2 I assume that Dimitri would agree on that, too.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-06-22 06:00 EDT-------
@Canonical: Is a final decision made by Dev, how to implement this request? Many thx in advance..

[Dimitri John Ledkov (xnox)]

@fheimes
No, in comment #2 I did not agree to bump the parameter.

@heinz-werner_seeck

My previous comment was not responded to. LUKS2 and Argon2i have no effect on steady state RAM usage, and thus should not affect crashkernel memory requirements significantly. Normally, Argon2i decryption has high memory pressure only once during unlocking of the drive, but not during the steady state operation.

Please explain how the new crashkernel size has been calculated, and why is it so much larger than before?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-06-22 07:57 EDT-------
Using an encrypted root volume will more or less always fail, as the default crashkernel size is too small (and has always been) for decryption of a LUKS volume using Argon2i. I'd assume this is true for all architectures. The check should not be done based on the architecture but based on whether the volume containing the root filesystem is encrypted or not.

[Dimitri John Ledkov (xnox)]

encrypted volume of the system being dump, or the encrypted volume which is the target where to store the dump? (and thus more memory is needed to unlock the drive?)

[Thadeu Lima de Souza Cascardo (cascardo)]

@xnox, the unlock must happen for the root filesystem during dump, independently from where it's going to dump to, though default configuration is into local /var/crash/.

And even if the memory parameter is picked up depending on the system size and crashkernel does that too, crashkernel takes memory out of the system that cannot be used during production.

My opinion is that even we could tell how much memory would be necessary for dumping, this is a matter of policy. A user with 2GiB can easily unlock an Argon2i encrypted root filesystem with memory parameter close to 1GB during boot. But would such user would be surprised to have now only half of that memory available after boot because the other half was reserved? Maybe the user only needed 1GiB for production and added the extra 1GiB for crashkernel. Maybe the user didn't expect that.

Fortunately, the option is configurable. Unfortunately, we don't have an easy way to advise the user how much memory will be necessary and it can even change when system configuration changes. And I wouldn't still just change the parameter without telling the user.

So, my suggestion here is that we try to improve the defaults, consider changing s390x to match other arches that change that depending on system memory size. In the future, we could have some tool that advise a different setting. But right now, users are supposed to test that the current setting works for them, change it to a working setting, maybe consider changing their VM size because of that.

So, I am closing this issue as "Won't Fix".
Cascardo.

[Frank Heimes (fheimes)]

I agree that the crashkernel size has especially an impact on smaller system.
And right now the size is with 128MB obviously mainly optimized for smaller system, but does not really take the needs of bigger ones into account.
However, this is done in a better way on other platforms, where we have a setting that is based on ranges, which provides more flexibility, I think.

So I would suggest to introduce such a staged, range-based approach for s390x too, like "crashkernel=384M-1G:128M,1G-3G:256M,3G-:512M"
The crashkernel setting is of course configurable, but hitting slightly better values by default would improve the user experience and reduce the need for manual adjustments - instead of having to adjust it 'all the time' (which of course a bit overdrawn).

And btw. I would estimate that an avg. s390x system has nowadys about 4GB RAM; and with the new installer I guess we even have a slightly higher minimal req. for RAM anyway (and not much people will install with a higher RAM footprint and reduce it post-install - and I'm also not sure if it's possible to install on a system with only 384MB RAM these days ...).

So could such a range-based setting become a valid compromise?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-07-06 13:42 EDT-------
Note if you want to use LUKS2 with protected key dm-crypt, we do not recommend to use argon type PBKDF but to rather use PKKDF2, because passphrased do not contribute to the security of the secure key in the key slot.
Yet I do not know whether it is reasonable to protect dump volumes with protectedkeys due to the dependency ob access to crypto adapters.

[Thadeu Lima de Souza Cascardo (cascardo)]

I am going to work out on a change to the default crashkernel value on s390x to match what we do for ppc64el. That will improve the situation on default installations.

[Thadeu Lima de Souza Cascardo (cascardo)]

Attached is a fix for s390x that is also built at my ppa:cascardo/kdump2.

[Thadeu Lima de Souza Cascardo (cascardo)]

So, of course the patch is useless, as s390x uses zipl. I will work on upgrade paths during this week, so we can allow upgrades to have the new setting when the old setting is kept as is.

[Thadeu Lima de Souza Cascardo (cascardo)]

I pushed a version for focal on my ppa. It is based on the latest version found on groovy, so there should be other changes besides the one for setting a better default of crashkernel on /etc/zipl.conf.

It's at ppa:cascardo/kdump2. Note that if the user has changed the value from the previous default, it will be kept as is. This is done per entry.

Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-09-24 04:57 EDT-------
Quick question, is my understanding correct, that if I upgrade the package the crashkernel size should be increased in zipl.conf?

[Thadeu Lima de Souza Cascardo (cascardo)]

If the user has not changed the value to something else, the size should be increased after upgrade. If the user has changed, we keep the user's value.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-10-01 09:43 EDT-------
Took me a bit longer than expected, as I ran into some other unrelated issue. Unfortunately I was not successful. What I did on a pristine Ubuntu 20.04.1 install with a luks-encrypted setup was:

$ add-apt-repository ppa:cascardo/kdump2
$ apt update
$ apt install kexec-tools
$ apt install kdump-tools

The initrd was rebuild and the zipl was run, however zipl.conf wasn't updated with a new crashkernel value and after rebooting it was still the old value of 196 in the kernel command line.

Here's the output of dpkg -l for kexec-tools and kdump-tools

ii kexec-tools 1:2.0.18-1ubuntu1 s390x tools to support fast kexec r
ii kdump-tools 1:1.6.7-4ubuntu1+cascardo2 s390x scripts and tools fo

Here's my cryptab content:
vda6_crypt UUID=3b1c23fa-0cc4-4ba1-b47e-974f79db1b5c none luks,discard

fstab:
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/mapper/vgubuntu-root / ext4 errors=remount-ro 0 1
# /boot was on /dev/vda5 during installation
UUID=44950dee-686a-4946-b64e-8b54976d254a /boot ext4 defaults 0 2
/dev/mapper/vgubuntu-swap_1 none swap sw 0 0

lsblk output:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 6G 0 disk
??vda1 252:1 0 512M 0 part
??vda2 252:2 0 1K 0 part
??vda5 252:5 0 731M 0 part /boot
??vda6 252:6 0 4.8G 0 part
??vda6_crypt 253:0 0 4.8G 0 crypt
??vgubuntu-root 253:1 0 3.8G 0 lvm /
??vgubuntu-swap_1 253:2 0 976M 0 lvm [SWAP]

Maybe I did something wrong. Let me know.

[Thadeu Lima de Souza Cascardo (cascardo)]

Where does that 196 value come from? Was it present on zipl.conf after a pristine install? I wouldn't expect that on a pristine Ubuntu 20.04.1 install, so we would need to investigate where that value comes from. I know s390-tools itself used to add that crashkernel value there, but that only worked for upgrades from pre-16.04 to 16.04, AFAIU. So, it would not apply on such recent installs. Was it manually added there?

Thanks.
Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-10-01 12:49 EDT-------
The zipl.conf content was there right after the installation, before even installing the kdump tools. Hmm ... but I was using the legacy di installer from http://ports.ubuntu.com/dists/focal/main/installer-s390x/current/legacy-images/generic/. Are you saying that with the new installer there should be no crashkernel in the command line?
Can I force the command line to be changed by removing the crashkernel parameter and re-running kdump-tools installation?

[Thadeu Lima de Souza Cascardo (cascardo)]

There should be no difference between using di or the new installer, unless there is something the installer is doing that is causing this. If you remove the crashkernel line from zipl.conf and run dpkg-reconfigure kdump-tools, it should add the new configuration value. Otherwise, it will keep whatever other value is there. The only exception is when you are upgrading from an older version of kdump-tools and it finds the value that old kdump-tools would put in there.

There is a chance I am misreading the code from s390-tools package (the one with zipl). It used to add the crashkernel parameter there, but only when upgrading from versions pre-Xenial. I even tested that on a new system, when removing and reinstalling s390-tools, there was no crashkernel line in there.

We are trying to reproduce this locally and will investigate what may be causing this. One other option would be to also replace that specific value. I would rather not do it unless really necessary, because otherwise we might replace a value put there by a user.

Cascardo.

[Thadeu Lima de Souza Cascardo (cascardo)]

So, after some investigation, it looks like the zipl-installer d-i component is responsible for creating /etc/zipl.conf with the default crashkernel parameter with value 196M.

With that into consideration, one wonders how kdump-tools would previously set its default value of "384M-:128M" in there. In any case, 196M would work better for most users anyway. But I still wonder how this was not reported before. Maybe other installation methods or upgrades would lead to a different situation.

But as we understand the current situation now, and it would be too late to fix d-i at this point, I believe it is acceptable to replace that value when installing or upgrading kdump-tools. I will work on a new test version of the package.

Thanks.
Cascardo.

[Thadeu Lima de Souza Cascardo (cascardo)]

So, it looks like the latest version from the same ppa:cascardo/kdump2 should be working much better now, handling that specific situation from a new install containing crashkernel=196M. I tested both upgrade paths from older kdump-tools or new installs, and they are working fine starting with either no crashkernel parameter, that specific value of 196M or a different value, like 320M. In the first and second cases, we ended up with the new default value. On the third case, the user-configured value was kept.

Version is 1:1.6.7-4ubuntu1+cascardo6.

Regards.
Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-10-14 07:51 EDT-------
Sorry for not responfing earlier. After installing a new system I can see that the update happening:

$ cat /proc/cmdline
root=/dev/mapper/vgubuntu-root crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M

However it still turns out that - if using an encrypted volume - it's not possible to unlock the volume from the kdump kernel for a ~2GB VM as the 320MB are still not sufficient.

Bumping the VM memory to 4GB will allow the kdump kernel to unlock the volume.

I think that either 512M should be the lowest value used for the crashkernel, or at least a warning should be emitted, alerting the system administrator that for a < 4GB machine kdump is not going to work with an encrypted root/dump volume.

[Thadeu Lima de Souza Cascardo (cascardo)]

I think 512M for a 2GB system is a little too much to reserve. Can you double check the details of that LUKS partition? Was it created during an installation with a 2GB system? Or did you reduce the VM size after installation?

Can you send the output of cryptsetup luksDump /dev/XXX?

Thanks.
Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-10-29 09:59 EDT-------
With 2G, the crashkernel reservation will be 320K, which will not suffice to unlock the LUKS volume when processing kdump. The installation was done using 2G.
Changing the VM size to 4+G will reserve 512K, and with this size kdump is working. I agree that 512K out of 2G is probably on the brink of too much.

As I said, a warning could be emitted that crashkernel < 512k will not allow to unlock a LUKS volume (at least if argon2i is used as PBKDF).

Below is the luksDump output

LUKS header information
Version: 2
Epoch: 4
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: ac7595b4-0b0f-45ef-b745-c6f571876158
Label: (no label)
Subsystem: (no subsystem)
Flags: (no flags)

Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 512 [bytes]

Keyslots:
0: luks2
Key: 512 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: argon2i
Time cost: 4
Memory: 270246
Threads: 1
Salt: 38 7f 60 45 c6 57 26 50 72 7a a7 e3 18 59 24 24
58 7a 9b 95 dd 87 3c 19 43 88 1e 5b 98 35 fd 3d
AF stripes: 4000
AF hash: sha256
Area offset:32768 [bytes]
Area length:258048 [bytes]
Digest ID: 0
1: luks2
Key: 512 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: argon2i
Time cost: 4
Memory: 235479
Threads: 1
Salt: 02 b8 0a f3 7e 0b 1c 23 77 9b 0e b0 19 c8 69 b4
bb 0c 3c 1e fd f9 24 7d 8a bf 6e 64 3b 17 79 31
AF stripes: 4000
AF hash: sha256
Area offset:290816 [bytes]
Area length:258048 [bytes]
Digest ID: 0
Tokens:
Digests:
0: pbkdf2
Hash: sha256
Iterations: 142469
Salt: 62 73 02 e9 44 09 b0 44 0c 87 33 d4 8f 68 5e f7
3c f6 a5 68 5a 8f b6 2d a6 35 9c 02 9c ba c0 76
Digest: df 1a 9f 29 0e 2f 25 87 5f 11 11 c7 01 85 fd 2e
e6 6a 98 65 bd b5 54 77 29 2d c5 b3 12 48 67 a1

[Dimitri John Ledkov (xnox)]

Are these installs with zkey/paes or without?

Because, in Ubuntu, when zkey was introduced I have reverted the s390-tools upstream change to lower the argon2i settings. Due to my lack of understanding of the security features there. And later, we have made similar choices for TPM backed encryption elsewhere on the Ubuntu platofrm. Thus for example, for zkey/paes Imho argon2i should be used, but with a lower benchmark criteria capped at 200ms trial, instead of the builtin default of 2000ms trial. Otherwise, the RAM requirements to dump back onto paes/zkey encrypted volume will ever grow with the machine RAM size.

To test this out, install the system with little ram (ie. 1GB), then deactive, bump ram to 16GB. And I suspect that kdump to zkey/paes volume will then just work.

For non-protected (no zkey/paes) encryption with just passthrases, the argon2i is the only protection we can provide, and yes it will always run install time benchmark and will always use ever increasing amount of ram to perform unlock. If we want to always have kdump working onto non-protected zkey/paes drives, we must introspect the luks volume argon2i benchmark details and base the reserved RAM off that. Because in theory, some future cryptsetup might change the benchmark, and thus result in different amount of RAM requirement to unlock the drive.

[Dimitri John Ledkov (xnox)]

https://github.com/snapcore/secboot/pull/127

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-11-13 04:03 EDT-------
This is a setup without crypto cards (Secure Execution guests don't support passthrough yet). I think that we can live with the new kdump default settings, as I wouldn't want to suggest putting half of the memory away for kdump on small VMs. The recommendation should be to use pbkdf2 for machines < 4GB without crypto cards. Ideally, the tools would emit a warning when configuring the command line. Alternatively, this could be documented in the Installation Guide.

[Dimitri John Ledkov (xnox)]

@mihajlov

that is a good point. When there are less than 4GB of RAM, the default cryptsetup benchmark should probably be capped at 128M of RAM, rather than 1-2GB. Will ponder about that.

[Frank Heimes (fheimes)]

This ticket is now discussed and worked on in a broader scope and is therefore dependent on:
→ https://salsa.debian.org/debian/kdump-tools/-/merge_requests/5
→ https://bugs.debian.org/983486

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-05-07 04:07 EDT-------
@Canonical, will this topic be addressed at least with 21.10? Many thx

[Thadeu Lima de Souza Cascardo (cascardo)]

Aside from the new default crashkernel (or rather, whether to keep the 2G-4G:320M option or simply not setup crashkernel for the 2G-4G case), how does the new version of kdump-tools on ppa:cascardo/kdump2 work?

To give you an answer, it does upgrade the crashkernel value on zipl.conf to a new default whatever value is configured on the current installed /etc/zipl.conf, including if it's not configured by default.

However, in case we change the value in the future to a different one, ucf will be able to simply do the update in case the user has kept the old default value and has not done any other change on /etc/zipl.conf. In this latter case, the user will be asked whether to keep the old version or the new version.

So, basically, this allows us to update the default in the future without keeping track of which versions supported which defaults.

The only drawback is that the user might be prompted a little more often about changes in /etc/zipl.conf.

Regards.
Cascardo.

[Thadeu Lima de Souza Cascardo (cascardo)]

This is a debdiff for the proposed solution for kdump-tools, based on latest version on impish.

[Frank Heimes (fheimes)]

@cascardo 'ppa:cascardo/kdump2' was okay for me, when I tried it last time - however, not sure if s/o from IBM tried it, too.

I think that the approach that you've described is totally valid and fine.
And the debdiff looks reasonable and incl. the 2G-4G case).

Since situations where dumps are needed are rare anyway, and people are usually careful and mostly follow manual steps any, I do not see an issue at all with a few prompts with questions on zipl.conf.

So definitely a +1 from me.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-12-06 04:21 EDT-------
Coming back to the default for 2-4G, which will not allow the decryption of the root volume, see comments above. Will this be addressed via a warning message during install/update or at least some documentation?

[Thadeu Lima de Souza Cascardo (cascardo)]

There is already documentation about it. For example, from https://ubuntu.com/server/docs/kernel-crash-dump

"If the dump does not work due to OOM (Out Of Memory) error, then try increasing the amount of reserved memory by editing /etc/default/grub.d/kdump-tools.cfg. For example, to reserve 512 megabytes"

There are multiple reasons why an OOM might happen during kdump, and there are no plans on detecting them on a case-by-case, so we won't do it for this specific case of root volume decryption.

If we use a very large default that might work on almost all cases, it will take much more memory than is necessary for many users. On the other hand, using a default that won't work for most users is not any better.

So, noticing that any default will affect all users, including those that do not enable root volume encryption or do not use argon2i; we have 3 options for the new default:

1) Keep 2G-4G:384M; some users won't be able to collect kdump unless they reconfigure the crashkernel option, but most users will, by reserving not more than 384M from a 2G VM.
2) Raise it to 2G-4G: 512M, it reserves more memory, and users with 2G to 4G VMs using argon2i for root volume encryption might be able to collect kdump for the sake of reserving more memory by default for users that don't use root volume encryption.
3) Remove the 2G-4G section of crashkernel: users with VMs smaller than 4G won't be able to collect kdump unless they configure and test a crashkernel configuration by themselves.

I still think the first option is the better one.

Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-12-06 06:33 EDT-------
Thanks for the pointer. I am perfectly fine with option 1. It would still help, if the documentation would call out that for less than 4G together with an encrypted root filesystem (using the LUKS2 defaults) kdump will not work.

[Frank Heimes (fheimes)]

So we have a generic 'Kernel Crash Dump' section in the Ubuntu Server Guide:
https://ubuntu.com/server/docs/kernel-crash-dump
that I may expand with a specific note about this situation, once it's rolled out.
I agree, that would make sense.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Fix for groovy" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Thadeu Lima de Souza Cascardo (cascardo)]

The build failed on jammy because of a new shellcheck warning, which was fixed in Debian already. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1002270.

I have uploaded a merge to a PPA, where it built just fine. I am attaching the debdiff for that.

Cascardo.

[Launchpad Janitor (janitor)]

This bug was fixed in the package kdump-tools - 1:1.6.10ubuntu1

---------------
kdump-tools (1:1.6.10ubuntu1) jammy; urgency=low

  * Merge from Debian unstable. Remaining changes:
    - debian/control/tests: sleep while waiting for systemd service.
    - Bump amd64 crashkernel from 384M-:128M to 512M-:192M.
    - Add a systemd-resolved service dependency in order kdump-tools is able
      to resolve DNS when in kdump boot.
    - Update default s390x crashkernel.
      - Install the updated zipl.conf with ucf, so users will be able to
        decide whether to pick any crashkernel changes.

kdump-tools (1:1.6.10) unstable; urgency=medium

  * kdump-config: Fix FTBFS due to new shellcheck warning (Closes: #1002270).

 -- dann frazier <email address hidden> Thu, 24 Mar 2022 11:42:57 -0600

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2022-03-30 09:22 EDT-------
With the bug being fixed (in kdump-tools - 1:1.6.10ubuntu1) we can close this item.
Changing IBM bugzilla status to: ==> CLOSED