128M is not enough for kdump on s390 LPARs

[bugproxy]

== Comment: #0 - Michael Holzheu <email address hidden> - 2016-03-31 10:59:26 ==
With the current Ubuntu default setting "crashkernel=128M" kdump on LPARs crashes with out-of-memory (see attachment "dmesg_lpar_out_of_mem_128M.txt").

On z/VM guests 128M seems to be sufficient.

One reason on our test LPAR is that a lot of devices are attached (see attachment "lscss_lpar.txt") which are not required for kdump but consume a lot of memory because the s390 CIO layer allocates data structures in the kernel for those devices.

We can disable the devices by using the "cio_ignore=" kernel parameter in "/etc/default/kdump-tools". For example, on our LPAR that uses DASD 0.0.e934 for /var/crash, we added the following line to disable the devices:

KDUMP_CMDLINE_APPEND="irqpoll maxcpus=1 cio_ignore=all,!condev,!0.0.e934"

For more information on the "cio_ignore=" kernel parameter see:
https://github.com/torvalds/linux/blob/master/Documentation/s390/CommonIO

Even with "cio_ignore=" we still get out-of-memory with "crashkernel=128M".

With "crashkernel=196M" and "cio_ignore=" we are able to create a dump on our LPAR. We currently do not know why kdump with "cio_ignore=" on LPAR consumes more memory than on z/VM guests.

== Comment: #1 - Michael Holzheu <email address hidden> - 2016-03-31 11:03:15 ==
Kernel messages of kdump out-of-memory crash on LPAR with many devices without cio_ignore parameter and 128M crashkernel memory.

== Comment: #2 - Michael Holzheu <email address hidden> - 2016-03-31 11:04:10 ==
Output of lscss showing all attached (not online) devices on the LPAR.

== Comment: #3 - Michael Holzheu <email address hidden> - 2016-03-31 11:07:35 ==
To solve this issue our recommendation is:

1) Increase "crashkernel=" default to 196M on Ubuntu for s390.

2) Document that KDUMP_CMDLINE_APPEND with "cio_ignore=" can be used to decrease memory consumption for kdump on systems with many devices that are not required for kdump.

The most user friendly solution would be to automatically determine the required kdump devices and set the correct "cio_ignore=" kernel parameter. But this is not trivial, because it can be difficult to find out the required devices for stacked setups like LVM or for network dump.

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Dimitri John Ledkov (xnox)]

128MB is just a sensible default enabled by default everywhere, if in your case you need more do adjust it accordingly in /etc/zipl.conf and execute `sudo zipl` and then reboot.

There is no dynamic sizing / computation of zipl.conf and/or crashkernel paramemter on Ubuntu, and on all architectures one may need to adjust it to lower/higher values as needed.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-31 11:55 EDT-------
IMHO a good method would be to set in /etc/default/kdump-tools the default values explicitely in KDUMP_CMDLINE , e.g.
KDUMP_CMDLINE="irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service"
and to change upon every start/restart of kdump-tools the KDUMP_CMDLINE_APPEND to
KDUMP_CMDLINE_APPEND="<cio_ignore_statement>", where the cio_ignore-statement can be easily computed by executing 'cio_ignore -k', which creates an cio_ignore statement for all enabled ccw devices.

[Dimitri John Ledkov (xnox)]

I do not want to increase it beyond 128MB by default, as this amount of memory is reserved by default on all installations big or small. And there will always be newer kernel (that reserves more memory) or more devices that require higher values. Adjusting kernel args in /etc/zipl.conf is well-known and well-documented.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-31 12:31 EDT-------
(In reply to comment #9)
> IMHO a good method would be to set in /etc/default/kdump-tools the default
> values explicitely in KDUMP_CMDLINE , e.g.
> KDUMP_CMDLINE="irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service"
> and to change upon every start/restart of kdump-tools the
> KDUMP_CMDLINE_APPEND to
> KDUMP_CMDLINE_APPEND="<cio_ignore_statement>", where the
> cio_ignore-statement can be easily computed by executing 'cio_ignore -k',
> which creates an cio_ignore statement for all enabled ccw devices.

where the cio_ignore-statement can be easily computed by executing 'cio_ignore -u && cio_ignore -k'

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-31 12:50 EDT-------
(In reply to comment #11)
> I do not want to increase it beyond 128MB by default, as this amount of
> memory is reserved by default on all installations big or small. And there
> will always be newer kernel (that reserves more memory) or more devices that
> require higher values.

Ok, if we go with the 128M default our current assumption is that on all LPARs kdump for Ubuntu will fail.

I am not sure how many customers are aware of the fact that they have to manually tune the kdump settings. Therefore what about some documentation like the following:

Depending on your system setup it can be necessary to increase the default setting of the "crashkernel=" kernel parameter. It is recommended to test the kdump setup with "echo c > /proc/sysrq-trigger".
If the crashkernel value is not high enough, kdump will crash with out-of-memory error messages. You can see the message on the operating system messages. In case kdump does not work you have to increase the value in /etc/zipl.conf.

Do you already have a similar documentation already available?

[Louis Bouchard (louis)]

***WARNING***

KDUMP_CMDLINE_APPEND is used to add parameters to the command used by kexec to reboot when the system crashes. Adding crashkernel= there is useless and will most likely be ignored.

The crashkernel= parameter has to be made available on the system when it first boots so having it in /etc/zipl.conf it the proper way to do it.

[Dimitri John Ledkov (xnox)]

"Ok, if we go with the 128M default our current assumption is that on all LPARs kdump for Ubuntu will fail." -> well, that doesn't sound good either, cause then essentially we take away 128M of RAM away without actually providing anything to the LPAR installations =(

[Louis Bouchard (louis)]

If there is a way to determine if we're on a zVM or Lpar, I could adapt kdump-config to add the appropriate crashkernel=value pair in /etc/zipl.conf and re-run zipl upon installation of kdump-tools, which is required in order to capture a dump.

[Dimitri John Ledkov (xnox)]

On 1 April 2016 at 12:22, Louis Bouchard <email address hidden> wrote:
> If there is a way to determine if we're on a zVM or Lpar, I could adapt
> kdump-config to add the appropriate crashkernel=value pair in
> /etc/zipl.conf and re-run zipl upon installation of kdump-tools, which
> is required in order to capture a dump.
>

$ systemd-detect-virt
zvm
(aka z/VM)

$ systemd-detect-virt
none
(aka LPAR)

--
Regards,

Dimitri.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-06 16:43 EDT-------
*** Bug 139821 has been marked as a duplicate of this bug. ***

[Launchpad Janitor (janitor)]

This bug was fixed in the package zipl-installer - 0.0.33ubuntu3

---------------
zipl-installer (0.0.33ubuntu3) xenial; urgency=medium

  * Reluctantly Bump crashkernel to 196M, as otherwise 128M is not enough
    by default to collect a kdump on any LPAR, defeating the intended
    purpose in the first place. (LP: #1564475)

 -- Dimitri John Ledkov <email address hidden> Mon, 11 Apr 2016 12:49:46 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package s390-tools - 1.34.0-0ubuntu5

---------------
s390-tools (1.34.0-0ubuntu5) xenial; urgency=medium

  * Add condition to cpacfstatsd.service to only start on LPARs (which may
    still fail, if LPAR is not configured with the right counters).
  * Add a system group for cpacfstatsd.service.
  * Reluctantly Bump crashkernel to 196M, as otherwise 128M is not enough
    by default to collect a kdump on any LPAR, defeating the intended
    purpose in the first place. (LP: #1564475)

 -- Dimitri John Ledkov <email address hidden> Mon, 11 Apr 2016 12:51:36 +0100

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-14 11:37 EDT-------
IMO this is partially fixed.
Crashkernel is set to 196M now, which is the major part of the fix.
On our LPARs crashkernel=196M is not sufficient, as already stated in Michael's bug description. Reducing the visible ccw device via cio_ignore is required. I either expect a fix like described in LP comment #6 (LTC comment # 12) or a link to a documentation update.

[Dimitri John Ledkov (xnox)]

@thorsten

If I understand correctly cio_ignore command needs to be on the running instance that is about to crash, rather than on the re-exec kernel. Thus it needs to be computed and added to e.g. /etc/zipl.conf. At the moment we do not generate/update /etc/zipl.conf in an automated way, but the more I think about it the better it sounds to e.g. able to specify all crashdump parameters, computed cio_ignore, correct root= argument, and generate menu items for every installed ubuntu kernel (rather than just the last one), and recovery stanzas too. I'll open a wishlist bug about that.

BTW, does it make sense to compute and use `cio_ignore -k -u` generated command line by default? on one hand kernel will use less memory, on the other hand `lszdev` will be quite empty and one will have a harder time to discover additional devices one can bring online.

I have, for now, added things to https://wiki.ubuntu.com/S390X -> feel free to edit and/or improve that. And I will open a new bug report to get those updates into the Ubuntu Server Guide.

[Dimitri John Ledkov (xnox)]

Server guide bug number https://bugs.launchpad.net/serverguide/+bug/1570644

[Dimitri John Ledkov (xnox)]

s390-tools zipl generation bug report https://bugs.launchpad.net/ubuntu/+source/s390-tools/+bug/1570645

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-15 05:15 EDT-------
(In reply to comment #25)
> @thorsten
>
> If I understand correctly cio_ignore command needs to be on the running
> instance that is about to crash, rather than on the re-exec kernel. Thus it

@Dimitri,
that's not correct. The LPAR kdump would also work fine, if the zipl.conf has NO cio_ignore statement, just the KDUMP_CMDLINE_APPEND= statement.

> needs to be computed and added to e.g. /etc/zipl.conf. At the moment we do
> not generate/update /etc/zipl.conf in an automated way, but the more I think
> about it the better it sounds to e.g. able to specify all crashdump
> parameters, computed cio_ignore, correct root= argument, and generate menu
> items for every installed ubuntu kernel (rather than just the last one), and
> recovery stanzas too. I'll open a wishlist bug about that.

That sounds reasonable.

> BTW, does it make sense to compute and use `cio_ignore -k -u` generated
> command line by default? on one hand kernel will use less memory, on the
> other hand `lszdev` will be quite empty and one will have a harder time to
> discover additional devices one can bring online.

Exactly, that is the conflict. From performance point of view, it is appreciable to have cio_ignore enabled, since an LPAR boot with thousands of devices takes a little bit longer. For our Test LPARs with many shared resources we prefer and recommend to have cio_ignore enabled. In a resonably set up customer environment the disadvantage of the more inconvenient device handling is more relevant than the benfits of cio_ignore. Feel free to hear other opinions on that topic, e.g. from your z Systems experts Frank and Christian ;-)
But for the KDUMP_CMDLINE_APPEND= statement is does NOT harm, and I recommend to add this automatically upon every restart of kdump-tools.service

> I have, for now, added things to https://wiki.ubuntu.com/S390X -> feel free
> to edit and/or improve that. And I will open a new bug report to get those
> updates into the Ubuntu Server Guide.

That reads good and is sufficient to close this bug. Thank you!

[Dimitri John Ledkov (xnox)]

I see, thanks a lot!.

Opened https://bugs.launchpad.net/ubuntu/+source/makedumpfile/+bug/1570775 to generate and use cio_ignore in kdump.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-15 08:13 EDT-------
(In reply to comment #25)
> I have, for now, added things to https://wiki.ubuntu.com/S390X -> feel free
> to edit and/or improve that. And I will open a new bug report to get those
> updates into the Ubuntu Server Guide.

Dimitri,
upon second reading I want at least to replace crashdebug by crashkernel, i.e.:

Depending on the number of available devices the amount of memory specified by the "crashkernel" kernel parameter in /etc/zipl.conf may not be sufficient. One can either increase it further, or limit the amount of devices visible to the kernel, and thus lower the requirements for the "crashkernel" setting.

... and if you think it's appropriate, please remind to invoke "zipl -V" after each change of /etc/zipl.conf (for the z Systems newbies)

Nevertheless, I also appreciate your LP1570775. Thanks again.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-15 09:39 EDT-------
well, here my complete proposed wording:

Depending on the number of available devices the amount of memory specified by the "crashkernel" kernel parameter in /etc/zipl.conf may not be sufficient. One can either increase it further, or limit the amount of devices visible to the kernel, and thus lower the requirements for the "crashkernel" setting.

To ignore devices you can run the "cio_ignore" tool to generate an appropriate stanza to ignore all devices, except the currently active/in-use.

$ sudo cio_ignore -u -k
cio_ignore=all,!c000-c002,!e000,!e100

You can simply add the "cio_ignore" kernel parameter with the stanza from above to KDUMP_CMDLINE_APPEND in /etc/default/kdump-tools and activate the setting with "systemctl restart kdump-tools.service".

As an alternative which blacklists the channel devices also for the production system, you can add the generated stanza to the boot parameters in /etc/zipl.conf. To activate the setting call "zipl -V" and then reboot the system.