On Mon, Jun 24, 2019 at 11:59:48AM -0000, bugproxy wrote:
> ------- Comment From <email address hidden> 2019-06-24 07:49 EDT-------
> Thanks for the change. With it, try-restart is being triggered for
> kdump-tools service after CPU add operation but systemd reported
> failure with below logs:
>
> Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service.
> Jun 24 06:47:06 ubuntu systemd[1]: Starting Kernel crash dump capture service...
> Jun 24 06:47:06 ubuntu kdump-tools[2023]: Starting kdump-tools: * Creating symlink /var/lib/kdump/vmlinuz
> Jun 24 06:47:06 ubuntu kdump-tools[2023]: * Creating symlink /var/lib/kdump/initrd.img
> Jun 24 06:47:06 ubuntu kdump-tools[2023]: Modified cmdline:BOOT_IMAGE=/vmlinux-5.0.0-17-generic root=/dev/mapper/ubuntu--vg-root ro systemd.unit=kdump-tools-dump.service maxcpus=1 irqpo
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Main process exited, code=killed, status=15/TERM
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result 'signal'.
> Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service.
> Jun 24 06:47:06 ubuntu systemd[1]: Starting Kernel crash dump capture service...
> Jun 24 06:47:06 ubuntu kdump-tools[2071]: Starting kdump-tools: * Creating symlink /var/lib/kdump/vmlinuz
> Jun 24 06:47:06 ubuntu kdump-tools[2071]: * Creating symlink /var/lib/kdump/initrd.img
> Jun 24 06:47:06 ubuntu kdump-tools[2071]: Modified cmdline:BOOT_IMAGE=/vmlinux-5.0.0-17-generic root=/dev/mapper/ubuntu--vg-root ro systemd.unit=kdump-tools-dump.service maxcpus=1 irqpo
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Main process exited, code=killed, status=15/TERM
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result 'signal'.
> Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service.
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Start request repeated too quickly.
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result 'signal'.
> Jun 24 06:47:06 ubuntu systemd[1]: Failed to start Kernel crash dump capture service.
>
> ---
> Looks like a ratelimit issue with systemd. Is there some systemd option to workaround it?
>
> I am running the below command on a PowerVM machine:
>
> # drmgr -c cpu -r -q 1 (to remove a core)
> # drmgr -c cpu -a -q 1 (to add it back -> this triggers 8 CPU online udev events as SMT is 8)
>
> To conclude, udev rule alone is not sufficient. Need a way to address the multiple
> requests at once..
There are these systemd options, which default to a burst limit of 5 restart in
the interval of 10s.
One other option that I prefer, howerver, is resetting the start rate limit
counter by using systemctl reset-failed kdump-tools.service on the udev rule.
On Mon, Jun 24, 2019 at 11:59:48AM -0000, bugproxy wrote: kdump/vmlinuz kdump/initrd. img BOOT_IMAGE= /vmlinux- 5.0.0-17- generic root=/dev/ mapper/ ubuntu- -vg-root ro systemd. unit=kdump- tools-dump. service maxcpus=1 irqpo service: Main process exited, code=killed, status=15/TERM service: Failed with result 'signal'. kdump/vmlinuz kdump/initrd. img BOOT_IMAGE= /vmlinux- 5.0.0-17- generic root=/dev/ mapper/ ubuntu- -vg-root ro systemd. unit=kdump- tools-dump. service maxcpus=1 irqpo service: Main process exited, code=killed, status=15/TERM service: Failed with result 'signal'. service: Start request repeated too quickly. service: Failed with result 'signal'.
> ------- Comment From <email address hidden> 2019-06-24 07:49 EDT-------
> Thanks for the change. With it, try-restart is being triggered for
> kdump-tools service after CPU add operation but systemd reported
> failure with below logs:
>
> Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service.
> Jun 24 06:47:06 ubuntu systemd[1]: Starting Kernel crash dump capture service...
> Jun 24 06:47:06 ubuntu kdump-tools[2023]: Starting kdump-tools: * Creating symlink /var/lib/
> Jun 24 06:47:06 ubuntu kdump-tools[2023]: * Creating symlink /var/lib/
> Jun 24 06:47:06 ubuntu kdump-tools[2023]: Modified cmdline:
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.
> Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service.
> Jun 24 06:47:06 ubuntu systemd[1]: Starting Kernel crash dump capture service...
> Jun 24 06:47:06 ubuntu kdump-tools[2071]: Starting kdump-tools: * Creating symlink /var/lib/
> Jun 24 06:47:06 ubuntu kdump-tools[2071]: * Creating symlink /var/lib/
> Jun 24 06:47:06 ubuntu kdump-tools[2071]: Modified cmdline:
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.
> Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service.
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.
> Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.
> Jun 24 06:47:06 ubuntu systemd[1]: Failed to start Kernel crash dump capture service.
>
> ---
> Looks like a ratelimit issue with systemd. Is there some systemd option to workaround it?
>
> I am running the below command on a PowerVM machine:
>
> # drmgr -c cpu -r -q 1 (to remove a core)
> # drmgr -c cpu -a -q 1 (to add it back -> this triggers 8 CPU online udev events as SMT is 8)
>
> To conclude, udev rule alone is not sufficient. Need a way to address the multiple
> requests at once..
There are these systemd options, which default to a burst limit of 5 restart in
the interval of 10s.
One other option that I prefer, howerver, is resetting the start rate limit
counter by using systemctl reset-failed kdump-tools.service on the udev rule.
Can you try that?
Thanks.
Cascardo.