containers are not gracefully stopped on shutdown/reboot

Bug #1778913 reported by Brent Eagles
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Alex Schultz

Bug Description

There is no mechanism to gracefully shutdown containers when shutting down/rebooting a node. They keep running until systemd performs a process cleanup and sends SIGTERM to all running process. If they are still running for some interval after the SIGTERM, systemd sends SIGKILL. It is unknown what impact that this might have.

Example from console logs:

[ 1556.163360] type=1700 audit(1529955121.410:884): dev=qr-90058585-3f prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
[ 1558.874838] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 1558.886653] systemd-shutdown[1]: Sending SIGKILL to PID 2044 (docker-containe).
[ 1558.892701] systemd-shutdown[1]: Sending SIGKILL to PID 2079 (kolla_start).
[ 1558.897848] systemd-shutdown[1]: Sending SIGKILL to PID 2189 (ceilometer-agen).
[ 1558.903574] systemd-shutdown[1]: Sending SIGKILL to PID 2190 (docker-containe).
[ 1558.909220] systemd-shutdown[1]: Sending SIGKILL to PID 2217 (docker-containe).
[ 1558.914799] systemd-shutdown[1]: Sending SIGKILL to PID 2277 (docker-containe).
[ 1558.920219] systemd-shutdown[1]: Sending SIGKILL to PID 2317 (kolla_start).
[ 1558.924792] systemd-shutdown[1]: Sending SIGKILL to PID 2329 (kolla_start).
[ 1558.932073] systemd-shutdown[1]: Sending SIGKILL to PID 2376 (kolla_start).
[ 1558.943927] systemd-shutdown[1]: Sending SIGKILL to PID 2584 (aodh-notifier: ).
[ 1558.952055] systemd-shutdown[1]: Sending SIGKILL to PID 2588 (docker-containe).
[ 1558.956779] systemd-shutdown[1]: Sending SIGKILL to PID 2612 (cinder-schedule).
[ 1558.961131] systemd-shutdown[1]: Sending SIGKILL to PID 2690 (neutron-l3-agen).

It's worth noting that projects like neutron refined the systemd service behavior over years including the addition of order-dependent cleanup oneshot services like "neutron-ovs-cleanup". These refinements addressed actual issues so some form of graceful shutdown mechanism is desirable to avoid unexpected, difficult to debug, issues.

Brent Eagles (beagles)
Changed in tripleo:
status: New → Triaged
importance: Undecided → Medium
milestone: none → stein-1
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :
tags: added: containers
Changed in tripleo:
importance: Medium → High
milestone: stein-1 → rocky-3
Revision history for this message
Alex Schultz (alex-schultz) wrote :
Changed in tripleo:
status: Triaged → Fix Released
assignee: nobody → Alex Schultz (alex-schultz)
tags: added: queens-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers