When issuing a normal reboot command on an overcloud node the following stop sequence can take place:
------------- -----------------------------
| Pacemaker | | paunch-container-shutdown |
------------- -----------------------------
\ /
\ /
\ /
-----------------
| docker |
-----------------
We currently have two issues:
A) paunch-container-shutdown has no 'After=pacemaker.service' in the service definition
We want this for two reasons:
A.1) We want the normal openstack services to be able to stop before the DB/rabbit so they can run any actions that need to store state on shutdown correctly.
A.2) paunch-container-shutdown stops 10 containers concurrently at the same time which on some environments creates quite the load. Doing that and shutting down the HA containers at the same time is less than desirable.
B) If there are docker plugins that are allowed to stop before docker and also before pacemaker, it might happen that stopping them down during the pacemaker stop will cause a bunch of timeouts and a failure to stop containers:
Sep 13 17:53:00.821030 controller-0.localdomain pacemakerd[6147]: notice: Shutting down Pacemaker
Sep 13 17:54:15.798026 controller-0.localdomain lrmd[6284]: warning: galera-bundle-docker-0_monitor_60000 process (PID 226329) timed out
Sep 13 17:54:15.799004 controller-0.localdomain lrmd[6284]: warning: galera-bundle-docker-0_monitor_60000:226329 - timed out after 20000ms
One of these plugins is 'rhel-push-plugin.service'
So B) is the root cause for the timeout, but it is quite desirable to fix A) as well.
A) is fixed via https:/