Pacemaker package upgrades stop but fail to start pacemaker resulting in HA outage
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack HA Cluster Charm |
Invalid
|
Critical
|
Unassigned | ||
init-system-helpers (Ubuntu) |
Opinion
|
Undecided
|
Unassigned | ||
pacemaker (Ubuntu) |
Fix Released
|
Critical
|
Unassigned | ||
Xenial |
Fix Released
|
Critical
|
Unassigned | ||
Zesty |
Fix Released
|
Critical
|
Unassigned | ||
Artful |
Fix Released
|
Critical
|
Unassigned | ||
Bionic |
Fix Released
|
Critical
|
Unassigned |
Bug Description
[Impact]
upgrades of the pacemaker package don't restart pacemaker after the package upgrade, resulting in down HA clusters.
[Test Case]
sudo apt install pacemaker
sudo systemctl start pacemaker
sudo dpkg-reconfigure pacemaker
pacemaker daemons will not be restarted.
[Regression Potential]
Minimal, earlier and later versions provide the defaults in the lsb header.
[Original Bug Report]
We have found on our openstack charm-hacluster implementations that the pacemaker .deb packaging along with the upstream pacemaker configuration result in pacemaker stopping but not starting upon package upgrade (while attended or unattended).
This was seen on three separate Xenial clouds. Both Mitaka and Ocata.
The package upgrade today was to pacemaker 1.1.14-2ubuntu1.2.
It appears that pacemaker.prerm stops the service using
"invoke-rc.d pacemaker stop" and then the pacemaker.postinst attempts to start the service, but silently fails due to policy denial. It appears the policy check fails because /etc/rcX.
I have not checked whether this affects trusty environments.
I'd suggest on systems that use systemd, the pacemaker.postinst script should check if the service is enabled and start it with systemctl commands rather than using the cross-platform compatible invoke-rc.d wrappers. Or upstream pacemaker should get default start/stop entries.
Our default runlevel on cloud init built images appears to be 5 (graphical), so at least 5 should be present in /etc/init.
tags: | added: cpe-onsite |
Changed in pacemaker (Ubuntu Zesty): | |
importance: | Undecided → Critical |
Changed in pacemaker (Ubuntu Artful): | |
importance: | Undecided → Critical |
Changed in pacemaker (Ubuntu Xenial): | |
status: | New → Triaged |
Changed in pacemaker (Ubuntu Zesty): | |
status: | New → Triaged |
Changed in pacemaker (Ubuntu Xenial): | |
importance: | Undecided → Critical |
Changed in pacemaker (Ubuntu Artful): | |
status: | New → Triaged |
description: | updated |
tags: | added: patch |
tags: | added: uosci |
no longer affects: | init-system-helpers (Ubuntu Bionic) |
no longer affects: | init-system-helpers (Ubuntu Artful) |
no longer affects: | init-system-helpers (Ubuntu Zesty) |
no longer affects: | init-system-helpers (Ubuntu Xenial) |
Changed in init-system-helpers (Ubuntu): | |
status: | Confirmed → Opinion |
From the charm perspective we need to determine if the charm does anything beyond the packaging that could lead to this.
The charm runs:
update-rc.d -f pacemaker defaults
Testing.