Pacemaker reports haproxy_monitor not running
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Gnocchi Charm |
Fix Released
|
High
|
James Page | ||
OpenStack AODH Charm |
Fix Released
|
High
|
James Page | ||
OpenStack Barbican Charm |
Fix Released
|
High
|
James Page | ||
OpenStack Designate Charm |
Fix Released
|
High
|
James Page | ||
charms.openstack |
Fix Released
|
Undecided
|
Liam Young |
Bug Description
When deploying 3 units with cs-hacluster-49, charm 18.08 (cs:aodh--17, designate-21, , after a short while (sometimes minutes, sometimes hours) pacemaker reports the following:
(example):
ubuntu@
Last updated: Thu Nov 1 02:23:56 2018 Last change: Wed Sep 19 05:07:40 2018 by hacluster via crmd on juju-5b85c6-
Stack: corosync
Current DC: juju-5b85c6-
3 nodes and 6 resources configured
Online: [ juju-5b85c6-
Full list of resources:
Resource Group: grp_aodh_vips
res_
res_
res_
Clone Set: cl_res_aodh_haproxy [res_aodh_haproxy]
Started: [ juju-5b85c6-
Failed Actions:
* res_aodh_
last-
* res_aodh_
last-
* res_aodh_
last-
This affects at the least aodh, designate, and gnocchi.
Looking a a bunch of affected units, I'm seeing unit logs during update-status hooks which call render_config(), and a bunch of relation-get, network-list, etc. If that's restarting or refreshing services, that'll trigger pacemaker to report a failure. Looking at process ages, for example in a Gnocchi unit, I see the apache and wsgi processes are only a matter of hours old when no changes were made recently. Example log from the same time as 'last-rc-change' from a gnocchi unit: https:/
~$ juju run --unit gnocchi/0 'charms.reactive -p get_flags'
['amqp.connected',
'charm.installed',
'charms.
'charms.
'charms.
'charms.
'charms.
'charms.
'cluster.
'cluster.
'config.rendered',
'coordinator-
'coordinator-
'db.synced',
'gnocchi-
'ha.available',
'ha.connected',
'haproxy.
'identity-
'identity-
'identity-
'metric-
'run-default-
'shared-
'shared-
'ssl.enabled',
'storage-
'storage-
'storage-
summary: |
- Pacemaker reports haproxy_montor not running + Pacemaker reports haproxy_monitor not running |
Changed in charms.openstack: | |
assignee: | nobody → Liam Young (gnuoy) |
Changed in charm-designate: | |
status: | Fix Committed → Fix Released |
Changed in charm-barbican: | |
status: | Fix Committed → Fix Released |
Changed in charm-aodh: | |
status: | Fix Committed → Fix Released |
Changed in charm-gnocchi: | |
status: | Fix Committed → Fix Released |
Logs from an aodh unit - it appears to be re-rendering the config without checking if it needs to - https:/ /pastebin. canonical. com/p/C47KMbMRm c/