Testing with the keystone change that waits on vip. I now see that ceilometer-agent-central gets stopped gracefully.


root@juju-thedac-machine-1:/home/ubuntu# systemctl status ceilometer-agent-central
● ceilometer-agent-central.service - Ceilometer Agent Central
   Loaded: loaded (/lib/systemd/system/ceilometer-agent-central.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2017-02-16 18:40:44 UTC; 46min ago
 Main PID: 15130 (code=exited, status=0/SUCCESS)

Feb 16 18:40:19 juju-thedac-machine-1 ceilometer-agent-central[15130]: 2017-02-16 18:40:19.724 15130 WARNING ceilometer.agent.discovery.endpoint [req-ab1a62
Feb 16 18:40:20 juju-thedac-machine-1 ceilometer-agent-central[15130]: 2017-02-16 18:40:20.305 15130 ERROR ceilometer.agent.manager [req-ab1a62aa-d4f0-4c9e-
Feb 16 18:40:20 juju-thedac-machine-1 ceilometer-agent-central[15130]: 2017-02-16 18:40:20.306 15130 ERROR ceilometer.agent.manager [req-ab1a62aa-d4f0-4c9e-
Feb 16 18:40:22 juju-thedac-machine-1 ceilometer-agent-central[15130]: 2017-02-16 18:40:22.007 15130 ERROR ceilometer.agent.manager [req-ab1a62aa-d4f0-4c9e-
Feb 16 18:40:22 juju-thedac-machine-1 ceilometer-agent-central[15130]: 2017-02-16 18:40:22.707 15130 WARNING ceilometer.neutron_client [req-ab1a62aa-d4f0-4c
Feb 16 18:40:22 juju-thedac-machine-1 ceilometer-agent-central[15130]: 2017-02-16 18:40:22.709 15130 ERROR ceilometer.agent.manager [req-ab1a62aa-d4f0-4c9e-
Feb 16 18:40:22 juju-thedac-machine-1 ceilometer-agent-central[15130]: 2017-02-16 18:40:22.743 15130 ERROR ceilometer.agent.manager [req-ab1a62aa-d4f0-4c9e-
Feb 16 18:40:22 juju-thedac-machine-1 ceilometer-agent-central[15130]: 2017-02-16 18:40:22.745 15130 ERROR ceilometer.agent.manager [req-ab1a62aa-d4f0-4c9e-
Feb 16 18:40:44 juju-thedac-machine-1 systemd[1]: Stopping Ceilometer Agent Central...
Feb 16 18:40:44 juju-thedac-machine-1 systemd[1]: Stopped Ceilometer Agent Central.

root@juju-thedac-machine-1:/home/ubuntu# date
Thu Feb 16 19:27:05 UTC 2017

This suggests corosync could be to blame. And lo and behold:

root@juju-thedac-machine-1:/home/ubuntu# crm status
Last updated: Thu Feb 16 19:28:13 2017          Last change: Thu Feb 16 18:41:18 2017 by hacluster via crmd on juju-thedac-machine-1
Stack: corosync
Current DC: juju-thedac-machine-2 (version 1.1.14-70404b0) - partition with quorum
3 nodes and 5 resources configured

Online: [ juju-thedac-machine-1 juju-thedac-machine-2 juju-thedac-machine-3 ]

Full list of resources:

 Resource Group: grp_ceilometer_vips
     res_ceilometer_ens2_vip    (ocf::heartbeat:IPaddr2):       Started juju-thedac-machine-2
 Clone Set: cl_ceilometer_haproxy [res_ceilometer_haproxy]
     Started: [ juju-thedac-machine-1 juju-thedac-machine-2 juju-thedac-machine-3 ]
 res_ceilometer_polling (ocf::openstack:ceilometer-polling):    Started juju-thedac-machine-1

Failed Actions:
* res_ceilometer_polling_monitor_0 on juju-thedac-machine-2 'not installed' (5): call=73, status=Not installed, exitreason='none',
    last-rc-change='Thu Feb 16 18:40:44 2017', queued=0ms, exec=2ms
* res_ceilometer_polling_monitor_0 on juju-thedac-machine-3 'not installed' (5): call=73, status=Not installed, exitreason='none',
    last-rc-change='Thu Feb 16 18:40:44 2017', queued=0ms, exec=1ms