ceilometer connection fails trying to connect to gnocchi-vip

[John George]

ERROR ceilometer ConnectFailure: Unable to establish connection to http://192.168.33.7:8041/v1/resource_type/ceph_account: HTTPConnectionPool(host='192.168.33.7', port=8041): Max retries exceeded with url: /v1/resource_type/ceph_account (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa0ec1ebd90>: Failed to establish a new connection: [Errno 113] No route to host',))

However, in the unit-hacluster-gnocchi-1.log it appears that the VIP may not actually be ready.

2018-02-08 05:59:15 DEBUG juju-log ha:57: Configuring and (maybe) restarting corosync

[Nobuto Murata (nobuto)]

LP: #1748286 and LP: #1746548 might have be related each other.

[Chris Gregan (cgregan)]

subscribed to field high due to FEs experiencing it in field deploys

[Jason Hobbs (jason-hobbs)]

We think this may be related to bug 1749280 - see https://bugs.launchpad.net/charm-ceilometer/+bug/1749280/comments/4

Also maybe related to bug 1746548.

[David Ames (thedac)]

While digging through the bundle I noticed we have aa-profile-mode: enforce for ceph-osd. James Page recently ran into a bug with ceph-osd and apparmor that caused gnocchi to lock up and cause all kinds of cascading failures.

I'll confer with James Page to confirm.

If anyone is able please test a deploy with aa-profile-mode disabled for ceph-osd and see if there is a difference in behavior.

All of the similar bugs may be due to this root cause. I'll do my own testing as well.