OSA run: units related to ceph got an empty ceph.conf

[Andreas Hasenack]

This is almost identical to https://bugs.launchpad.net/charms/+source/cinder/+bug/1453934. The same thing happened, but this time with glance.

After an OSA run, where we relate glance to ceph, glance image upload failed. Further inspection showed that /etc/ceph/ceph.conf on all 3 glance units was basically empty:
"""
###############################################################################
# [ WARNING ]
# glance configuration file maintained by Juju
# local changes may be overwritten.
###############################################################################
[global]
log to syslog =
 err to syslog =
 clog to syslog =
"""

After I manually destroyed the ceph-glance relation, and created it back again, then /etc/ceph/ceph.conf on the 3 glance units had valid configuration:
"""
###############################################################################
# [ WARNING ]
# glance configuration file maintained by Juju
# local changes may be overwritten.
###############################################################################
[global]
auth_supported = cephx
 #keyring = /etc/ceph/$cluster.$name.keyring
 keyring = /etc/ceph/ceph.$name.keyring
 mon host = 10.1.4.142 10.1.4.152 10.1.4.158
log to syslog = false
 err to syslog = false
 clog to syslog = false
"""

And glance started to work then.

[Chris Holcombe (xfactor973)]

I'm going to try and reproduce this and see where it leads me

[Ryan Beisner (1chb1n)]

FYI - adding bug link as it's a separate issue in the same chunk of c-h code that generates this ceph.conf file.
https://bugs.launchpad.net/charm-helpers/+bug/1468511

[Chris Holcombe (xfactor973)]

Repro worked! Tracking down what is going wrong. It's interesting that the ceph client keyring is created successfully but the conf file is not.

[Chris Holcombe (xfactor973)]

I think there's a logic bug in there. It looks like if you add-relation, let it fail, remove-relation, let it clean up and then add-relation again it gets further than the fails on librados permissions.

[Ryan Beisner (1chb1n)]

Can we please get the following info on this (and any other potential ubuntu openstack charm bugs)?:

* What Ubuntu-OpenStack combo was deployed? ie. trusty-icehouse, trusty-kilo, etc.
* `juju get <service>` output on the relevant services
* `juju status --format yaml` output [OK, ALREADY ATTACHED]

This information is necessary in order to attempt to reproduce the occurrence.

Thank you.

[Ryan Beisner (1chb1n)]

FYI: Trusty-Juno is where the reporter observed this behavior, as indicated in the all-machines log.

unit-neutron-gateway-0[4945]: 2015-05-11 16:50:08 INFO unit.neutron-gateway/0.install logger.go:40 Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno InRelease

[Alberto Donato (ack)]

We've seen this issue with trusty/kilo.

[Liam Young (gnuoy)]

I have not been able to reproduce the bug as it's a race between ceph and glance. If glance is ready before ceph has posted all the required relation data then the Ceph context is empty and when ceph.conf is rendered the data is missing. Although this is not ideal from what I can see this should only be a transative state and when ceph posts the full set of data hooks should fire on the glance unit and rerender the ceph.conf fully populated so the bug is that ceph.conf is not being rendered once the context is complete.

I think I found the cause of the bug which has another symptom which is easy to reproduce:

1) Deploy ceph and relate to glance (I used this bundle: http://paste.ubuntu.com/12123358/)
2) juju status --format=tabular | grep 'ceph/' | wc -l
   3
2) juju run --unit glance/0 "grep mon /var/lib/charm/glance/ceph.conf" 2>/dev/null
   mon host = 10.5.42.205 10.5.42.206 10.5.42.207
3) juju add-unit ceph
4) Wait for ceph/3 to finish deploying
5) juju status --format=tabular | grep 'ceph/' | wc -l
   4
6) juju run --unit glance/0 "grep mon /var/lib/charm/glance/ceph.conf" 2>/dev/null
   mon host = 10.5.42.205 10.5.42.206 10.5.42.207

This list of mon hosts has not changed. The reason for this is that the glance charm will not render the ceph.conf unless it has a broker_rsp from the unit it is talking to, in this case ceph/3. But the only unit who replies with a broker_resp is the leader which is (probably) not ceph3.

[Liam Young (gnuoy)]

Looks like there are three bugs conspiring to cause this:

1) Ceph config is registered with incomplete context when identity relation fires
2) CephContext does not stop looping through relids when it has a complete context
   which means if ceph/0 has a full set of data but ceph/1 does not. The returned
   context is incomplete
3) When once a glance unit has issued a CephRequests it can never issue another
   one because the relation data making the request is never cleared or changed
   so a subsequent request sets the same relation data as was there previously
   being no hooks fire on the ceph side.
4) When once the ceph leader has responded to a CephRequests it can never issue another
   response because the relation data making the request is never cleared or changed
   so a subsequent response sets the same relation data as was there previously
   being no hooks fire on the client side.