OSA run: units related to ceph got an empty ceph.conf
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Landscape Server | Status tracked in Trunk | |||||
| 15.07 |
High
|
Chris Glass | |||
| Cisco-odl |
High
|
Chris Glass | |||
| Trunk |
High
|
Chris Glass | |||
| ceph (Juju Charms Collection) |
High
|
Liam Young | |||
| cinder (Juju Charms Collection) |
High
|
Liam Young | |||
| cinder-ceph (Juju Charms Collection) |
High
|
Liam Young | |||
| glance (Juju Charms Collection) |
High
|
Liam Young | |||
| nova-compute (Juju Charms Collection) |
High
|
Liam Young |
Bug Description
This is almost identical to https:/
After an OSA run, where we relate glance to ceph, glance image upload failed. Further inspection showed that /etc/ceph/ceph.conf on all 3 glance units was basically empty:
"""
#######
# [ WARNING ]
# glance configuration file maintained by Juju
# local changes may be overwritten.
#######
[global]
log to syslog =
err to syslog =
clog to syslog =
"""
After I manually destroyed the ceph-glance relation, and created it back again, then /etc/ceph/ceph.conf on the 3 glance units had valid configuration:
"""
#######
# [ WARNING ]
# glance configuration file maintained by Juju
# local changes may be overwritten.
#######
[global]
auth_supported = cephx
#keyring = /etc/ceph/
keyring = /etc/ceph/
mon host = 10.1.4.142 10.1.4.152 10.1.4.158
log to syslog = false
err to syslog = false
clog to syslog = false
"""
And glance started to work then.
Related branches
- Edward Hope-Morley: Approve on 2015-09-11
-
Diff: 1195 lines (+885/-62)7 files modifiedhooks/ceph_broker.py (+12/-2)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
hooks/hooks.py (+9/-1)
tests/charmhelpers/contrib/amulet/utils.py (+234/-52)
tests/charmhelpers/contrib/openstack/amulet/deployment.py (+20/-5)
tests/charmhelpers/contrib/openstack/amulet/utils.py (+359/-0)
unit_tests/test_ceph_broker.py (+27/-0)
- Edward Hope-Morley: Approve on 2015-09-11
-
Diff: 1305 lines (+886/-104) (has conflicts)8 files modifiedcharmhelpers/contrib/openstack/context.py (+8/-9)
charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
hooks/glance_relations.py (+13/-19)
templates/ceph.conf (+2/-1)
tests/charmhelpers/contrib/amulet/utils.py (+234/-52)
tests/charmhelpers/contrib/openstack/amulet/deployment.py (+20/-5)
tests/charmhelpers/contrib/openstack/amulet/utils.py (+359/-0)
unit_tests/test_glance_relations.py (+26/-16)
- Edward Hope-Morley: Approve on 2015-09-11
- Chris Glass (community): Approve on 2015-09-10
-
Diff: 812 lines (+612/-30)5 files modifiedcharmhelpers/contrib/openstack/context.py (+8/-9)
charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
tests/contrib/openstack/test_os_contexts.py (+36/-4)
tests/contrib/storage/test_linux_ceph.py (+337/-8)
tests/helpers.py (+7/-7)
- Edward Hope-Morley: Approve on 2015-09-11
-
Diff: 1252 lines (+866/-91)7 files modifiedhooks/charmhelpers/contrib/openstack/context.py (+8/-9)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
hooks/nova_compute_hooks.py (+14/-23)
tests/charmhelpers/contrib/amulet/utils.py (+234/-52)
tests/charmhelpers/contrib/openstack/amulet/deployment.py (+20/-5)
tests/charmhelpers/contrib/openstack/amulet/utils.py (+359/-0)
unit_tests/test_nova_compute_hooks.py (+7/-0)
- Edward Hope-Morley: Approve on 2015-09-11
-
Diff: 1274 lines (+867/-101)7 files modifiedhooks/charmhelpers/contrib/openstack/context.py (+8/-9)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
hooks/cinder_hooks.py (+13/-19)
tests/charmhelpers/contrib/amulet/utils.py (+234/-52)
tests/charmhelpers/contrib/openstack/amulet/deployment.py (+20/-5)
tests/charmhelpers/contrib/openstack/amulet/utils.py (+359/-0)
unit_tests/test_cinder_hooks.py (+9/-14)
- Edward Hope-Morley: Needs Fixing on 2015-09-11
- Ryan Beisner: Needs Fixing on 2015-09-08
-
Diff: 1382 lines (+929/-101)8 files modifiedhooks/charmhelpers/contrib/openstack/context.py (+8/-9)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
hooks/cinder_hooks.py (+13/-22)
tests/basic_deployment.py (+52/-8)
tests/charmhelpers/contrib/amulet/utils.py (+234/-52)
tests/charmhelpers/contrib/openstack/amulet/deployment.py (+20/-5)
tests/charmhelpers/contrib/openstack/amulet/utils.py (+359/-0)
unit_tests/test_cinder_hooks.py (+19/-3)
- Chris Glass (community): Approve on 2015-09-18
-
Diff: 812 lines (+612/-30)5 files modifiedcharmhelpers/contrib/openstack/context.py (+8/-9)
charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
tests/contrib/openstack/test_os_contexts.py (+36/-4)
tests/contrib/storage/test_linux_ceph.py (+337/-8)
tests/helpers.py (+7/-7)
- Chris Glass (community): Approve on 2015-09-22
-
Diff: 487 lines (+278/-38)8 files modifiedhooks/ceph_broker.py (+12/-2)
hooks/charmhelpers/cli/__init__.py (+1/-5)
hooks/charmhelpers/cli/commands.py (+4/-4)
hooks/charmhelpers/cli/hookenv.py (+23/-0)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
hooks/charmhelpers/core/hookenv.py (+1/-20)
hooks/charmhelpers/core/host.py (+2/-2)
hooks/hooks.py (+11/-3)
- Chris Glass (community): Approve on 2015-09-22
-
Diff: 643 lines (+295/-82)11 files modifiedhooks/charmhelpers/cli/__init__.py (+1/-5)
hooks/charmhelpers/cli/commands.py (+4/-4)
hooks/charmhelpers/cli/hookenv.py (+23/-0)
hooks/charmhelpers/contrib/openstack/context.py (+8/-9)
hooks/charmhelpers/contrib/openstack/utils.py (+7/-5)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
hooks/charmhelpers/contrib/storage/linux/utils.py (+3/-2)
hooks/charmhelpers/core/hookenv.py (+1/-20)
hooks/charmhelpers/core/host.py (+2/-2)
hooks/cinder_hooks.py (+13/-19)
unit_tests/test_cinder_hooks.py (+9/-14)
- Chris Glass (community): Approve on 2015-09-23
- Ryan Beisner: Needs Fixing on 2015-09-22
-
Diff: 751 lines (+357/-82)12 files modifiedhooks/charmhelpers/cli/__init__.py (+1/-5)
hooks/charmhelpers/cli/commands.py (+4/-4)
hooks/charmhelpers/cli/hookenv.py (+23/-0)
hooks/charmhelpers/contrib/openstack/context.py (+8/-9)
hooks/charmhelpers/contrib/openstack/utils.py (+7/-5)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
hooks/charmhelpers/contrib/storage/linux/utils.py (+3/-2)
hooks/charmhelpers/core/hookenv.py (+1/-20)
hooks/charmhelpers/core/host.py (+2/-2)
hooks/cinder_hooks.py (+13/-22)
tests/basic_deployment.py (+52/-8)
unit_tests/test_cinder_hooks.py (+19/-3)
- Chris Glass (community): Approve on 2015-09-22
-
Diff: 623 lines (+294/-73)11 files modifiedhooks/charmhelpers/cli/__init__.py (+1/-5)
hooks/charmhelpers/cli/commands.py (+4/-4)
hooks/charmhelpers/cli/hookenv.py (+23/-0)
hooks/charmhelpers/contrib/openstack/context.py (+8/-9)
hooks/charmhelpers/contrib/openstack/utils.py (+7/-5)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
hooks/charmhelpers/core/hookenv.py (+1/-20)
hooks/charmhelpers/core/host.py (+2/-2)
hooks/glance_relations.py (+13/-19)
templates/ceph.conf (+2/-1)
unit_tests/test_glance_relations.py (+9/-6)
- Chris Glass (community): Approve on 2015-09-23
-
Diff: 634 lines (+295/-73)12 files modifiedhooks/charmhelpers/cli/__init__.py (+1/-5)
hooks/charmhelpers/cli/commands.py (+4/-4)
hooks/charmhelpers/cli/hookenv.py (+23/-0)
hooks/charmhelpers/contrib/openstack/context.py (+8/-9)
hooks/charmhelpers/contrib/openstack/utils.py (+7/-5)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+224/-2)
hooks/charmhelpers/contrib/storage/linux/utils.py (+3/-2)
hooks/charmhelpers/core/hookenv.py (+1/-20)
hooks/charmhelpers/core/host.py (+2/-2)
hooks/nova_compute_hooks.py (+14/-23)
metadata.yaml (+1/-1)
unit_tests/test_nova_compute_hooks.py (+7/-0)
Andreas Hasenack (ahasenack) wrote : | #2 |
Chris Holcombe (xfactor973) wrote : | #3 |
Ryan Beisner (1chb1n) wrote : | #4 |
FYI - adding bug link as it's a separate issue in the same chunk of c-h code that generates this ceph.conf file.
https:/
Chris Holcombe (xfactor973) wrote : | #5 |
Repro worked! Tracking down what is going wrong. It's interesting that the ceph client keyring is created successfully but the conf file is not.
Chris Holcombe (xfactor973) wrote : | #6 |
I think there's a logic bug in there. It looks like if you add-relation, let it fail, remove-relation, let it clean up and then add-relation again it gets further than the fails on librados permissions.
Ryan Beisner (1chb1n) wrote : | #7 |
Can we please get the following info on this (and any other potential ubuntu openstack charm bugs)?:
* What Ubuntu-OpenStack combo was deployed? ie. trusty-icehouse, trusty-kilo, etc.
* `juju get <service>` output on the relevant services
* `juju status --format yaml` output [OK, ALREADY ATTACHED]
This information is necessary in order to attempt to reproduce the occurrence.
Thank you.
Ryan Beisner (1chb1n) wrote : | #8 |
FYI: Trusty-Juno is where the reporter observed this behavior, as indicated in the all-machines log.
unit-neutron-
Alberto Donato (ack) wrote : | #9 |
We've seen this issue with trusty/kilo.
Liam Young (gnuoy) wrote : | #10 |
I have not been able to reproduce the bug as it's a race between ceph and glance. If glance is ready before ceph has posted all the required relation data then the Ceph context is empty and when ceph.conf is rendered the data is missing. Although this is not ideal from what I can see this should only be a transative state and when ceph posts the full set of data hooks should fire on the glance unit and rerender the ceph.conf fully populated so the bug is that ceph.conf is not being rendered once the context is complete.
I think I found the cause of the bug which has another symptom which is easy to reproduce:
1) Deploy ceph and relate to glance (I used this bundle: http://
2) juju status --format=tabular | grep 'ceph/' | wc -l
3
2) juju run --unit glance/0 "grep mon /var/lib/
mon host = 10.5.42.205 10.5.42.206 10.5.42.207
3) juju add-unit ceph
4) Wait for ceph/3 to finish deploying
5) juju status --format=tabular | grep 'ceph/' | wc -l
4
6) juju run --unit glance/0 "grep mon /var/lib/
mon host = 10.5.42.205 10.5.42.206 10.5.42.207
This list of mon hosts has not changed. The reason for this is that the glance charm will not render the ceph.conf unless it has a broker_rsp from the unit it is talking to, in this case ceph/3. But the only unit who replies with a broker_resp is the leader which is (probably) not ceph3.
Changed in glance (Juju Charms Collection): | |
status: | New → In Progress |
importance: | Undecided → High |
assignee: | nobody → Liam Young (gnuoy) |
Liam Young (gnuoy) wrote : | #11 |
Looks like there are three bugs conspiring to cause this:
1) Ceph config is registered with incomplete context when identity relation fires
2) CephContext does not stop looping through relids when it has a complete context
which means if ceph/0 has a full set of data but ceph/1 does not. The returned
context is incomplete
3) When once a glance unit has issued a CephRequests it can never issue another
one because the relation data making the request is never cleared or changed
so a subsequent request sets the same relation data as was there previously
being no hooks fire on the ceph side.
4) When once the ceph leader has responded to a CephRequests it can never issue another
response because the relation data making the request is never cleared or changed
so a subsequent response sets the same relation data as was there previously
being no hooks fire on the client side.
Changed in ceph (Juju Charms Collection): | |
status: | New → In Progress |
Changed in cinder (Juju Charms Collection): | |
status: | New → In Progress |
Changed in cinder-ceph (Juju Charms Collection): | |
status: | New → In Progress |
Changed in nova-compute (Juju Charms Collection): | |
status: | New → In Progress |
Changed in ceph (Juju Charms Collection): | |
importance: | Undecided → High |
Changed in cinder (Juju Charms Collection): | |
importance: | Undecided → High |
Changed in cinder-ceph (Juju Charms Collection): | |
importance: | Undecided → High |
Changed in nova-compute (Juju Charms Collection): | |
importance: | Undecided → High |
Changed in ceph (Juju Charms Collection): | |
assignee: | nobody → Liam Young (gnuoy) |
Changed in cinder (Juju Charms Collection): | |
assignee: | nobody → Liam Young (gnuoy) |
Changed in cinder-ceph (Juju Charms Collection): | |
assignee: | nobody → Liam Young (gnuoy) |
Changed in nova-compute (Juju Charms Collection): | |
assignee: | nobody → Liam Young (gnuoy) |
Changed in ceph (Juju Charms Collection): | |
milestone: | none → 15.10 |
Changed in cinder (Juju Charms Collection): | |
milestone: | none → 15.10 |
Changed in cinder-ceph (Juju Charms Collection): | |
milestone: | none → 15.10 |
Changed in glance (Juju Charms Collection): | |
milestone: | none → 15.10 |
Changed in nova-compute (Juju Charms Collection): | |
milestone: | none → 15.10 |
tags: | added: backport-potential openstack sts |
summary: |
- OSA run: glance related to ceph got an empty ceph.conf + OSA run: units related to ceph got an empty ceph.conf |
tags: | added: landscape-release-29 |
tags: | added: landscape |
Changed in ceph (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in cinder (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in glance (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in nova-compute (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in cinder-ceph (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in cinder (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in ceph (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in glance (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in nova-compute (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in cinder-ceph (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
tags: | removed: landscape-release-29 |
I'm going to try and reproduce this and see where it leads me