metrics collection hook fails

Bug #1663584 reported by James Page on 2017-02-10
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack ceph-mon charm
Undecided
Chris Holcombe
juju
Medium
Unassigned
ceph-mon (Juju Charms Collection)
Undecided
Chris Holcombe

Bug Description

unit-ceph-mon-2: 12:48:36 INFO unit.ceph-mon/2.juju-log mds:4: Processing 3 ceph broker requests
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics error: not implemented for restricted context not implemented
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics error: not implemented for restricted context not implemented
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics error: not implemented for restricted context not implemented
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics error: not implemented for restricted context not implemented
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics error: leadership status unknown: not implemented for restricted context not implemented
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics Traceback (most recent call last):
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/var/lib/juju/agents/unit-ceph-mon-1/charm/hooks/collect-metrics", line 669, in <module>
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics hooks.execute(sys.argv)
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/var/lib/juju/agents/unit-ceph-mon-1/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics self._hooks[hook_name]()
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/var/lib/juju/agents/unit-ceph-mon-1/charm/hooks/collect-metrics", line 121, in collect_metrics
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics if not is_leader():
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/var/lib/juju/agents/unit-ceph-mon-1/charm/hooks/charmhelpers/core/hookenv.py", line 853, in inner_translate_exc2
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics return f(*args, **kwargs)
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/var/lib/juju/agents/unit-ceph-mon-1/charm/hooks/charmhelpers/core/hookenv.py", line 883, in is_leader
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics raise CalledProcessError(retcode, cmd, output=output)
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics subprocess.CalledProcessError: Command '['is-leader', '--format=json']' returned non-zero exit status 1

James Page (james-page) wrote :

Looks like is_leader does not work in the collect-metrics hook in juju 2.1 beta5 at least.

Changed in ceph-mon (Juju Charms Collection):
assignee: nobody → Chris MacNaughton (chris.macnaughton)
status: New → In Progress
James Page (james-page) on 2017-02-10
Changed in ceph-mon (Juju Charms Collection):
status: In Progress → Incomplete
Chris Holcombe (xfactor973) wrote :

We can fall back on the ceph is_leader() function. I remember reading that the metrics-collect env is special and many juju functions are not available. I'll change it and propose the patch again.

Changed in juju:
status: New → Triaged
importance: Undecided → High
milestone: none → 2.2.0
James Page (james-page) on 2017-02-23
Changed in charm-ceph-mon:
assignee: nobody → Chris MacNaughton (chris.macnaughton)
status: New → Incomplete
Changed in ceph-mon (Juju Charms Collection):
status: Incomplete → Invalid
Casey Marshall (cmars) wrote :

collect-metrics runs in a restricted hook context because it can execute concurrently with other "lifecycle" hooks. This prevents a long-running hook from interrupting data collection.

A workaround for this would be to set leadership (or other desired info) in charmhelpers unitdata, which is accessible from collect-metrics.

Longer term, we could look into allowing read-only hook tools to work in collect-metrics, with the caveat that it would be a point-in-time snapshot of a potentially changing state.

Curtis Hovey (sinzui) on 2017-03-24
Changed in juju:
milestone: 2.2-beta1 → 2.2-beta2
Changed in juju:
milestone: 2.2-beta2 → 2.2-beta3
Changed in juju:
milestone: 2.2-beta3 → 2.2-beta4
Changed in juju:
milestone: 2.2-beta4 → 2.2-rc1
Tim Penhey (thumper) wrote :

James, if this is a seriously blocking issue, we can put it on the stakeholders board and track from there.

Changed in juju:
milestone: 2.2-rc1 → none
importance: High → Medium
tags: added: jujuc metrics uniter

I think that this can be re-worked as Chris suggested earlier with the ceph is_leader() function, rather than relying on Juju's leader semantics to restrict metric collection to a single unit.

Changed in charm-ceph-mon:
assignee: Chris MacNaughton (chris.macnaughton) → Chris Holcombe (xfactor973)
Changed in ceph-mon (Juju Charms Collection):
assignee: Chris MacNaughton (chris.macnaughton) → Chris Holcombe (xfactor973)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers