is-leader not available during collect-metrics

[James Page]

unit-ceph-mon-2: 12:48:36 INFO unit.ceph-mon/2.juju-log mds:4: Processing 3 ceph broker requests
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics error: not implemented for restricted context not implemented
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics error: not implemented for restricted context not implemented
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics error: not implemented for restricted context not implemented
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics error: not implemented for restricted context not implemented
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics error: leadership status unknown: not implemented for restricted context not implemented
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics Traceback (most recent call last):
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/var/lib/juju/agents/unit-ceph-mon-1/charm/hooks/collect-metrics", line 669, in <module>
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics hooks.execute(sys.argv)
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/var/lib/juju/agents/unit-ceph-mon-1/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics self._hooks[hook_name]()
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/var/lib/juju/agents/unit-ceph-mon-1/charm/hooks/collect-metrics", line 121, in collect_metrics
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics if not is_leader():
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/var/lib/juju/agents/unit-ceph-mon-1/charm/hooks/charmhelpers/core/hookenv.py", line 853, in inner_translate_exc2
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics return f(*args, **kwargs)
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/var/lib/juju/agents/unit-ceph-mon-1/charm/hooks/charmhelpers/core/hookenv.py", line 883, in is_leader
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics raise CalledProcessError(retcode, cmd, output=output)
unit-ceph-mon-1: 12:49:37 INFO unit.unit-ceph-mon-1.collect-metrics subprocess.CalledProcessError: Command '['is-leader', '--format=json']' returned non-zero exit status 1

[James Page (james-page)]

Looks like is_leader does not work in the collect-metrics hook in juju 2.1 beta5 at least.

[Chris Holcombe (xfactor973)]

We can fall back on the ceph is_leader() function. I remember reading that the metrics-collect env is special and many juju functions are not available. I'll change it and propose the patch again.

[Casey Marshall (cmars)]

collect-metrics runs in a restricted hook context because it can execute concurrently with other "lifecycle" hooks. This prevents a long-running hook from interrupting data collection.

A workaround for this would be to set leadership (or other desired info) in charmhelpers unitdata, which is accessible from collect-metrics.

Longer term, we could look into allowing read-only hook tools to work in collect-metrics, with the caveat that it would be a point-in-time snapshot of a potentially changing state.

[Tim Penhey (thumper)]

James, if this is a seriously blocking issue, we can put it on the stakeholders board and track from there.

[Chris MacNaughton (chris.macnaughton)]

I think that this can be re-worked as Chris suggested earlier with the ceph is_leader() function, rather than relying on Juju's leader semantics to restrict metric collection to a single unit.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.