k8s charm fails to access peer relation in peer relation-joined hook

[Stuart Bishop]

Adding a second unit to a k8s charm deployment, my charm fails to run 'relation-get --format=json -' inside the peer relation-joined hook. The command returns a failure code, printing 'ERROR permission denied'.

application-pgk8s: 20:23:05 INFO unit.pgk8s/4.juju-log pgpeer:3: **** pgpeer-relation-joined DEBUG ****
application-pgk8s: 20:23:05 DEBUG unit.pgk8s/4.pgpeer-relation-joined ERROR permission denied
application-pgk8s: 20:23:05 ERROR unit.pgk8s/4.juju-log pgpeer:3: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-pgk8s-4/charm/reactive/pg.py", line 19, in debug
    pprint(hookenv.execution_environment(), out)
  File "lib/charmhelpers/core/hookenv.py", line 168, in execution_environment
    context['rel'] = relation_get()
  File "lib/charmhelpers/core/hookenv.py", line 82, in wrapper
    res = func(*args, **kwargs)
  File "lib/charmhelpers/core/hookenv.py", line 439, in relation_get
    return json.loads(subprocess.check_output(_args).decode('UTF-8'))
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['relation-get', '--format=json', '-']' returned non-zero exit status 1.

[Stuart Bishop (stub)]

$ juju --version
2.5.2-bionic-amd64

[james beedy (jamesbeedy)]

bump!

[james beedy (jamesbeedy)]

Fumbling around trying to see what is going on here, I've commented out all charm code pertaining to the peer relation to see if things still error when a second unit is added.

Commenting out all peer relation charm code allowed the second unit to deploy successfully https://paste.ubuntu.com/p/fNPGgmGbMR/

Next, I wanted to add back in bits of the peer relation to see where it breaks.

When the only peer relation code was that in metadata.yaml https://github.com/omnivector-solutions/layer-slurmd-k8s/commit/9f39b24b32d94c17a983b61ba513eb12abb1694a

Aadding a second unit of the application worked, there was a warning message in the log "WARNING unit.slurmd/1.juju-log slurmctld:3: No RelationFactory found in relations.slurmd.peers",
but no relation-get error, everything successfully deployed with no hook errors.

The next step I took was to add back the relation factory back to peers.py - which is where it broke again

https://github.com/omnivector-solutions/layer-slurmd-k8s/commit/71455dcebfb793f706bd76d57a5b3a420ba74d5f

Notice the charm handler for the peer relation is still commented out - https://github.com/omnivector-solutions/layer-slurmd-k8s/blob/debug_peer_relation/src/reactive/slurmd_k8s.py#L130,L156