k8s charm fails to access peer relation in peer relation-joined hook

Bug #1818230 reported by Stuart Bishop
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Ian Booth

Bug Description

Adding a second unit to a k8s charm deployment, my charm fails to run 'relation-get --format=json -' inside the peer relation-joined hook. The command returns a failure code, printing 'ERROR permission denied'.

application-pgk8s: 20:23:05 INFO unit.pgk8s/4.juju-log pgpeer:3: **** pgpeer-relation-joined DEBUG ****
application-pgk8s: 20:23:05 DEBUG unit.pgk8s/4.pgpeer-relation-joined ERROR permission denied
application-pgk8s: 20:23:05 ERROR unit.pgk8s/4.juju-log pgpeer:3: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-pgk8s-4/charm/reactive/pg.py", line 19, in debug
    pprint(hookenv.execution_environment(), out)
  File "lib/charmhelpers/core/hookenv.py", line 168, in execution_environment
    context['rel'] = relation_get()
  File "lib/charmhelpers/core/hookenv.py", line 82, in wrapper
    res = func(*args, **kwargs)
  File "lib/charmhelpers/core/hookenv.py", line 439, in relation_get
    return json.loads(subprocess.check_output(_args).decode('UTF-8'))
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['relation-get', '--format=json', '-']' returned non-zero exit status 1.

Revision history for this message
Stuart Bishop (stub) wrote :

$ juju --version
2.5.2-bionic-amd64

Tim Penhey (thumper)
Changed in juju:
status: New → Triaged
importance: Undecided → High
milestone: none → 2.5.3
assignee: nobody → Ian Booth (wallyworld)
Changed in juju:
milestone: 2.5.3 → 2.5.4
Changed in juju:
milestone: 2.5.4 → 2.5.5
Changed in juju:
milestone: 2.5.6 → 2.5.8
Changed in juju:
milestone: 2.5.8 → 2.5.9
Revision history for this message
james beedy (jamesbeedy) wrote :

bump!

Revision history for this message
james beedy (jamesbeedy) wrote :

Fumbling around trying to see what is going on here, I've commented out all charm code pertaining to the peer relation to see if things still error when a second unit is added.

Commenting out all peer relation charm code allowed the second unit to deploy successfully https://paste.ubuntu.com/p/fNPGgmGbMR/

Next, I wanted to add back in bits of the peer relation to see where it breaks.

When the only peer relation code was that in metadata.yaml https://github.com/omnivector-solutions/layer-slurmd-k8s/commit/9f39b24b32d94c17a983b61ba513eb12abb1694a

Aadding a second unit of the application worked, there was a warning message in the log "WARNING unit.slurmd/1.juju-log slurmctld:3: No RelationFactory found in relations.slurmd.peers",
but no relation-get error, everything successfully deployed with no hook errors.

The next step I took was to add back the relation factory back to peers.py - which is where it broke again

https://github.com/omnivector-solutions/layer-slurmd-k8s/commit/71455dcebfb793f706bd76d57a5b3a420ba74d5f

Notice the charm handler for the peer relation is still commented out - https://github.com/omnivector-solutions/layer-slurmd-k8s/blob/debug_peer_relation/src/reactive/slurmd_k8s.py#L130,L156

Ian Booth (wallyworld)
Changed in juju:
milestone: 2.5.9 → 2.7-beta1
Ian Booth (wallyworld)
Changed in juju:
milestone: 2.7-beta1 → 2.6.10
status: Triaged → In Progress
Ian Booth (wallyworld)
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.