keystone nrpe relation throws python traceback

[Brad Marshall]

While deploying Openstack with juju 1 and xenial, I've run into an odd problem relating the keystone charm to nrpe charm, it's throwing a python traceback:

2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined Traceback (most recent call last):
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/var/lib/juju/agents/unit-nrpe-16/charm/hooks/nrpe-external-master-relation-
joined", line 3, in <module>
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined services.manage()
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/var/lib/juju/agents/unit-nrpe-16/charm/hooks/services.py", line 27, in manage
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined nrpe_helpers.NagiosInfo(),
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/var/lib/juju/agents/unit-nrpe-16/charm/hooks/nrpe_helpers.py", line 203, in __init__
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined self['nagios_hostname'] = self.principal_relation.nagios_hostname()
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/var/lib/juju/agents/unit-nrpe-16/charm/hooks/nrpe_helpers.py", line 165, in nagios_hostname
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined principal_unitname = hookenv.principal_unit()
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/var/lib/juju/agents/unit-nrpe-16/charm/hooks/charmhelpers/core/hookenv.py", line 220, in principal_unit
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined md = _metadata_unit(unit)
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/var/lib/juju/agents/unit-nrpe-16/charm/hooks/charmhelpers/core/hookenv.py", line 520, in _metadata_unit
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined return yaml.safe_load(md)
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 94, in safe_load
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined return load(stream, SafeLoader)
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined loader = Loader(stream)
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/usr/lib/python3/dist-packages/yaml/loader.py", line 24, in __init__
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined Reader.__init__(self, stream)
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined self.determine_encoding()
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined self.update_raw()
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined data = self.stream.read(size)
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined return codecs.ascii_decode(input, self.errors)[0]
2017-10-04 07:10:50 INFO nrpe-external-master-relation-joined UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 311: ordinal not in range(128)
2017-10-04 07:10:50 ERROR juju.worker.uniter.operation runhook.go:107 hook "nrpe-external-master-relation-joined" failed: exit status 1

This is only occuring with the keystone charm, none of the others are giving an error similar to this, and its happening across multiple redeploys of the charm.

Keystone is using charm version 17.02, nrpe is from lp:nrpe-charm.

Please let me know if you need any further information.

[Liam Young (gnuoy)]

It looks like there is a specific code path for a subordinate to find out information about the principle for juju 1.X:

def principal_unit():
    """Returns the principal unit of this unit, otherwise None"""
    # Juju 2.2 and above provides JUJU_PRINCIPAL_UNIT
    principal_unit = os.environ.get('JUJU_PRINCIPAL_UNIT', None)
    # If it's empty, then this unit is the principal
    if principal_unit == '':
        return os.environ['JUJU_UNIT_NAME']
    elif principal_unit is not None:
        return principal_unit
    # For Juju 2.1 and below, let's try work out the principle unit by
    # the various charms' metadata.yaml.
    for reltype in relation_types():
        for rid in relation_ids(reltype):
            for unit in related_units(rid):
                md = _metadata_unit(unit)
                if not md:
                    continue
                subordinate = md.pop('subordinate', None)
                if not subordinate:
                    return unit
    return None

It looks like the principle is returning invalid yaml.

[Liam Young (gnuoy)]

<tl;dr> I really don't think this is a bug in the keystone charm.</tl;dr>

This bug can be reproduced a 100% of the time by joining keystone to lp:nrpe-charm
with juju 1. If "juju resolved --retry nrpe/0" is then run the hook again errors
but if a debug-hooks session is running on nrpe/0 and nrpe-external-master-relation-joined
is run by hand the hook completes succesfully.

The charm can also be fixed by updating charmhlpers in the nrpe charm with the following patch:

diff --git a/hooks/charmhelpers/core/hookenv.py b/hooks/charmhelpers/core/hookenv.py
index 899722f..1a1f1fc 100644
--- a/hooks/charmhelpers/core/hookenv.py
+++ b/hooks/charmhelpers/core/hookenv.py
@@ -516,7 +516,8 @@ def _metadata_unit(unit):
     joineddir = os.path.join(basedir, unitdir, 'charm', 'metadata.yaml')
     if not os.path.exists(joineddir):
         return None
- with open(joineddir) as md:
+ with open(joineddir, 'r', encoding='utf-8') as md:
+ log('_metadata_unit: Loading {} '.format(joineddir))
         return yaml.safe_load(md)

However it would need some testing, particularly where there are multiple subordinates connected to the same principle.

I am going to mark this as a bug against lp:nrpe-charm as that is where it is surfacing but
it might equally be a bug against juju 1.25.X or charm-helpers

[Haw Loeung (hloeung)]

It's actually this non-ASCII character in the metadata.yaml file:

| implements OpenStack’s Identity API.
| ^

| $ file metadata.yaml
| metadata.yaml: UTF-8 Unicode text

I've submitted a PR to fix this in the Keystone charm, but I do agree that this should also be fixed in the charmhelpers code.

https://github.com/openstack/charm-keystone/pull/4

[Haw Loeung (hloeung)]

For the keystone charm - https://review.openstack.org/#/c/513361/

[Haw Loeung (hloeung)]

For charm-helpers - https://github.com/juju/charm-helpers/pull/25

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513361
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=b4d8247f76bc756f504c5f98242bd0b7d3278a78
Submitter: Zuul
Branch: master

commit b4d8247f76bc756f504c5f98242bd0b7d3278a78
Author: Haw Loeung <email address hidden>
Date: Thu Oct 19 21:32:38 2017 +1100

    Remove non-ASCII character.

    Change-Id: I5d4ec43dd7eb3c2512d330262cceceda4c3a55eb
    Closes-Bug: #1721200

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.08
Review: https://review.openstack.org/513795

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513795
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=91490b7daf7511a717f75f62b57fc3f97cc6d740
Submitter: Zuul
Branch: stable/17.08

commit 91490b7daf7511a717f75f62b57fc3f97cc6d740
Author: Haw Loeung <email address hidden>
Date: Thu Oct 19 21:32:38 2017 +1100

    Remove non-ASCII character.

    Change-Id: I5d4ec43dd7eb3c2512d330262cceceda4c3a55eb
    Closes-Bug: #1721200
    (cherry picked from commit b4d8247f76bc756f504c5f98242bd0b7d3278a78)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.02
Review: https://review.openstack.org/514260

[Drew Freiberger (afreiberger)]

copying Haw's comment in gerrit here for the stable/17.02 backport:

For Juju 2.x, that's no longer 'juju action do' but 'juju run-action'. I think the tests need to be updated but looking at the latest changes to charmhelper's amulet test, run_action() is being deprecated (c1f9acd4527c82708be1ba871ae54b7faace45e1).

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitrii Shcherbakov (<email address hidden>) on branch: stable/17.02
Review: https://review.openstack.org/514260