relation-get fails if a standby attempts to query the master's connection details too early.
2016-08-19 11:44:30 INFO db-admin-relation-changed error: cannot read settings for unit "postgresql/2" in relation "telegraf:postgresql postgresql:db-admin": settings not found
2016-08-19 11:44:30 INFO db-admin-relation-changed Traceback (most recent call last):
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/var/lib/juju/agents/unit-postgresql-3/charm/hooks/db-admin-relation-changed", line 18, in <module>
2016-08-19 11:44:30 INFO db-admin-relation-changed main()
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charms/reactive/__init__.py", line 78, in main
2016-08-19 11:44:30 INFO db-admin-relation-changed bus.dispatch()
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charms/reactive/bus.py", line 434, in dispatch
2016-08-19 11:44:30 INFO db-admin-relation-changed _invoke(other_handlers)
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charms/reactive/bus.py", line 417, in _invoke
2016-08-19 11:44:30 INFO db-admin-relation-changed handler.invoke()
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charms/reactive/bus.py", line 291, in invoke
2016-08-19 11:44:30 INFO db-admin-relation-changed self._action(*args)
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/var/lib/juju/agents/unit-postgresql-3/charm/reactive/postgresql/client.py", line 79, in mirror_master
2016-08-19 11:44:30 INFO db-admin-relation-changed db_relation_mirror(rel)
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/var/lib/juju/agents/unit-postgresql-3/charm/reactive/postgresql/client.py", line 149, in db_relation_mirror
2016-08-19 11:44:30 INFO db-admin-relation-changed rel.local.update({k: master_info.get(k) for k in master_keys})
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/var/lib/juju/agents/unit-postgresql-3/charm/reactive/postgresql/client.py", line 149, in <dictcomp>
2016-08-19 11:44:30 INFO db-admin-relation-changed rel.local.update({k: master_info.get(k) for k in master_keys})
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/lib/python3.4/_collections_abc.py", line 422, in get
2016-08-19 11:44:30 INFO db-admin-relation-changed return self[key]
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/lib/python3.4/collections/__init__.py", line 904, in __getitem__
2016-08-19 11:44:30 INFO db-admin-relation-changed if key in self.data:
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charmhelpers/context.py", line 163, in data
2016-08-19 11:44:30 INFO db-admin-relation-changed return hookenv.relation_get(rid=self.relid, unit=self.unit)
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charmhelpers/core/hookenv.py", line 72, in wrapper
2016-08-19 11:44:30 INFO db-admin-relation-changed res = func(*args, **kwargs)
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charmhelpers/core/hookenv.py", line 357, in relation_get
2016-08-19 11:44:30 INFO db-admin-relation-changed return json.loads(subprocess.check_output(_args).decode('UTF-8'))
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/lib/python3.4/subprocess.py", line 620, in check_output
2016-08-19 11:44:30 INFO db-admin-relation-changed raise CalledProcessError(retcode, process.args, output=output)
2016-08-19 11:44:30 INFO db-admin-relation-changed subprocess.CalledProcessError: Command '['relation-get', '--format=json', '-r', 'db-admin:25', '-', 'postgresql/2']' returned non-zero exit status 1
The bug here is that charmhelpers. core.hookenv. relation_ get() may raise a subprocess. CalledProcessEr ror when attempting to access a relation that is not fully setup. While this is arguably a Juju bug, we need to make relation_get more robust and handle the exception. Maybe log a WARNING or ERROR too.