graylog charm can't connect to graylog api during mongodb relation changed

[Jason Hobbs]

This is with cs:~llama-charmers-next/graylog-2, which we're using due to 1878745

2020-05-18 02:55:56 INFO juju-log mongodb:242: Error calling graylog api: HTTPConnectionPool(host='127.0.0.1', port=9000): Max retries exceeded with url: /api/?pretty=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7949bd6da0>: Failed to establish a new connection: [Errno 111] Connection refused',))
2020-05-18 02:55:56 ERROR juju-log mongodb:242: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-graylog-1/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-graylog-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-graylog-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-graylog-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-graylog-1/charm/reactive/graylog.py", line 329, in configure_graylog_api
    _verify_rest_api_is_alive()
  File "/var/lib/juju/agents/unit-graylog-1/charm/reactive/graylog.py", line 958, in _verify_rest_api_is_alive
    raise ApiTimeout()
reactive.graylog.ApiTimeout

2020-05-18 02:55:56 DEBUG mongodb-relation-changed Traceback (most recent call last):
2020-05-18 02:55:56 DEBUG mongodb-relation-changed File "/var/lib/juju/agents/unit-graylog-1/charm/hooks/mongodb-relation-changed", line 22, in <module>
2020-05-18 02:55:56 DEBUG mongodb-relation-changed main()
2020-05-18 02:55:56 DEBUG mongodb-relation-changed File "/var/lib/juju/agents/unit-graylog-1/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
2020-05-18 02:55:56 DEBUG mongodb-relation-changed bus.dispatch(restricted=restricted_mode)
2020-05-18 02:55:56 DEBUG mongodb-relation-changed File "/var/lib/juju/agents/unit-graylog-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
2020-05-18 02:55:56 DEBUG mongodb-relation-changed _invoke(other_handlers)
2020-05-18 02:55:56 DEBUG mongodb-relation-changed File "/var/lib/juju/agents/unit-graylog-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
2020-05-18 02:55:56 DEBUG mongodb-relation-changed handler.invoke()
2020-05-18 02:55:56 DEBUG mongodb-relation-changed File "/var/lib/juju/agents/unit-graylog-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
2020-05-18 02:55:56 DEBUG mongodb-relation-changed self._action(*args)
2020-05-18 02:55:56 DEBUG mongodb-relation-changed File "/var/lib/juju/agents/unit-graylog-1/charm/reactive/graylog.py", line 329, in configure_graylog_api
2020-05-18 02:55:56 DEBUG mongodb-relation-changed _verify_rest_api_is_alive()
2020-05-18 02:55:56 DEBUG mongodb-relation-changed File "/var/lib/juju/agents/unit-graylog-1/charm/reactive/graylog.py", line 958, in _verify_rest_api_is_alive
2020-05-18 02:55:56 DEBUG mongodb-relation-changed raise ApiTimeout()
2020-05-18 02:55:56 DEBUG mongodb-relation-changed reactive.graylog.ApiTimeout
2020-05-18 02:55:56 ERROR juju.worker.uniter.operation runhook.go:132 hook "mongodb-relation-changed" failed: exit status 1
2020-05-18 02:55:56 INFO juju.worker.uniter resolver.go:126 awaiting error resolution for "relation-changed" hook
2020-05-18 02:55:56 INFO juju.worker.uniter resolver.go:126 awaiting error resolution for "relation-changed" hook
2020-05-18 02:57:14 INFO juju.worker.uniter resolver.go:126 awaiting error resolution for "relation-changed" hook

Example run: https://solutions.qa.canonical.com/#/qa/testRun/1a763cc7-c65a-4f0a-b31c-06dddc4ed1fb

This happens everytime.

[Paul Goins (vultaire)]

Based upon my review of the code, this happens after the ES and MongoDB relations have been handled and configured locally. The Graylog REST API should at this point come up, but there may be some period before it comes up in which this error could be encountered.

Merge request which may address this: https://code.launchpad.net/~vultaire/charm-graylog/+git/graylog-charm/+merge/384135