stop hook failed when removing a second unit

[Paul Collins]

When removing a second graylog unit (which scenario apparently doesn't work), the stop hook failed, which means that manual intervention (juju resolved --no-retry graylog/1) is required to complete the operation.

2018-09-26 22:49:19 INFO juju-log Reactive main running for hook stop
2018-09-26 22:49:19 INFO juju-log Initializing Snap Layer
2018-09-26 22:49:19 DEBUG stop none
2018-09-26 22:49:19 INFO juju-log Invoking reactive handler: reactive/graylog.py:86:blocked_es
2018-09-26 22:49:19 INFO juju-log Invoking reactive handler: reactive/graylog.py:420:configure_mongodb_connection
2018-09-26 22:49:19 DEBUG stop ERROR no relation id specified
2018-09-26 22:49:20 ERROR juju-log Hook error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/charms/reactive/__init__.py", line 72, in main
    bus.dispatch(restricted=restricted_mode)
  File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 375, in dispatch
    _invoke(other_handlers)
  File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 351, in _invoke
    handler.invoke()
  File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 173, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-graylog-1/charm/reactive/graylog.py", line 424, in configure_mongodb_connection
    for mongo_host in mongodb.connection_strings():
  File "/var/lib/juju/agents/unit-graylog-1/charm/hooks/relations/mongodb/requires.py", line 49, in connection_strings
    'hostname': conv.get_remote('hostname'),
  File "/usr/local/lib/python3.5/dist-packages/charms/reactive/relations.py", line 787, in get_remote
    units = hookenv.related_units(relation_id)
  File "/usr/local/lib/python3.5/dist-packages/charmhelpers/core/hookenv.py", line 75, in wrapper
    res = func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/charmhelpers/core/hookenv.py", line 462, in related_units
    subprocess.check_output(units_cmd_line).decode('UTF-8')) or []
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['relation-list', '--format=json']' returned non-zero exit status 2

2018-09-26 22:49:20 DEBUG stop Traceback (most recent call last):
2018-09-26 22:49:20 DEBUG stop File "/var/lib/juju/agents/unit-graylog-1/charm/hooks/stop", line 19, in <module>
2018-09-26 22:49:20 DEBUG stop main()
2018-09-26 22:49:20 DEBUG stop File "/usr/local/lib/python3.5/dist-packages/charms/reactive/__init__.py", line 72, in main
2018-09-26 22:49:20 DEBUG stop bus.dispatch(restricted=restricted_mode)
2018-09-26 22:49:20 DEBUG stop File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 375, in dispatch
2018-09-26 22:49:20 DEBUG stop _invoke(other_handlers)
2018-09-26 22:49:20 DEBUG stop File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 351, in _invoke
2018-09-26 22:49:20 DEBUG stop handler.invoke()
2018-09-26 22:49:20 DEBUG stop File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 173, in invoke
2018-09-26 22:49:20 DEBUG stop self._action(*args)
2018-09-26 22:49:20 DEBUG stop File "/var/lib/juju/agents/unit-graylog-1/charm/reactive/graylog.py", line 424, in configure_mongodb_connection
2018-09-26 22:49:20 DEBUG stop for mongo_host in mongodb.connection_strings():
2018-09-26 22:49:20 DEBUG stop File "/var/lib/juju/agents/unit-graylog-1/charm/hooks/relations/mongodb/requires.py", line 49, in connection_strings
2018-09-26 22:49:20 DEBUG stop 'hostname': conv.get_remote('hostname'),
2018-09-26 22:49:20 DEBUG stop File "/usr/local/lib/python3.5/dist-packages/charms/reactive/relations.py", line 787, in get_remote
2018-09-26 22:49:20 DEBUG stop units = hookenv.related_units(relation_id)
2018-09-26 22:49:20 DEBUG stop File "/usr/local/lib/python3.5/dist-packages/charmhelpers/core/hookenv.py", line 75, in wrapper
2018-09-26 22:49:20 DEBUG stop res = func(*args, **kwargs)
2018-09-26 22:49:20 DEBUG stop File "/usr/local/lib/python3.5/dist-packages/charmhelpers/core/hookenv.py", line 462, in related_units
2018-09-26 22:49:20 DEBUG stop subprocess.check_output(units_cmd_line).decode('UTF-8')) or []
2018-09-26 22:49:20 DEBUG stop File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
2018-09-26 22:49:20 DEBUG stop **kwargs).stdout
2018-09-26 22:49:20 DEBUG stop File "/usr/lib/python3.5/subprocess.py", line 708, in run
2018-09-26 22:49:20 DEBUG stop output=stdout, stderr=stderr)
2018-09-26 22:49:20 DEBUG stop subprocess.CalledProcessError: Command '['relation-list', '--format=json']' returned non-zero exit status 2
2018-09-26 22:49:20 ERROR juju.worker.uniter.operation runhook.go:114 hook "stop" failed: exit status 1

[Erich Eichinger (eeichinger)]

I am running into the same problem. For testing/evaluation I deployed a single graylog instance

```
applications:
  graylog:
    charm: 'cs:graylog-23'
    num_units: 1
    series: bionic
    to:
      - '0'
  elasticsearch:
    charm: 'cs:elasticsearch-32'
    num_units: 1
    series: bionic
    to:
      - '1'
  mongodb:
    charm: 'cs:mongodb-51'
    num_units: 1
    series: bionic
    to:
      - '2'
relations:
  - - 'mongodb:database'
    - 'graylog:mongodb'
  - - 'elasticsearch:client'
    - 'graylog:elasticsearch'
machines:
  '0': {}
  '1': {}
  '2': {}
```

then try to remove it with

   juju remove-application graylog

this fails with the error described above

[Tom Haddon (mthaddon)]

This has been released in cs:graylog-28. Please test and let us know if we can mark this bug as fix released.

[Erich Eichinger (eeichinger)]

the fix still doesn't allow removing and re-adding mongodb and/or elasticsearch relations. Attached is a patch that correctly handles this