juju remove-unit ceph-osd/5 breaks

[Gábor Mészáros]

Running the command in the subject resulted at secrets-storage-relation-departed hook to fail. Charm unit log:

2020-03-19 16:05:39 INFO juju-log secrets-storage:289: Unknown hook secrets-storage-relation-departed - skipping.
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed Traceback (most recent call last):
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed File "/var/lib/juju/agents/unit-ceph-osd-5/charm/hooks/secrets-storage-relation-departed", line 877, in <module>
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed assess_status()
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed File "/var/lib/juju/agents/unit-ceph-osd-5/charm/hooks/secrets-storage-relation-departed", line 812, in assess_status
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed if not vaultlocker.vault_relation_complete():
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed File "/var/lib/juju/agents/unit-ceph-osd-5/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 107, in vault_relation_complete
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed vault_kv()
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed File "/var/lib/juju/agents/unit-ceph-osd-5/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 60, in __call__
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed token=token
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed File "/var/lib/juju/agents/unit-ceph-osd-5/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 123, in retrieve_secret_id
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed response = client._post('/v1/sys/wrapping/unwrap')
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1239, in _post
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed return self.__request('post', url, **kwargs)
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1278, in __request
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed self.__raise_error(response.status_code, text, errors=errors)
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1284, in __raise_error
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed raise exceptions.InvalidRequest(message, errors=errors)
2020-03-19 16:05:41 DEBUG secrets-storage-relation-departed hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist
2020-03-19 16:05:41 ERROR juju.worker.uniter.operation runhook.go:132 hook "secrets-storage-relation-departed" failed: exit status 1

The workaround I used was to make token = last_token (with juju run ... relation-set )
and then it passed the failing line: File "/var/lib/juju/agents/unit-ceph-osd-5/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 60.

But I believe the issue is that it removes first the relation to vault and then tries to fetch a secret_id with an invalidated token key.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716549
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=0ecc33223cdc6a7a1b162cb93ca622df013f9b8e
Submitter: Zuul
Branch: master

commit 0ecc33223cdc6a7a1b162cb93ca622df013f9b8e
Author: Chris MacNaughton <email address hidden>
Date: Wed Apr 1 12:56:11 2020 +0200

    Be more defensive when checking Vault

    When relations are departing during a unit removal, the relation
    to Vault can loose enough data that the vault-relation-departed hook
    raises an exception. The checks for Vault readiness should be more
    defensive to ensure that charms can successfully depart their relations

    Depends-On https://github.com/juju/charm-helpers/pull/445
    Closes-Bug: #1868282

    Change-Id: I0b6226f0f3500aef7304f8e8b38d06daebfd0c20