[xenial->bionic] (queens) vault fails to upgrade if all vault units are sealed beforehand (fails during post-series-upgrade hook)

[Alex Kavanagh]

During series upgrade, the lead vault unit (the first one to be upgraded during series upgrade), failed to unseal and, thus went into hook error:

Model Controller Cloud/Region Version SLA Timestamp
mojo tinwood-serverstack serverstack/serverstack 2.7.7 unsupported 06:36:15Z

App Version Status Scale Charm Store Rev OS Notes
vault 1.1.1 error 3 vault local 0 ubuntu
vault-hacluster blocked 3 hacluster local 13 ubuntu

Unit Workload Agent Machine Public address Ports Message
vault/0 active idle 61 172.20.0.73 8200/tcp Unit is ready (active: true, mlock: enabled)
  vault-hacluster/1 blocked idle 172.20.0.73 HA services shutdown, peers are ready for series upgrade
vault/1* error idle 62 172.20.0.70 8200/tcp hook failed: "post-series-upgrade"
  vault-hacluster/0* blocked idle 172.20.0.70 Resource: res_vault-ext_92638bc_vip not running
vault/2 active idle 63 172.20.0.57 8200/tcp Unit is ready (active: false, mlock: enabled)
  vault-hacluster/2 blocked idle 172.20.0.57 HA services shutdown, peers are ready for series upgrade

Machine State DNS Inst id Series AZ Message
61 started 172.20.0.73 a32fc17b-4ce1-474e-9335-b7ac548b4127 xenial nova ACTIVE
62 started 172.20.0.70 c6b855d4-b293-493e-b30e-c0959e690635 xenial nova ACTIVE
63 started 172.20.0.57 3953490a-2dc7-4748-9f74-e3267235fe97 xenial nova ACTIVE

The last few lines in the log, prior to the error:

unit-vault-1: 04:10:52 DEBUG unit.vault/1.juju-log Rendering vault.hcl.j2
unit-vault-1: 04:10:53 DEBUG unit.vault/1.juju-log Changing permissions on existing content: 33152 -> 384
unit-vault-1: 04:10:53 DEBUG unit.vault/1.juju-log Rendering vault systemd configuation
unit-vault-1: 04:10:53 DEBUG unit.vault/1.juju-log Changing permissions on existing content: 33188 -> 420
unit-vault-1: 04:10:53 DEBUG unit.vault/1.juju-log Opening vault port
unit-vault-1: 04:10:53 INFO unit.vault/1.juju-log Invoking reactive handler: reactive/vault_handlers.py:253:mysql_setup
unit-vault-1: 04:10:54 INFO unit.vault/1.juju-log Invoking reactive handler: reactive/vault_handlers.py:282:database_not_ready
unit-vault-1: 04:10:54 INFO unit.vault/1.juju-log Invoking reactive handler: reactive/vault_handlers.py:365:cluster_connected
unit-vault-1: 04:10:54 INFO unit.vault/1.juju-log Invoking reactive handler: reactive/vault_handlers.py:544:prime_assess_status
unit-vault-1: 04:10:54 INFO unit.vault/1.juju-log Invoking reactive handler: reactive/vault_handlers.py:765:publish_ca_info
unit-vault-1: 04:14:58 ERROR unit.vault/1.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-vault-1/charm/reactive/vault_handlers.py", line 770, in publish_ca_info
    if not client_approle_authorized():
  File "/var/lib/juju/agents/unit-vault-1/charm/reactive/vault_handlers.py", line 717, in client_approle_authorized
    vault.get_local_client()
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/tenacity/__init__.py", line 329, in wrapped_f
    return self.call(f, *args, **kw)
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/tenacity/__init__.py", line 409, in call
    do = self.iter(retry_state=retry_state)
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/tenacity/__init__.py", line 368, in iter
    raise retry_exc.reraise()
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/tenacity/__init__.py", line 186, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/tenacity/__init__.py", line 412, in call
    result = fn(*args, **kwargs)
  File "/var/lib/juju/agents/unit-vault-1/charm/lib/charm/vault.py", line 252, in get_local_client
    client.auth_approle(app_role_id)
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2072, in auth_approle
    return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token)
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 1729, in auth
    **kwargs
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 159, in auth
    response = self.post(url, **kwargs).json()
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 103, in post
    return self.request('post', url, **kwargs)
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 233, in request
    utils.raise_for_error(response.status_code, text, errors=errors)
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/utils.py", line 43, in raise_for_error
    raise exceptions.VaultDown(message, errors=errors)
hvac.exceptions.VaultDown: Vault is sealed

...

Interestingly, it's failed at the "client.auth_approle(...)" call, that's been seen in another error, which I'm wondering is significant. Will try to verify if the unit is actually sealed or it's an HVAC bug.

[Alex Kavanagh (ajkavanagh)]

Possibly the same as https://bugs.launchpad.net/vault-charm/+bug/1889654 "
ocn rev 105 Unable to authorize approle after unseal".

[Alex Kavanagh (ajkavanagh)]

I'm beginning to suspect that this bug is not what it seems. There IS a bug in the upgrade charm test, but it might be because ALL the vault units were sealed prior to the test taking place, which means that when the 'post-series-upgrade' hook fired, there was no unsealed unit for the charm to talk to.

So it might be:

 1. The charm shouldn't be attempting to chat to vault during the post-series-upgrade hook at all.
 2. The series-upgrade should verify that vault is unsealed prior to starting the test.
 3. It shouldn't matter that vault is sealed or unsealed before series upgrade (although it will definitely be unsealed afterwards due to the reboot) - from a charm perspective. i.e. it shouldn't care, as it should care about the status of the vault (maybe)

Note, in the failure, it was the leader (first unit) that failed during upgrade. After it was 'helped' through, by manually unsealing the vault, the other two units upgrade okay.

I suspect we need to stop the post-series-upgrade hook from attempting to do certain actions on the vault.