vault charm gives permission denied running authorize-charm action

[Chris Procter]

I have a 3 node HA clustered vault installation.

Following the instructions to authorize the vault charm at https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-vault.html

On the leader I ran:
export VAULT_TOKEN=[root token]
vault token create -use-limit=1 -ttl=10m

then on the juju node ran
juju run-action vault/0 authorize-charm token=[generated token]

The action resulted in:

ubuntu@fnos-inf02:~$ juju show-action-output 540a5d76-e0e2-4af6-8a09-49f5e2113433
message: permission denied
status: failed

repeated runs with newly generated tokens produce the same result.

This is consistant across reinstallations of the cluster

[Chris Procter (chrisp262)]

ubuntu@juju-a9b1a9-12-lxd-15:~$ export VAULT_TOKEN=458edd7b-cb35-366f-cb37-805210dbde29
ubuntu@juju-a9b1a9-12-lxd-15:~$ vault token create -use-limit=1 -ttl=10m
Key Value
--- -----
token 4dd9f7fc-f784-aa68-4c64-941a32ac2b92
  ken_accessor 3c09dd08-aa0c-8a6b-21a3-6d8370669013
token_duration 10m
token_renewable true
token_policies [root]
ubuntu@juju-a9b1a9-12-lxd-15:~$ exit
logout
Connection to 172.17.20.214 closed.
ubuntu@fnos-inf02:~$ juju run-action vault/1 authorize-charm token=4dd9f7fc-f784-aa68-4c64-941a32ac2b92
Action queued with id: 540a5d76-e0e2-4af6-8a09-49f5e2113433
ubuntu@fnos-inf02:~$ juju show-action-output 540a5d76-e0e2-4af6-8a09-49f5e2113433
message: permission denied
status: failed
timing:
  completed: 2018-07-03 13:28:05 +0000 UTC
  enqueued: 2018-07-03 13:28:01 +0000 UTC
  started: 2018-07-03 13:28:01 +0000 UTC

[Chris Procter (chrisp262)]

My suspicion is that vault token create -use-limit=1 -ttl=10m is creating a single use token and the juju action is performing more than one request to vault so the subsequent ones are rejected.

So instead I ran vault token create -ttl=10m which creates an unlimited use token. By using this the action returns completed.

However all the OSDs immediatly drop into error state with "hook failed: "secrets-storage-relation-changed""

They have the following errors in their logs:

2018-07-03 19:51:03 DEBUG secrets-storage-relation-changed Failed to find physical volume "/dev/sda".
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed Device /dev/sda is not a valid LUKS device.
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 172.17.20.43
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed DEBUG:urllib3.connectionpool:http://172.17.20.43:8200 "POST /v1/auth/approle/login HTTP/1.1" 400 36
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed vaultlocker: missing client token
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed Traceback (most recent call last):
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/secrets-storage-relation-changed", line 630, in <module>
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed hooks.execute(sys.argv)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/charmhelpers/core/hookenv.py", line 823, in execute
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed self._hooks[hook_name]()
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/secrets-storage-relation-changed", line 574, in secrets_storage_changed
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed prepare_disks_and_activate()
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/secrets-storage-relation-changed", line 449, in prepare_disks_and_activate
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed config('osd-encrypt-keymanager'))
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "lib/ceph/utils.py", line 1399, in osdize
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed bluestore, key_manager)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "lib/ceph/utils.py", line 1461, in osdize_dev
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed key_manager)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "lib/ceph/utils.py", line 1594, in _ceph_volume
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed key_manager=key_manager))
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "lib/ceph/utils.py", line 1804, in _allocate_logical_volume
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed pv_dev = _initialize_disk(dev, dev_uuid, encrypt, key_manager)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "lib/ceph/utils.py", ...

[Liam Young (gnuoy)]

Thank you for getting to the bottom of this and reporting back. Can you raise a new bug for the secrets-storage-relation-changed failure please ?

I will get the documentation updated for the token creation.

[James Page (james-page)]

Docs updated to remove one-shot nature of token generated, just relying on the TTL to limit its use instead.