My suspicion is that vault token create -use-limit=1 -ttl=10m   is creating a single use token and the juju action is performing more than one request to vault so the subsequent ones are rejected.

So instead I ran vault token create -ttl=10m    which creates an unlimited use token.  By using this the action returns completed.

However all the OSDs immediatly drop into error state with "hook failed: "secrets-storage-relation-changed""

They have the following errors in their logs:

2018-07-03 19:51:03 DEBUG secrets-storage-relation-changed   Failed to find physical volume "/dev/sda".
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed Device /dev/sda is not a valid LUKS device.
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 172.17.20.43
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed DEBUG:urllib3.connectionpool:http://172.17.20.43:8200 "POST /v1/auth/approle/login HTTP/1.1" 400 36
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed vaultlocker: missing client token
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed Traceback (most recent call last):
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed   File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/secrets-storage-relation-changed", line 630, in <module>
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed     hooks.execute(sys.argv)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed   File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/charmhelpers/core/hookenv.py", line 823, in execute
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed     self._hooks[hook_name]()
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed   File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/secrets-storage-relation-changed", line 574, in secrets_storage_changed
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed     prepare_disks_and_activate()
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed   File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/secrets-storage-relation-changed", line 449, in prepare_disks_and_activate
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed     config('osd-encrypt-keymanager'))
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed   File "lib/ceph/utils.py", line 1399, in osdize
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed     bluestore, key_manager)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed   File "lib/ceph/utils.py", line 1461, in osdize_dev
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed     key_manager)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed   File "lib/ceph/utils.py", line 1594, in _ceph_volume
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed     key_manager=key_manager))
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed   File "lib/ceph/utils.py", line 1804, in _allocate_logical_volume
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed     pv_dev = _initialize_disk(dev, dev_uuid, encrypt, key_manager)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed   File "lib/ceph/utils.py", line 1767, in _initialize_disk
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed     dev,
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed   File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed     raise CalledProcessError(retcode, cmd)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed subprocess.CalledProcessError: Command '['vaultlocker', 'encrypt', '--uuid', 'd75547e4-c753-4b0a-80cb-c0c6febd0879', '/dev/sda']' returned non-zero exit status 1
2018-07-03 19:51:04 ERROR juju.worker.uniter.operation runhook.go:113 hook "secrets-storage-relation-changed" failed: exit status 1
2018-07-03 19:51:04 DEBUG juju.worker.uniter.operation executor.go:84 lock released


Which appears to be an error related to the vault-charm not pushing a client token to the OSD?