My suspicion is that vault token create -use-limit=1 -ttl=10m is creating a single use token and the juju action is performing more than one request to vault so the subsequent ones are rejected.
So instead I ran vault token create -ttl=10m which creates an unlimited use token. By using this the action returns completed.
However all the OSDs immediatly drop into error state with "hook failed: "secrets-storage-relation-changed""
They have the following errors in their logs:
2018-07-03 19:51:03 DEBUG secrets-storage-relation-changed Failed to find physical volume "/dev/sda".
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed Device /dev/sda is not a valid LUKS device.
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 172.17.20.43
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed DEBUG:urllib3.connectionpool:http://172.17.20.43:8200 "POST /v1/auth/approle/login HTTP/1.1" 400 36
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed vaultlocker: missing client token
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed Traceback (most recent call last):
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/secrets-storage-relation-changed", line 630, in <module>
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed hooks.execute(sys.argv)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/charmhelpers/core/hookenv.py", line 823, in execute
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed self._hooks[hook_name]()
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/secrets-storage-relation-changed", line 574, in secrets_storage_changed
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed prepare_disks_and_activate()
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-sas-ceph-osd-4/charm/hooks/secrets-storage-relation-changed", line 449, in prepare_disks_and_activate
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed config('osd-encrypt-keymanager'))
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "lib/ceph/utils.py", line 1399, in osdize
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed bluestore, key_manager)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "lib/ceph/utils.py", line 1461, in osdize_dev
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed key_manager)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "lib/ceph/utils.py", line 1594, in _ceph_volume
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed key_manager=key_manager))
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "lib/ceph/utils.py", line 1804, in _allocate_logical_volume
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed pv_dev = _initialize_disk(dev, dev_uuid, encrypt, key_manager)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "lib/ceph/utils.py", line 1767, in _initialize_disk
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed dev,
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed raise CalledProcessError(retcode, cmd)
2018-07-03 19:51:04 DEBUG secrets-storage-relation-changed subprocess.CalledProcessError: Command '['vaultlocker', 'encrypt', '--uuid', 'd75547e4-c753-4b0a-80cb-c0c6febd0879', '/dev/sda']' returned non-zero exit status 1
2018-07-03 19:51:04 ERROR juju.worker.uniter.operation runhook.go:113 hook "secrets-storage-relation-changed" failed: exit status 1
2018-07-03 19:51:04 DEBUG juju.worker.uniter.operation executor.go:84 lock released
Which appears to be an error related to the vault-charm not pushing a client token to the OSD?
My suspicion is that vault token create -use-limit=1 -ttl=10m is creating a single use token and the juju action is performing more than one request to vault so the subsequent ones are rejected.
So instead I ran vault token create -ttl=10m which creates an unlimited use token. By using this the action returns completed.
However all the OSDs immediatly drop into error state with "hook failed: "secrets- storage- relation- changed" "
They have the following errors in their logs:
2018-07-03 19:51:03 DEBUG secrets- storage- relation- changed Failed to find physical volume "/dev/sda". storage- relation- changed Device /dev/sda is not a valid LUKS device. storage- relation- changed DEBUG:urllib3. connectionpool: Starting new HTTP connection (1): 172.17.20.43 storage- relation- changed DEBUG:urllib3. connectionpool:http:// 172.17. 20.43:8200 "POST /v1/auth/ approle/ login HTTP/1.1" 400 36 storage- relation- changed vaultlocker: missing client token storage- relation- changed Traceback (most recent call last): storage- relation- changed File "/var/lib/ juju/agents/ unit-sas- ceph-osd- 4/charm/ hooks/secrets- storage- relation- changed" , line 630, in <module> storage- relation- changed hooks.execute( sys.argv) storage- relation- changed File "/var/lib/ juju/agents/ unit-sas- ceph-osd- 4/charm/ hooks/charmhelp ers/core/ hookenv. py", line 823, in execute storage- relation- changed self._hooks[ hook_name] () storage- relation- changed File "/var/lib/ juju/agents/ unit-sas- ceph-osd- 4/charm/ hooks/secrets- storage- relation- changed" , line 574, in secrets_ storage_ changed storage- relation- changed prepare_ disks_and_ activate( ) storage- relation- changed File "/var/lib/ juju/agents/ unit-sas- ceph-osd- 4/charm/ hooks/secrets- storage- relation- changed" , line 449, in prepare_ disks_and_ activate storage- relation- changed config( 'osd-encrypt- keymanager' )) storage- relation- changed File "lib/ceph/ utils.py" , line 1399, in osdize storage- relation- changed bluestore, key_manager) storage- relation- changed File "lib/ceph/ utils.py" , line 1461, in osdize_dev storage- relation- changed key_manager) storage- relation- changed File "lib/ceph/ utils.py" , line 1594, in _ceph_volume storage- relation- changed key_manager= key_manager) ) storage- relation- changed File "lib/ceph/ utils.py" , line 1804, in _allocate_ logical_ volume storage- relation- changed pv_dev = _initialize_ disk(dev, dev_uuid, encrypt, key_manager) storage- relation- changed File "lib/ceph/ utils.py" , line 1767, in _initialize_disk storage- relation- changed dev, storage- relation- changed File "/usr/lib/ python3. 5/subprocess. py", line 581, in check_call storage- relation- changed raise CalledProcessEr ror(retcode, cmd) storage- relation- changed subprocess. CalledProcessEr ror: Command '['vaultlocker', 'encrypt', '--uuid', 'd75547e4- c753-4b0a- 80cb-c0c6febd08 79', '/dev/sda']' returned non-zero exit status 1 uniter. operation runhook.go:113 hook "secrets- storage- relation- changed" failed: exit status 1 uniter. operation executor.go:84 lock released
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 DEBUG secrets-
2018-07-03 19:51:04 ERROR juju.worker.
2018-07-03 19:51:04 DEBUG juju.worker.
Which appears to be an error related to the vault-charm not pushing a client token to the OSD?