With the error in vault, we initialized vault and unsealed. vault showed it is in ready status. The error from juju log show uuid is d045f2c8-b705-4113-9291-2ef203600fb0 but this uuid does not exist. Where does the charm gets d045f2c8-b705-4113-9291-2ef203600fb0 from? It seems that with thedac's proposed fix, it did have the correct symlink for uuid 4c151bc2-8ad8-4c18-b4e8-58b22a66a6b2 but when the charm ran, it used uuid d045f2c8-b705-4113-9291-2ef203600fb0. /etc/udev/rules.d# cat bcache1.rules SUBSYSTEM=="block", ACTION=="add|change", ENV{CACHED_UUID}=="4c151bc2-8ad8-4c18-b4e8-58b22a66a6b2", SYMLINK+="disk/by-dname/bcache1" hook failed: "secrets-storage-relation-changed" 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed vaultlocker: Command '['cryptsetup', '--batch-mode', '--key-file', '-', 'open', 'UUID=d045f2c8-b705-4113-9291-2ef203600fb0', 'crypt-d045f2c8-b705-4113-9291-2ef203600fb0', '--type', 'luks']' returned non-zero exit status 4 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed Traceback (most recent call last): 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-5/charm/hooks/secrets-storage-relation-changed", line 579, in 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed main() 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-5/charm/hooks/secrets-storage-relation-changed", line 572, in main 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed hooks.execute(sys.argv) 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-5/charm/hooks/charmhelpers/core/hookenv.py", line 823, in execute 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed self._hooks[hook_name]() 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-5/charm/hooks/secrets-storage-relation-changed", line 556, in secrets_storage_changed 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed configure_local_ephemeral_storage() 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-5/charm/hooks/nova_compute_utils.py", line 883, in configure_local_ephemeral_storage 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed dev]) 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed File "/usr/lib/python3.5/subprocess.py", line 581, in check_call 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed raise CalledProcessError(retcode, cmd) 2018-07-06 07:54:37 DEBUG secrets-storage-relation-changed subprocess.CalledProcessError: Command '['vaultlocker', 'encrypt', '--uuid', 'd045f2c8-b705-4113-9291-2ef203600fb0', '/dev/disk/by-dname/bcache1']' returned non-zero exit status 1 =================== # vaultlocker encrypt --uuid d045f2c8-b705-4113-9291-2ef203600fb0 /dev/disk/by-dname/bcache1 DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 10.216.2.23 DEBUG:urllib3.connectionpool:http://10.216.2.23:8200 "POST /v1/auth/approle/login HTTP/1.1" 200 437 INFO:vaultlocker.dmcrypt:LUKS formatting /dev/disk/by-dname/bcache1 using UUID:d045f2c8-b705-4113-9291-2ef203600fb0 INFO:vaultlocker.dmcrypt:udevadm settle /dev/disk/by-uuid/d045f2c8-b705-4113-9291-2ef203600fb0 DEBUG:urllib3.connectionpool:http://10.216.2.23:8200 "PUT /v1/charm-vaultlocker/DCS1-CLP-NOD17/d045f2c8-b705-4113-9291-2ef203600fb0 HTTP/1.1" 204 0 DEBUG:urllib3.connectionpool:http://10.216.2.23:8200 "GET /v1/charm-vaultlocker/DCS1-CLP-NOD17/d045f2c8-b705-4113-9291-2ef203600fb0 HTTP/1.1" 200 866 INFO:vaultlocker.dmcrypt:LUKS opening d045f2c8-b705-4113-9291-2ef203600fb0 Device /dev/disk/by-uuid/d045f2c8-b705-4113-9291-2ef203600fb0 doesn't exist or access denied. vaultlocker: Command '['cryptsetup', '--batch-mode', '--key-file', '-', 'open', 'UUID=d045f2c8-b705-4113-9291-2ef203600fb0', 'crypt-d045f2c8-b705-4113-9291-2ef203600fb0', '--type', 'luks']' returned non-zero exit status 4 # cryptsetup --batch-mode --key-file - open UUID=d045f2c8-b705-4113-9291-2ef203600fb0 crypt-d045f2c8-b705-4113-9291-2ef203600fb0 --type luks Device /dev/disk/by-uuid/d045f2c8-b705-4113-9291-2ef203600fb0 doesn't exist or access denied. /dev/bcache# ls by-label by-uuid root@DCS1-CLP-NOD17:/dev/bcache# ls -al /dev/disk/by-uuid/* lrwxrwxrwx 1 root root 10 Jul 5 22:52 /dev/disk/by-uuid/05EC-7CF7 -> ../../sda1 lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/2d8a4580-df8a-4e79-9189-8931f66dc978 -> ../../sdf lrwxrwxrwx 1 root root 15 Jul 5 22:52 /dev/disk/by-uuid/38be3b4b-90dd-4bc3-8030-668042b4d572 -> ../../nvme0n1p1 lrwxrwxrwx 1 root root 10 Jul 5 22:52 /dev/disk/by-uuid/4c151bc2-8ad8-4c18-b4e8-58b22a66a6b2 -> ../../sda4 lrwxrwxrwx 1 root root 15 Jul 5 22:52 /dev/disk/by-uuid/576c447b-1e76-438a-8158-0b316e4f5c12 -> ../../nvme0n1p2 lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/58ca75fe-0bf2-4543-9a7e-151414193814 -> ../../sdd lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/8c7cb4cb-0e29-44e3-87ab-339c64400b9c -> ../../sdc lrwxrwxrwx 1 root root 10 Jul 5 22:52 /dev/disk/by-uuid/b1940fc7-80aa-49cc-b528-d46dc05fe7b5 -> ../../sda2 lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/b454f3b4-b17d-4ba8-b13a-001b9aedfbbe -> ../../sdg lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/cd955f04-0cbe-4f66-8ba9-198d3058f098 -> ../../sde lrwxrwxrwx 1 root root 13 Jul 5 22:52 /dev/disk/by-uuid/d0fbbf40-5a04-4152-9937-a2164c95bb8e -> ../../bcache7 lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/fac76349-eab1-4574-9fda-455126075b7f -> ../../sdb lrwxrwxrwx 1 root root 10 Jul 5 22:52 /dev/disk/by-uuid/fd6432a5-e436-450c-a240-3098024351d5 -> ../../sda3 ================== # cryptsetup luksDump /dev/disk/by-dname/bcache1 LUKS header information for /dev/disk/by-dname/bcache1 Version: 1 Cipher name: aes Cipher mode: xts-plain64 Hash spec: sha1 Payload offset: 4096 MK bits: 256 MK digest: a8 2c 89 2f 32 a3 30 99 ea 10 15 89 af 4b 24 e7 cd 07 4d a6 MK salt: 07 5c 0f 31 e2 e8 17 3b ca 95 10 33 b6 aa 17 c5 cc df 46 eb 99 1b ad b5 72 18 ff b5 79 b1 b4 64 MK iterations: 142125 UUID: e466562f-50f8-4b70-8842-9723f41be266 Key Slot 0: ENABLED Iterations: 575280 Salt: 21 b4 37 78 d3 47 21 d4 b2 3e 96 33 e5 30 fe 67 af 7f 65 d6 57 da b6 69 bb d2 71 77 a2 92 09 58 Key material offset: 8 AF stripes: 4000 Key Slot 1: DISABLED Key Slot 2: DISABLED Key Slot 3: DISABLED Key Slot 4: DISABLED Key Slot 5: DISABLED Key Slot 6: DISABLED Key Slot 7: DISABLED ===================== # ls -al /dev/disk/by-uuid/* lrwxrwxrwx 1 root root 10 Jul 5 22:52 /dev/disk/by-uuid/05EC-7CF7 -> ../../sda1 lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/2d8a4580-df8a-4e79-9189-8931f66dc978 -> ../../sdf lrwxrwxrwx 1 root root 15 Jul 5 22:52 /dev/disk/by-uuid/38be3b4b-90dd-4bc3-8030-668042b4d572 -> ../../nvme0n1p1 lrwxrwxrwx 1 root root 10 Jul 5 22:52 /dev/disk/by-uuid/4c151bc2-8ad8-4c18-b4e8-58b22a66a6b2 -> ../../sda4 lrwxrwxrwx 1 root root 15 Jul 5 22:52 /dev/disk/by-uuid/576c447b-1e76-438a-8158-0b316e4f5c12 -> ../../nvme0n1p2 lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/58ca75fe-0bf2-4543-9a7e-151414193814 -> ../../sdd lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/8c7cb4cb-0e29-44e3-87ab-339c64400b9c -> ../../sdc lrwxrwxrwx 1 root root 10 Jul 5 22:52 /dev/disk/by-uuid/b1940fc7-80aa-49cc-b528-d46dc05fe7b5 -> ../../sda2 lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/b454f3b4-b17d-4ba8-b13a-001b9aedfbbe -> ../../sdg lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/cd955f04-0cbe-4f66-8ba9-198d3058f098 -> ../../sde lrwxrwxrwx 1 root root 13 Jul 5 22:52 /dev/disk/by-uuid/d0fbbf40-5a04-4152-9937-a2164c95bb8e -> ../../bcache7 lrwxrwxrwx 1 root root 9 Jul 5 22:52 /dev/disk/by-uuid/fac76349-eab1-4574-9fda-455126075b7f -> ../../sdb lrwxrwxrwx 1 root root 10 Jul 5 22:52 /dev/disk/by-uuid/fd6432a5-e436-450c-a240-3098024351d5 -> ../../sda3 =========== /etc/udev/rules.d# cat bcache1.rules SUBSYSTEM=="block", ACTION=="add|change", ENV{CACHED_UUID}=="4c151bc2-8ad8-4c18-b4e8-58b22a66a6b2", SYMLINK+="disk/by-dname/bcache1"