From the most recent deployment we see the settle runs but is insufficient:
The format completes, the settle completes but open fails due to a missing symlink.
From nova-compute-kvm/0
2018-07-06 18:17:55 DEBUG secrets-storage-relation-changed INFO:vaultlocker.dmcrypt:LUKS formatting /dev/disk/by-dname/bcache1 using UUID:74d5e66b-c3bc-4380-a0d3-eb0878827df9 2018-07-06 18:17:59 DEBUG secrets-storage-relation-changed INFO:vaultlocker.dmcrypt:udevadm settle /dev/disk/by-uuid/74d5e66b-c3bc-4380-a0d3-eb0878827df9 2018-07-06 18:17:59 DEBUG secrets-storage-relation-changed DEBUG:urllib3.connectionpool:http://10.216.2.23:8200 "PUT /v1/charm-vaultlocker/DCS1-CLP-NOD7/74d5e66b-c3bc-4380-a0d3-eb0878827df9 HT TP/1.1" 204 0 2018-07-06 18:17:59 DEBUG secrets-storage-relation-changed DEBUG:urllib3.connectionpool:http://10.216.2.23:8200 "GET /v1/charm-vaultlocker/DCS1-CLP-NOD7/74d5e66b-c3bc-4380-a0d3-eb0878827df9 HT TP/1.1" 200 866 2018-07-06 18:17:59 DEBUG secrets-storage-relation-changed INFO:vaultlocker.dmcrypt:LUKS opening 74d5e66b-c3bc-4380-a0d3-eb0878827df9 2018-07-06 18:17:59 DEBUG secrets-storage-relation-changed Device /dev/disk/by-uuid/74d5e66b-c3bc-4380-a0d3-eb0878827df9 doesn't exist or access denied. 2018-07-06 18:17:59 DEBUG secrets-storage-relation-changed vaultlocker: Command '['cryptsetup', '--batch-mode', '--key-file', '-', 'open', 'UUID=74d5e66b-c3bc-4380-a0d3-eb0878827df9', 'crypt-7 4d5e66b-c3bc-4380-a0d3-eb0878827df9', '--type', 'luks']' returned non-zero exit status 4
I spent most of my afternoon trying to reproduce in our lab. This still feels like a race but I don't have a solution or proof yet.
From the most recent deployment we see the settle runs but is insufficient:
The format completes, the settle completes but open fails due to a missing symlink.
From nova-compute-kvm/0
2018-07-06 18:17:55 DEBUG secrets- storage- relation- changed INFO:vaultlocke r.dmcrypt: LUKS formatting /dev/disk/ by-dname/ bcache1 using UUID:74d5e66b- c3bc-4380- a0d3-eb0878827d f9 storage- relation- changed INFO:vaultlocke r.dmcrypt: udevadm settle /dev/disk/ by-uuid/ 74d5e66b- c3bc-4380- a0d3-eb0878827d f9 storage- relation- changed DEBUG:urllib3. connectionpool:http:// 10.216. 2.23:8200 "PUT /v1/charm- vaultlocker/ DCS1-CLP- NOD7/74d5e66b- c3bc-4380- a0d3-eb0878827d f9 HT storage- relation- changed DEBUG:urllib3. connectionpool:http:// 10.216. 2.23:8200 "GET /v1/charm- vaultlocker/ DCS1-CLP- NOD7/74d5e66b- c3bc-4380- a0d3-eb0878827d f9 HT storage- relation- changed INFO:vaultlocke r.dmcrypt: LUKS opening 74d5e66b- c3bc-4380- a0d3-eb0878827d f9 storage- relation- changed Device /dev/disk/ by-uuid/ 74d5e66b- c3bc-4380- a0d3-eb0878827d f9 doesn't exist or access denied. storage- relation- changed vaultlocker: Command '['cryptsetup', '--batch-mode', '--key-file', '-', 'open', 'UUID=74d5e66b- c3bc-4380- a0d3-eb0878827d f9', 'crypt-7 c3bc-4380- a0d3-eb0878827d f9', '--type', 'luks']' returned non-zero exit status 4
2018-07-06 18:17:59 DEBUG secrets-
2018-07-06 18:17:59 DEBUG secrets-
TP/1.1" 204 0
2018-07-06 18:17:59 DEBUG secrets-
TP/1.1" 200 866
2018-07-06 18:17:59 DEBUG secrets-
2018-07-06 18:17:59 DEBUG secrets-
2018-07-06 18:17:59 DEBUG secrets-
4d5e66b-
I spent most of my afternoon trying to reproduce in our lab. This still feels like a race but I don't have a solution or proof yet.