vaultlocker: /var/lib/nova/instances fails to mount at boot

[Frode Nordahl]

$ journalctl -xe -u var-lib-nova-instances.mount
-- Reboot --
Jan 11 16:09:25 ps5-ra1-n5 systemd[1]: Dependency failed for /var/lib/nova/instances.
-- Subject: A start job for unit var-lib-nova-instances.mount has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit var-lib-nova-instances.mount has finished with a failure.
--
-- The job identifier is 29 and the job result is dependency.
Jan 11 16:09:25 ps5-ra1-n5 systemd[1]: var-lib-nova-instances.mount: Job var-lib-nova-instances.mount/start failed with result 'dependency'.

$ journalctl -xe -u [e-mail]
-- Reboot --
Jan 11 16:10:00 ps5-ra1-n5 systemd[1]: Starting vaultlocker retrieve: 6da89b4b-c6aa-4895-b2a5-be9611c86c31...
-- Subject: A start job for unit [e-mail] has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit [e-mail] has begun execution.
--
-- The job identifier is 30.
Jan 11 16:10:00 ps5-ra1-n5 sh[4071]: DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): vault.ps5.canonical.com:8200
Jan 11 16:10:10 ps5-ra1-n5 sh[4071]: DEBUG:urllib3.connectionpool:https://vault.ps5.canonical.com:8200 "POST /v1/auth/approle/login HTTP/1.1" 200 518
Jan 11 16:10:10 ps5-ra1-n5 sh[4071]: INFO:vaultlocker.shell:Checking if /dev/mapper/crypt-6da89b4b-c6aa-4895-b2a5-be9611c86c31 exists.
Jan 11 16:10:10 ps5-ra1-n5 sh[4071]: DEBUG:urllib3.connectionpool:https://vault.ps5.canonical.com:8200 "GET /v1/charm-vaultlocker/ps5-ra1-n5/6da89b4b-c6aa-4895-b2a5-be9611c86c31 HTTP/1.1" 200 866
Jan 11 16:10:10 ps5-ra1-n5 sh[4071]: INFO:vaultlocker.dmcrypt:LUKS opening 6da89b4b-c6aa-4895-b2a5-be9611c86c31
Jan 11 16:10:16 ps5-ra1-n5 systemd[1]: [e-mail]: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit [e-mail] has successfully entered the 'dead' state.
Jan 11 16:10:16 ps5-ra1-n5 systemd[1]: Finished vaultlocker retrieve: 6da89b4b-c6aa-4895-b2a5-be9611c86c31.
-- Subject: A start job for unit [e-mail] has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit [e-mail] has finished successfully.
--
-- The job identifier is 30.

Made an attempt at adding a timeout and retry to the fstab entry, but that did not appear to have any effect.

Could this be a pure ordering issue that happens on this system for some reason?

[Frode Nordahl (fnordahl)]

The `vaultlocker-decrypt@.service` is running after networking, and that might be at odds with when systemd schedules the fstab compat/auto entries.

Creating systemd mount files directly on recent systems instead of relying on the fstab compat thing could be an approach to get more control of ordering.

[James Page (james-page)]

The charm adds some options to the fstab entry that should ensure that the vaultlocker unit is ordered before the mount executes (x-systemd.requires)