encrypted cryptsetup volume attach failure in barbican-tempest-test fails intermittently

[Ade Lee]

The Barbican grenade job is failing intermittently due to a failure when we attempt to use an attached encrypted volume.

The specific test being executed is here:
https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/test_volume_encryption.py (line 161 - test_encrypted_cinder_volumes_cryptsetup)

The failing test case occurs here:

http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f

The test that fails is: test_encrypted_cinder_volumes_cryptsetup, whereas the test
test_encrypted_cinder_volumes_luks passes.

So something is going on with cryptsetup, which fails with :
Could not stat /dev/vdb --- No such file or directory

For reference, the test http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/3a71288/ succeeded, so the problem appears to be intermittent.

[Matt Riedemann (mriedem)]

Related IRC discussion about this, note the comments from lyarwood about cryptsetup being flaky in os-brick:

http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-01-14.log.html#t2019-01-14T20:38:21

There was also talk in Berlin about deprecating the cryptsetup provider:

https://etherpad.openstack.org/p/BER-volume-encryption-forum

[Lee Yarwood (lyarwood)]

Finally got around to looking at the logs here and unfortunately I still can't see anything obvious. The host correctly connects to the iSCSI volume, formats the device and opens it using cryptsetup before it's then attached to the domain by Libvirt.

We could try rescanning for devices within the instance or waiting (with a timeout) until the device shows up but there's no guarantee that it will with the expected name.

[Matt Riedemann (mriedem)]

Bug 1820007 could be related.