There are systems where it has been confirmed that creating an encrypted volume (iSCSI or FC) from an image has resulted in the data being written unencrypted on the disk.
It is expected that this same issue could happen on the Nova side when connecting volumes to instances.
This not only results in an unusable volume (since it fails to attach on an instance), but also in a security risk, because the volume now has the image data unencrypted.
Cases were this frequently happened was when Linux was running in FIPS mode.
Upon investigation, this happens because after returning from the connect_volume call to the os-brick encryptor the symlink that should be pointing to the decrypted device mapper is pointing to the raw device instead.
Upon closer inspection turns out that it's not that the os-brick encryptor code is not replacing the symlink, but that there is a race condition between the os-brick code and the system's udev rules that ends up replacing the symlink.
It all happens because when cryptsetup opens the device (already formatted or not) it generates an kernel uevent that udevd detects and ends up triggering the system udev rules.
Setting the debug mode on udev allows us to see this event:
systemd-udevd[2610688]: inotify event: 8 for /dev/sda
systemd-udevd[2610688]: device /dev/sda closed, synthesising 'change'
. . .
systemd-udevd[2139145]: found 'b8:0' claiming '/run/udev/links/\x2fdisk\x2fby-id\x2fwwn-0x6001405c74cec22c8334adc84c610bcd'
systemd-udevd[2139145]: creating link '/dev/disk/by-id/wwn-0x6001405c74cec22c8334adc84c610bcd' to '/dev/sda'
systemd-udevd[2139145]: preserve already existing symlink '/dev/disk/by-id/wwn-0x6001405c74cec22c8334adc84c610bcd' to '../../sda
The claiming message is because the device already exists and it was pointing to the cryptsetup Device Mapper device, so it's overwritting it.
So this is the race condition that happens:
- cinder calls connect_volume => returns /dev/disk/by-id/wwwn-... linking to /dev/sda
- cinder call connect_volume on the encryptor
- os-brick calls cryptsetup
- cryptsetup triggers the udev rules
- os-brick makes /dev/disk/by-id/wwwn-... link to /dev/mapper/crypt-scsi-...
- the udev rule claims the symlink and makes it point to /dev/sda again
- cinder writes data to the symlink
Since it's a race condition it doesn't happen every time, as it depends on the speed the udev rules run. If the udev rule checking the /dev/disk/by-id/www- symlink happens "before" os-brick replacing it, then everything works as expected.
Fix proposed to branch: master
Review: https:/