Cinder do not encrypt volumes correctly(25% success ratio)

[Jan Wasilewski]

[1. Description]
I faced an issue with encryption of glance image to the volume. When glance image is small(1GB size), I observed that around 25% of my tests fails, because created volume is not encrypted even if it should be. Inside cinder logs I do not see anything suspicious, only after manual mount I can see that volume is not encrypted as it should be.

[2. Test case scenario]
I created a test script for it to measure failure ratio:

#!/bin/bash
result=result-file
> $result
for i in $(seq 1 20)
do
       openstack volume create --type LUKS --bootable --description "test-sda-volume" --size 2 --image my-image my-$i-test
       sleep 80
       openstack server create --flavor m8.c4 --network <my-net-id> --volume my-$i-test my-$i-test
       sleep 150
       openstack server show my-$i-test >> $result
       sleep 5
       openstack server delete my-$i-test
       sleep 10
       openstack volume delete my-$i-test
done

[3. Results from failure operation]
When problem occur I can see such error in nova-compute logs: https://paste.opendev.org/show/bnSUBvJtyo8zgoUwkGlH/

[4. Logs]
Beside of nova-compute logs mentioned above: https://paste.opendev.org/show/bnSUBvJtyo8zgoUwkGlH/
we can see such logs(with debug on) in cinder:
- from fail: https://paste.opendev.org/show/bVC394u7z52rK9PPtA1g/
- from correct operation: https://paste.opendev.org/show/bovL2It80CZQci7zVsU8/

I can't see any significant differences between failure and correct operation inside cinder logs.

[5. Manual mount of 'encrypted' volume]
I manually mounted broken volume and I was able to see data there:
https://paste.opendev.org/show/bOuis20HO3dls1Ed5bE8/

Similar operation for correctly encrypted volume was like that: https://paste.opendev.org/show/b2IxM19Gt6HXw2ZfHGps/

[6. SW versions]
# dpkg -l | grep cinder
ii cinder-api 2:16.4.2-0ubuntu2~cloud0 all Cinder storage service - API server
ii cinder-backup 2:16.4.2-0ubuntu2~cloud0 all Cinder storage service - Scheduler server
ii cinder-common 2:16.4.2-0ubuntu2~cloud0 all Cinder storage service - common files
ii cinder-scheduler 2:16.4.2-0ubuntu2~cloud0 all Cinder storage service - Scheduler server
ii cinder-volume 2:16.4.2-0ubuntu2~cloud0 all Cinder storage service - Volume server
ii python3-cinder 2:16.4.2-0ubuntu2~cloud0 all Cinder Python 3 libraries
ii python3-cinderclient 1:7.0.0-0ubuntu1~cloud0 all Python bindings to the OpenStack Volume API - Python 3.x

# dpkg -l | grep barbican
ii barbican-api 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - API Server
ii barbican-common 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - common files
ii barbican-keystone-listener 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - Keystone Listener
ii barbican-worker 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - Worker Node
ii python3-barbican 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - Python 3 files
ii python3-barbicanclient 4.10.0-0ubuntu1~cloud0 all OpenStack Key Management API client - Python 3.x

# dpkg -l | grep nova
ii nova-api 2:20.6.1-0ubuntu1~cloud1 all OpenStack Compute - API frontend
ii nova-common 2:20.6.1-0ubuntu1~cloud1 all OpenStack Compute - common files
ii nova-conductor 2:20.6.1-0ubuntu1~cloud1 all OpenStack Compute - conductor service
rc nova-consoleauth 2:19.3.2-0ubuntu1~cloud0 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:20.6.1-0ubuntu1~cloud1 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:20.6.1-0ubuntu1~cloud1 all OpenStack Compute - virtual machine scheduler
ii python3-nova 2:20.6.1-0ubuntu1~cloud1 all OpenStack Compute Python 3 libraries
ii python3-novaclient 2:15.1.0-0ubuntu2~cloud0 all client library for OpenStack Compute API - 3.x

In case anything else is needed, I'm ready to help

[Eric Harney (eharney)]

Thanks for the report. We have recently discovered this behavior elsewhere, it seems to be due to a race condition in device management from os-brick.

We believe that https://review.opendev.org/c/openstack/os-brick/+/836391 will fix this issue. (Being tracked in https://bugs.launchpad.net/os-brick/+bug/1967790 .)

If you could test that os-brick patch in your environment, it would be helpful to know if it fixes the issue fully for you.

[Jan Wasilewski (janwasilewski)]

Thank you for your post Eric. I took the fix which you mentioned to my environment which is up and running on ussuri right now and it is working correctly. However, I took only some of the changes as os-brick for ussuri vary a lot between master and ussuri branch:

https://review.opendev.org/c/openstack/os-brick/+/836391/8/os_brick/utils.py -> I took this as there is a main functionality of such fix

https://review.opendev.org/c/openstack/os-brick/+/836391/8/os_brick/initiator/linuxscsi.py -> as well as there is removed previous method

https://review.opendev.org/c/openstack/os-brick/+/836391/8/os_brick/privileged/rootwrap.py -> I took this as it's responsible for correct symlinks

https://review.opendev.org/c/openstack/os-brick/+/836391/8/os_brick/initiator/connectors/iscsi.py -> fix for iscsi with methods described in utils.py

I haven't changed other files(tests are more or less OK, was not crucial for my validation) as some of them are not available for ussuri(nvmof is not part of ussuri code) or some of them are not used by my env(rbd and fibre_channel).

Do you plan to apply such change for ussuri or other branches? Thanks in advance for info.

[Eric Harney (eharney)]

Fixed in os-brick as of https://review.opendev.org/#/q/Ie373ab050dcc0a35c749d9a53b6cf5ca060bcb58 .