Data corrupted in cinder nfs volume with encrypted volume type after detached

[Lisa Li]

Summary:
It fails to re-attach an encrypted volume created from nfs to an instance.

Env:
All in one with devstack and origin/mater repository.

Reproduce steps:
1. Create an cinder volume from nfs with encrypted volume type.
2. Attach this volume to an instance.
3. Detach it.
4. Re-attach this volume to the instance.

Expect result:
re-attach succeeds.

Actual result:
The command fails with following error mesage:
ne 89, in _open_volume^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m run_as_root=True, check_exit_code=True)^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/nova/nova/utils.py", line 389, in execute^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m return RootwrapProcessHelper().execute(*cmd, **kwargs)^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/nova/nova/utils.py", line 272, in execute^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m return processutils.execute(*cmd, **kwargs)^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 295, in execute^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m cmd=sanitized_cmd)^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mProcessExecutionError: Unexpected error while running command.^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323 volume-d55c2436-3453-47ef-977c-42ef2a334323^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mExit code: 4^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mStdout: u''^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mStderr: u"Device /dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323 doesn't exist or access denied.\n"^M

Analysis:
When cinder creates a nfs volume, it is created in the share folder. For example,
/opt/stack/data/cinder/mnt/690e8d688d986e9d44a1a00ed99912d0/volume-d55c2436-3453-47ef-977c-42ef2a334323

And when it is attached to an instance, the share folder is mounted to the compute node.
/opt/stack/data/nova/mnt/690e8d688d986e9d44a1a00ed99912d0/volume-d55c2436-3453-47ef-977c-42ef2a334323

-rw-rw-rw- 1 stack stack 1073741824 Oct 29 07:52 volume-d55c2436-3453-47ef-977c-42ef2a334323

As the volume type is encrypted, it needs to encryptors.attach_volume().
With current encryptors mechanism, the file is changed to
lrwxrwxrwx 1 nobody nogroup 55 Oct 29 05:09 volume-d55c2436-3453-47ef-977c-42ef2a334323 -> /dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323

It means the original cinder volume file is deleted, and it is a link pointed to the encrypted device.

When detached, the encrypted device(/dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323) is deleted, and the above volume-d55c2436-3453-47ef-977c-42ef2a334323 in share folder is left as a file link.

As a result, the volume is corrupted and re-attach fails

[Lisa Li (lisali)]

As Nova and Cinder use nova/volume/encryptors, I plan to move nova/volume/encryptors to os-brick. The bug will be fixed after the work.

[Augustina Ragwitz (auggy)]

Marked as confirmed because the bug has been assigned to Lisa Li and she is working on a fix.

[Anusha Unnam (anusha-unnam)]

@Lisa Li,
Are you still working on this bug?

[Lisa Li (lisali)]

Yes, I am working on this bug this release.

[Lisa Li (lisali)]

We can work together if you are interested.

[John Garbutt (johngarbutt)]

I don't see any patch upload, to un-assigning the bug.

[Lisa Li (lisali)]

The patch is submitted: https://review.openstack.org/#/c/342634/

[Sean Dague (sdague)]

Patch in merge conflict

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/342634
Reason: This review is > 6 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/452938

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lee Yarwood (<email address hidden>) on branch: master
Review: https://review.openstack.org/452938
Reason: Yup great point, moving the files around like this is going to bork any cinder operations on the volume while it is attached (snapshots etc).

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.