Data corrupted in cinder nfs volume with encrypted volume type after detached

Bug #1511255 reported by Lisa Li
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
os-brick
Confirmed
High
Unassigned

Bug Description

Summary:
It fails to re-attach an encrypted volume created from nfs to an instance.

Env:
All in one with devstack and origin/mater repository.

Reproduce steps:
1. Create an cinder volume from nfs with encrypted volume type.
2. Attach this volume to an instance.
3. Detach it.
4. Re-attach this volume to the instance.

Expect result:
re-attach succeeds.

Actual result:
The command fails with following error mesage:
ne 89, in _open_volume^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m run_as_root=True, check_exit_code=True)^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/nova/nova/utils.py", line 389, in execute^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m return RootwrapProcessHelper().execute(*cmd, **kwargs)^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/opt/stack/nova/nova/utils.py", line 272, in execute^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m return processutils.execute(*cmd, **kwargs)^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 295, in execute^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00m cmd=sanitized_cmd)^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mProcessExecutionError: Unexpected error while running command.^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksOpen --key-file=- /dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323 volume-d55c2436-3453-47ef-977c-42ef2a334323^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mExit code: 4^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mStdout: u''^M
^[[01;31m2015-10-29 08:09:08.771 TRACE oslo_messaging.rpc.dispatcher ^[[01;35m^[[00mStderr: u"Device /dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323 doesn't exist or access denied.\n"^M

Analysis:
When cinder creates a nfs volume, it is created in the share folder. For example,
/opt/stack/data/cinder/mnt/690e8d688d986e9d44a1a00ed99912d0/volume-d55c2436-3453-47ef-977c-42ef2a334323

And when it is attached to an instance, the share folder is mounted to the compute node.
/opt/stack/data/nova/mnt/690e8d688d986e9d44a1a00ed99912d0/volume-d55c2436-3453-47ef-977c-42ef2a334323

-rw-rw-rw- 1 stack stack 1073741824 Oct 29 07:52 volume-d55c2436-3453-47ef-977c-42ef2a334323

As the volume type is encrypted, it needs to encryptors.attach_volume().
With current encryptors mechanism, the file is changed to
lrwxrwxrwx 1 nobody nogroup 55 Oct 29 05:09 volume-d55c2436-3453-47ef-977c-42ef2a334323 -> /dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323

It means the original cinder volume file is deleted, and it is a link pointed to the encrypted device.

When detached, the encrypted device(/dev/mapper/volume-d55c2436-3453-47ef-977c-42ef2a334323) is deleted, and the above volume-d55c2436-3453-47ef-977c-42ef2a334323 in share folder is left as a file link.

As a result, the volume is corrupted and re-attach fails

Lisa Li (lisali)
Changed in nova:
assignee: nobody → Lisa Li (lisali)
Revision history for this message
Lisa Li (lisali) wrote :

As Nova and Cinder use nova/volume/encryptors, I plan to move nova/volume/encryptors to os-brick. The bug will be fixed after the work.

Revision history for this message
Augustina Ragwitz (auggy) wrote :

Marked as confirmed because the bug has been assigned to Lisa Li and she is working on a fix.

Changed in nova:
status: New → Confirmed
tags: added: volumes
removed: encryption
Revision history for this message
Anusha Unnam (anusha-unnam) wrote :

@Lisa Li,
Are you still working on this bug?

Revision history for this message
Lisa Li (lisali) wrote :

Yes, I am working on this bug this release.

Revision history for this message
Lisa Li (lisali) wrote :

We can work together if you are interested.

Revision history for this message
John Garbutt (johngarbutt) wrote :

I don't see any patch upload, to un-assigning the bug.

Changed in nova:
assignee: Lisa Li (lisali) → nobody
tags: added: nfs
Changed in nova:
importance: Undecided → High
Revision history for this message
Lisa Li (lisali) wrote :
Changed in nova:
assignee: nobody → Lisa Li (lisali)
tags: added: encryption
Lisa Li (lisali)
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
Sean Dague (sdague) wrote :

Patch in merge conflict

Changed in nova:
assignee: Lisa Li (lisali) → nobody
status: In Progress → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/342634
Reason: This review is > 6 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/452938

Changed in nova:
assignee: nobody → Lee Yarwood (lyarwood)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Lee Yarwood (<email address hidden>) on branch: master
Review: https://review.openstack.org/452938
Reason: Yup great point, moving the files around like this is going to bork any cinder operations on the volume while it is attached (snapshots etc).

Revision history for this message
Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

Changed in nova:
status: In Progress → Confirmed
assignee: Lee Yarwood (lyarwood) → nobody
Lee Yarwood (lyarwood)
no longer affects: os-brick
affects: nova → os-brick
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.