Feature to detach volume from a shelved(_offloaded) instance is broken

Bug #1808089 reported by Zhenyu Zheng
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Zhenyu Zheng
Queens
Fix Committed
Medium
Matt Riedemann
Rocky
Fix Committed
Medium
Zhenyu Zheng

Bug Description

In microversion 2.20 we added the ablility to detach/attach volumes from a shelved(_offloaded) instance:
http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/rest_api_version_history.rst?h=stable/rocky#n177

But after we uses the cinder new attach workflow, this feature is broken, as when we delete the attachment the volume status changed to "reserved" and we cannot detach the volume with this status.

http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py?h=stable/rocky#n4469

Dec 11 22:20:41 ubuntu <email address hidden>[2919]: INFO cinder.api.openstack.wsgi [req-7f18da0d-44db-4445-92df-0abc790dede9 req-3f6b8e1e-4072-400f-aa47-fcbffef3fe55 admin admin] DELETE http://10.76.150.197/volume/v3/5f8092daf17b4b6a83472595eacaa219/attachments/60d6c3e1-9ced-4e64-a2aa-c5dae2248f5b
Dec 11 22:20:41 ubuntu <email address hidden>[2919]: DEBUG cinder.api.openstack.wsgi [req-7f18da0d-44db-4445-92df-0abc790dede9 req-3f6b8e1e-4072-400f-aa47-fcbffef3fe55 admin admin] Empty body provided in request {{(pid=2922) get_body /opt/stack/cinder/cinder/api/openstack/wsgi.py:719}}
Dec 11 22:20:41 ubuntu <email address hidden>[2919]: DEBUG cinder.api.openstack.wsgi [req-7f18da0d-44db-4445-92df-0abc790dede9 req-3f6b8e1e-4072-400f-aa47-fcbffef3fe55 admin admin] Calling method 'version_select' {{(pid=2922) _process_stack /opt/stack/cinder/cinder/api/openstack/wsgi.py:873}}
Dec 11 22:20:41 ubuntu <email address hidden>[2919]: DEBUG cinder.volume.api [req-7f18da0d-44db-4445-92df-0abc790dede9 req-3f6b8e1e-4072-400f-aa47-fcbffef3fe55 admin admin] Remaining volume attachments: VolumeAttachmentList(objects=[VolumeAttachment(cb3a2f68-fc14-4a2e-9b2a-78e049a1b23c)]) {{(pid=2922) attachment_delete /opt/stack/cinder/cinder/volume/api.py:2186}}
Dec 11 22:20:41 ubuntu <email address hidden>[2919]: DEBUG cinder.volume.api [req-7f18da0d-44db-4445-92df-0abc790dede9 req-3f6b8e1e-4072-400f-aa47-fcbffef3fe55 admin admin] Adding status of: reserved to pending status list for volume. {{(pid=2922) attachment_delete /opt/stack/cinder/cinder/volume/api.py:2195}}
Dec 11 22:20:41 ubuntu <email address hidden>[2919]: DEBUG cinder.volume.api [req-7f18da0d-44db-4445-92df-0abc790dede9 req-3f6b8e1e-4072-400f-aa47-fcbffef3fe55 admin admin] Pending status list for volume during attachment-delete: [u'reserved'] {{(pid=2922) attachment_delete /opt/stack/cinder/cinder/volume/api.py:2199}}
Dec 11 22:20:42 ubuntu <email address hidden>[2919]: INFO cinder.api.openstack.wsgi [req-7f18da0d-44db-4445-92df-0abc790dede9 req-3f6b8e1e-4072-400f-aa47-fcbffef3fe55 admin admin] http://10.76.150.197/volume/v3/5f8092daf17b4b6a83472595eacaa219/attachments/60d6c3e1-9ced-4e64-a2aa-c5dae2248f5b returned with HTTP 200

And when I call detach volume on this, I got:
{
    "badRequest": {
        "message": "Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-70a9bdb8-d4a5-432c-b11b-d3ce6cf179c7)",
        "code": 400
    }
}

Tags: shelve volumes
description: updated
description: updated
Revision history for this message
Matt Riedemann (mriedem) wrote :

I'm pretty sure there is a tempest integration test for the scenario you're describing:

https://github.com/openstack/tempest/blob/ce4b7d246e09b4dfcd7f9ecef3b2318b010396ac/tempest/api/compute/volumes/test_attach_volume.py#L162

Specifically test_detach_volume_shelved_or_offload_server creates a server, shelve offloads it, attaches a volume to the shelved offloaded server, then detaches the volume, and then unshelves the server.

So I'm not sure what you're describing is a break in the existing API?

When you say, "And when I call detach volume on this, I got:" - which API are you specifically referring to? A compute (nova) API or a block-storage (cinder) API?

tags: added: volumes
Changed in nova:
status: New → Incomplete
Revision history for this message
Matt Riedemann (mriedem) wrote :

Can you provide a step by step recreate scenario?

Revision history for this message
Zhenyu Zheng (zhengzhenyu) wrote :

Hi, thanks for the reply, the test step could be as follows:

1. Boot a instance
2. attach a volume to the vm
3. shelve the vm
4. try to detach the volume

Revision history for this message
Zhenyu Zheng (zhengzhenyu) wrote :

I checked the tempest cases you mentioned, the key point is that as you mentioned,
VOLUMES ARE ATTACHED AFTER THE INSTANCE GOT SHELVED, and for this kind of volume,
they are in "in-use" status. Meanning this is only testing about that people
can attach volume to shelved instances and will be able to detach these newly
attach volumes, but cannot detach volumes that are already attached to instances
before shelve.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/624959

Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
status: Incomplete → In Progress
Revision history for this message
Zhenyu Zheng (zhengzhenyu) wrote :

The resize case is OK because in compute.manager._finish_resize() we
updated the attachments with connectors, so the volumes are in
'in-use' status.

Matt Riedemann (mriedem)
tags: added: shelve
Revision history for this message
Matt Riedemann (mriedem) wrote :
Changed in nova:
importance: Undecided → Medium
Changed in nova:
assignee: Zhenyu Zheng (zhengzhenyu) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Zhenyu Zheng (zhengzhenyu)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/624959
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=41b982c9feec3105247bc72d23d1470bcabc3a0f
Submitter: Zuul
Branch: master

commit 41b982c9feec3105247bc72d23d1470bcabc3a0f
Author: Kevin_Zheng <email address hidden>
Date: Thu Dec 13 19:31:13 2018 +0800

    Don't call begin_detaching when detaching volume from shelved vm

    When shelve an instance, if the instance has volume attached,
    with new attach/detach flow, we will delete the old attachment
    and create a new attachment, the volume status will be ``reserved``.

    If the user tries to detach these volumes, it fails due to that
    Cinder does not allow a begin_detaching() call on a `reserved` volume.

    Actually for shelved instances, we can just skip this step and
    directly detach it.

    Change-Id: Ib1799feebbd8f4b0f389168939df7e5e90c8add1
    closes-bug: #1808089

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/633960

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/633962

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/633960
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e7dc43bf848105fdd24b89216676e3a56f1f6689
Submitter: Zuul
Branch: stable/rocky

commit e7dc43bf848105fdd24b89216676e3a56f1f6689
Author: Kevin_Zheng <email address hidden>
Date: Thu Dec 13 19:31:13 2018 +0800

    Don't call begin_detaching when detaching volume from shelved vm

    When shelve an instance, if the instance has volume attached,
    with new attach/detach flow, we will delete the old attachment
    and create a new attachment, the volume status will be ``reserved``.

    If the user tries to detach these volumes, it fails due to that
    Cinder does not allow a begin_detaching() call on a `reserved` volume.

    Actually for shelved instances, we can just skip this step and
    directly detach it.

    Change-Id: Ib1799feebbd8f4b0f389168939df7e5e90c8add1
    closes-bug: #1808089
    (cherry picked from commit 41b982c9feec3105247bc72d23d1470bcabc3a0f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/633962
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b8704a1d1cfe0b1fecfb21fb9373b0659f9a611d
Submitter: Zuul
Branch: stable/queens

commit b8704a1d1cfe0b1fecfb21fb9373b0659f9a611d
Author: Kevin_Zheng <email address hidden>
Date: Thu Dec 13 19:31:13 2018 +0800

    Don't call begin_detaching when detaching volume from shelved vm

    When shelve an instance, if the instance has volume attached,
    with new attach/detach flow, we will delete the old attachment
    and create a new attachment, the volume status will be ``reserved``.

    If the user tries to detach these volumes, it fails due to that
    Cinder does not allow a begin_detaching() call on a `reserved` volume.

    Actually for shelved instances, we can just skip this step and
    directly detach it.

    Change-Id: Ib1799feebbd8f4b0f389168939df7e5e90c8add1
    closes-bug: #1808089
    (cherry picked from commit 41b982c9feec3105247bc72d23d1470bcabc3a0f)
    (cherry picked from commit e7dc43bf848105fdd24b89216676e3a56f1f6689)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.0.0rc1

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.10

This issue was fixed in the openstack/nova 17.0.10 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.2.0

This issue was fixed in the openstack/nova 18.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.