Volumes are not detached when deleting shelved offloaded servers

[Matt Riedemann]

I performed the following steps in a functional test within Nova:

1. Create a server and wait for it to be ACTIVE.
2. Shelve the server (it's automatically offloaded from the compute host).
3. Attach a volume to the shelved offloaded server (using microversion 2.20 or later).
4. Wait for the volume to show up as attached to the server and be in 'in-use' state.
5. Delete the server. This triggers a "local delete" in the API code since instance.host is None.

Expected result:

The volume is detached from the server and in 'available' status again.

Actual result:

The volume status is still 'in-use' even though the server is deleted.

This is a regression in Ocata with the cells v2 code that was added in the local delete flow in the API:

https://github.com/openstack/nova/blob/2a41eb8786fa1654bbecb50c2a15fdd96fdc4767/nova/compute/api.py#L1887

The problem is with cells v2 when the instance is mapped to a cell, and we do a local delete, we're destroying the instance in the cell but we aren't cleaning up the volumes like we would normally do in the old local delete flow where the instance.host is None but not mapped to a cell:

https://github.com/openstack/nova/blob/2a41eb8786fa1654bbecb50c2a15fdd96fdc4767/nova/compute/api.py#L2207

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/449334

[Matt Riedemann (mriedem)]

This was reported to me by Kevin Zheng earlier today.

[Zhenyu Zheng (zhengzhenyu)]

@Matt, I tested, and turns out the port also didn't got cleaned.

[Zhenyu Zheng (zhengzhenyu)]

root@SZX1000280461:/var/log/nova# nova list
/usr/local/lib/python2.7/dist-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
  warnings.warn(msg)
+--------------------------------------+------+--------+------------+-------------+-----------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+-----------------+
| c5cf093c-f223-44c3-8f41-aefe4a7012c2 | test | ERROR | unshelving | Shutdown | Kevin=10.0.0.75 |
+--------------------------------------+------+--------+------------+-------------+-----------------+
root@SZX1000280461:/var/log/nova# neutron port-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+------+----------------------------------+-------------------+------------------------------------------------------------------------------------------------------------+
| id | name | tenant_id | mac_address | fixed_ips |
+--------------------------------------+------+----------------------------------+-------------------+------------------------------------------------------------------------------------------------------------+
| 41da3d46-26a9-4422-afad-88fef2e211dd | | 4f705543436648abbacf5719d3397c0f | fa:16:3e:4d:8a:95 | {"subnet_id": "30e611e7-3815-4e18-a26e-7c3532458c88", "ip_address": "10.0.0.76"} |
| 486416f2-189d-4904-a90f-28739e3338bf | | | fa:16:3e:a5:18:f3 | {"subnet_id": "7ae788b2-7545-40e0-b478-4a3725a8b1ab", "ip_address": "172.24.4.10"} |
| | | | | {"subnet_id": "5f32aae0-c7c0-494f-9e84-314a266cb914", "ip_address": "2001:db8::b"} |
| 4b5d8268-9ef9-49f3-b16d-462c5af5502d | | 4f705543436648abbacf5719d3397c0f | fa:16:3e:b2:f8:19 | {"subnet_id": "30e611e7-3815-4e18-a26e-7c3532458c88", "ip_address": "10.0.0.66"} |
| 4cf9764d-84e8-43f9-8f49-d25b1bed67b5 | | 2f8692b9912d48f5a56142359ab07aa6 | fa:16:3e:e1:49:3c | {"subnet_id": "fba02ca5-3f48-4994-9289-25568c9d5256", "ip_address": "10.0.0.2"} |
| | | | | {"subnet_id": "1306f4cd-b95f-4e98-9dab-6df198e877cd", "ip_address": "fd31:6e59:2d0:0:f816:3eff:fee1:493c"} |
| 664c37f1-a118-4f15-bbd3-893e1ae384c1 | | 4f705543436648abbacf5719d3397c0f | fa:16:3e:15:18:b0 | {"subnet_id": "30e611e7-3815-4e18-a26e-7c3532458c88", "ip_address": "10.0.0.75"} <= THIS ONE |
| 6b150472-c3ce-485b-8f9d-757f1ae7f416 | | | fa:16:3e:61:62:1c | {...

[Zhenyu Zheng (zhengzhenyu)]

root@SZX1000280461:/var/log/nova# neutron port-show 664c37f1-a118-4f15-bbd3-893e1ae384c1
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+-----------------------+----------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | SZX1000280461 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| created_at | 2017-03-25T07:53:37Z |
| description | |
| device_id | c5cf093c-f223-44c3-8f41-aefe4a7012c2 |
| device_owner | compute:nova |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "30e611e7-3815-4e18-a26e-7c3532458c88", "ip_address": "10.0.0.75"} |
| id | 664c37f1-a118-4f15-bbd3-893e1ae384c1 |
| mac_address | fa:16:3e:15:18:b0 |
| name | |
| network_id | 0b0abf9b-d1ed-40f8-86b9-2143f950bee4 |
| port_security_enabled | True |
| project_id | 4f705543436648abbacf5719d3397c0f |
| revision_number | 10 |
| security_groups | 28d3e561-7cf6-4703-ac83-e364cdd4d684 |
| status | DOWN |
| tags | |
| tenant_id | 4f705543436648abbacf5719d3397c0f |
| updated_at | 2017-03-25T08:03:24Z ...

[Matt Riedemann (mriedem)]

Yeah that's because we're not calling nova.compute.api.API._local_delete which deallocates networking:

https://github.com/openstack/nova/blob/a3df985a6e966069b366c7fd170832422ccda33e/nova/compute/api.py#L2201

And also cleans up the attached volumes:

https://github.com/openstack/nova/blob/a3df985a6e966069b366c7fd170832422ccda33e/nova/compute/api.py#L2207

[Matt Riedemann (mriedem)]

Just for the order of making changes, I'm going to fix bug 1678326 before this one since the changes are in the same spot in the code.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/449334
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3ae12fdc6f428fc500f5c234f03a8ffc8286e7ba
Submitter: Jenkins
Branch: master

commit 3ae12fdc6f428fc500f5c234f03a8ffc8286e7ba
Author: Matt Riedemann <email address hidden>
Date: Thu Mar 23 15:49:27 2017 -0400

    Regression test for local delete with an attached volume

    Once an instance is in a cell and we do a local delete from
    the API, we aren't actually detaching volumes and destroying
    BDMs.

    Related-Bug: #1675570

    Change-Id: Ie3e2dfd4b0f1bb3dff4080f460bf8bb40d69f4f4

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/454413

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/454413
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2b66c6780a9bb12afcbe59b2502e01258498032e
Submitter: Jenkins
Branch: stable/ocata

commit 2b66c6780a9bb12afcbe59b2502e01258498032e
Author: Matt Riedemann <email address hidden>
Date: Thu Mar 23 15:49:27 2017 -0400

    Regression test for local delete with an attached volume

    Once an instance is in a cell and we do a local delete from
    the API, we aren't actually detaching volumes and destroying
    BDMs.

    Related-Bug: #1675570

    Change-Id: Ie3e2dfd4b0f1bb3dff4080f460bf8bb40d69f4f4
    (cherry picked from commit 3ae12fdc6f428fc500f5c234f03a8ffc8286e7ba)

[Matt Riedemann (mriedem)]

I apparently mixed up the commit message and didn't put the correct bug in there, but this is fixed with this change:

https://review.openstack.org/#/c/453859/

and that's been merged in stable/ocata as well.