Neutron port got deleted when attach interface failed

[Zhenyu Zheng]

When attach an pre created Neutron port to Nova instances and got failed, the port will be deleted in Neutron:

How to reproduce:

Step 1: create a new Neutron Port:

root@SZX1000191849:/opt/stack/nova# neutron port-create e1775804-7670-482c-bf1a-309fff85f598
Created a new port:
+-----------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | unbound |
| binding:vnic_type | normal |
| created_at | 2016-11-28T02:20:26Z |
| description | |
| device_id | |
| device_owner | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "6cf2ed09-8394-4e3e-81d5-3880fe1c6984", "ip_address": "172.24.4.2"} |
| | {"subnet_id": "6b3902ca-deea-46f0-97e9-80c70243413e", "ip_address": "2001:db8::6"} |
| id | fd240a85-53cb-467a-8e43-330bb5c928ab |
| mac_address | fa:16:3e:cc:fb:9e |
| name | |
| network_id | e1775804-7670-482c-bf1a-309fff85f598 |
| port_security_enabled | True |
| project_id | 405a8b8100ae47ffaa89730681ee400f |
| revision_number | 6 |
| security_groups | 9ea518bb-ee12-489a-ac6a-3434d2c081c4 |
| status | DOWN |
| tenant_id | 405a8b8100ae47ffaa89730681ee400f |
| updated_at | 2016-11-28T02:20:27Z |
+-----------------------+------------------------------------------------------------------------------------+
root@SZX1000191849:/opt/stack/nova# neutron port-list
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------+
| 453879db-6bc2-4a40-89eb-588a15cf33c4 | | fa:16:3e:19:9e:27 | {"subnet_id": "ce211f7b-e808-4222-80e4-9bec9113afa3", "ip_address": "10.0.0.9"} |
| | | | {"subnet_id": "5ff9100a-02ff-40a2-8c33-0f0d285b72ad", "ip_address": "2001:db8:8000::8"} |
| 4cd1ad8e-86df-4454-a187-08bb3faa32ec | | fa:16:3e:60:b4:8f | {"subnet_id": "ce211f7b-e808-4222-80e4-9bec9113afa3", "ip_address": "10.0.0.13"} |
| | | | {"subnet_id": "5ff9100a-02ff-40a2-8c33-0f0d285b72ad", "ip_address": "2001:db8:8000::5"} |
| 74a0b31a-3a26-4b20-bdb2-0bb0d343c551 | | fa:16:3e:12:6b:9a | {"subnet_id": "6cf2ed09-8394-4e3e-81d5-3880fe1c6984", "ip_address": "172.24.4.10"} |
| | | | {"subnet_id": "6b3902ca-deea-46f0-97e9-80c70243413e", "ip_address": "2001:db8::a"} |
| 8fc6af24-78f8-4234-9a5a-c348ba873aff | | fa:16:3e:e3:25:62 | {"subnet_id": "ce211f7b-e808-4222-80e4-9bec9113afa3", "ip_address": "10.0.0.1"} |
| eaf6a086-a146-4e45-8c4c-a54b209e80c4 | | fa:16:3e:5d:89:bd | {"subnet_id": "ce211f7b-e808-4222-80e4-9bec9113afa3", "ip_address": "10.0.0.2"} |
| | | | {"subnet_id": "5ff9100a-02ff-40a2-8c33-0f0d285b72ad", "ip_address": "2001:db8:8000::2"} |
| ebce0f96-984b-4890-bd6b-dc561973b615 | | fa:16:3e:da:1b:7f | {"subnet_id": "5ff9100a-02ff-40a2-8c33-0f0d285b72ad", "ip_address": "2001:db8:8000::1"} |
| fd240a85-53cb-467a-8e43-330bb5c928ab | | fa:16:3e:cc:fb:9e | {"subnet_id": "6cf2ed09-8394-4e3e-81d5-3880fe1c6984", "ip_address": "172.24.4.2"} |
| | | | {"subnet_id": "6b3902ca-deea-46f0-97e9-80c70243413e", "ip_address": "2001:db8::6"} |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------+

Step 2: Attach it to the Nova instance, and error is raised by intension

root@SZX1000191849:/opt/stack/nova# nova interface-attach e1213eee-5378-472c-8f66-a35b7bdb09e3 --port-id fd240a85-53cb-467a-8e43-330bb5c928ab
ERROR (ClientException): Failed to attach network adapter device to e1213eee-5378-472c-8f66-a35b7bdb09e3 (HTTP 500) (Request-ID: req-7ab972a5-91d7-49d9-96bc-f063cf4dd1f2)

Step 3: Check the neutron ports again, the port is gone.

root@SZX1000191849:/opt/stack/nova# neutron port-list
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------+
| 453879db-6bc2-4a40-89eb-588a15cf33c4 | | fa:16:3e:19:9e:27 | {"subnet_id": "ce211f7b-e808-4222-80e4-9bec9113afa3", "ip_address": "10.0.0.9"} |
| | | | {"subnet_id": "5ff9100a-02ff-40a2-8c33-0f0d285b72ad", "ip_address": "2001:db8:8000::8"} |
| 4cd1ad8e-86df-4454-a187-08bb3faa32ec | | fa:16:3e:60:b4:8f | {"subnet_id": "ce211f7b-e808-4222-80e4-9bec9113afa3", "ip_address": "10.0.0.13"} |
| | | | {"subnet_id": "5ff9100a-02ff-40a2-8c33-0f0d285b72ad", "ip_address": "2001:db8:8000::5"} |
| 74a0b31a-3a26-4b20-bdb2-0bb0d343c551 | | fa:16:3e:12:6b:9a | {"subnet_id": "6cf2ed09-8394-4e3e-81d5-3880fe1c6984", "ip_address": "172.24.4.10"} |
| | | | {"subnet_id": "6b3902ca-deea-46f0-97e9-80c70243413e", "ip_address": "2001:db8::a"} |
| 8fc6af24-78f8-4234-9a5a-c348ba873aff | | fa:16:3e:e3:25:62 | {"subnet_id": "ce211f7b-e808-4222-80e4-9bec9113afa3", "ip_address": "10.0.0.1"} |
| eaf6a086-a146-4e45-8c4c-a54b209e80c4 | | fa:16:3e:5d:89:bd | {"subnet_id": "ce211f7b-e808-4222-80e4-9bec9113afa3", "ip_address": "10.0.0.2"} |
| | | | {"subnet_id": "5ff9100a-02ff-40a2-8c33-0f0d285b72ad", "ip_address": "2001:db8:8000::2"} |
| ebce0f96-984b-4890-bd6b-dc561973b615 | | fa:16:3e:da:1b:7f | {"subnet_id": "5ff9100a-02ff-40a2-8c33-0f0d285b72ad", "ip_address": "2001:db8:8000::1"} |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------+

This is due to when we do clean up here:
http://git.openstack.org/cgit/openstack/nova/tree/nova/network/neutronv2/api.py#n1201
we treat this port as nova-created port as it has not been included in the preexisting ports.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/403568

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/452577

[Matt Riedemann (mriedem)]

Kevin, what release did you test this against? See the questions and comments between myself and John Garbutt in the review here:

https://review.openstack.org/#/c/452577/

In short, it seems the existing code should be working. The instance.info_cache.network_info should be getting updated after the port is attached and associated with the instance, such that when we deallocate the port on failure, the network info cache should have the preserve_on_delete flag set on that VIF in the NetworkInfo object.

[Zhenyu Zheng (zhengzhenyu)]

@Matt, Hi, I was testing on Master back in Nov. 2016 so it must be Ocata.

And I tested again with the current master in my devstack env:

I raise exception.InterfaceAttachFailed(instance_uuid=instance.uuid) directly at:
http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n1392 to create the attach
failure scenario.

and then I create a port:
root@SZX1000291919:/opt/stack/nova# neutron port-create 72cc44fa-575b-4eee-8b31-62628a6849a6
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Created a new port:
+-----------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | unbound |
| binding:vnic_type | normal |
| created_at | 2017-04-19T02:06:20Z |
| description | |
| device_id | |
| device_owner | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "146fd32e-e896-4b98-8edb-60583d3fa886", "ip_address": "172.24.4.9"} |
| | {"subnet_id": "8b9f5ac6-0dfa-4c7b-bcb1-e3f10874570c", "ip_address": "2001:db8::9"} |
| id | f87cac88-0303-4e0e-9831-9e843db892ef |
| mac_address | fa:16:3e:6e:b2:03 |
| name | |
| network_id | 72cc44fa-575b-4eee-8b31-62628a6849a6 |
| port_security_enabled | True |
| project_id | 3e1e1a22750840d591509807b0251acb |
| revision_number | 6 ...

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/452577
Reason: Kevin figured out what the root issue was in https://review.openstack.org/#/c/403568/ so let's focus on that fix which is simpler.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/403568
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=115cf068a6d48cdf8b0d20a3c5a779bb8120aa9b
Submitter: Jenkins
Branch: master

commit 115cf068a6d48cdf8b0d20a3c5a779bb8120aa9b
Author: Kevin_Zheng <email address hidden>
Date: Mon Nov 28 16:12:51 2016 +0800

    Don't delete neutron port when attach failed

    Currently, when attaching neutron pre-existing
    port to an instance, if the attach failed, it
    will also be deleted in Neutron side due to
    bad judgement of the who created the port by
    reading not up-to-date info_cache.

    The workflow starts at:
    https://github.com/openstack/nova/blob/9ed0d6114/nova/network/neutronv2/api.py#L881
    ordered_ports and preexisting_port_ids are
    the same when attaching a preexisting port
    to an instance and it calls
    https://github.com/openstack/nova/blob/9ed0d6114/nova/network/base_api.py#L246
    which calls back into the neutronv2 api code
    https://github.com/openstack/nova/blob/9ed0d6114/nova/network/neutronv2/api.py#L1274
    and at this point, compute_utils.refresh_info_cache_for_instance(context,
    instance) won't have the newly attached port in it(see
    debug log: http://paste.openstack.org/show/613232/)
    because _build_network_info_model() is going to
    process it. The instance obj in memoryt with old
    info_cache will be used at rollback process and
    causing the miss-judging.

    This patch fixed it by updating instance.info_cache
    to the new ic after it is created.

    Co-Authored-By: <email address hidden>
    Change-Id: Ib323b74d4ea1e874b476ab5addfc6bc79cb7c751
    closes-bug: #1645175

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/607614

[Matt Riedemann (mriedem)]

Newton is end of life so marked as won't fix for that release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/607614
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cbf3b7c70331fc2a7e7fcf3fa2551d806000b967
Submitter: Zuul
Branch: stable/ocata

commit cbf3b7c70331fc2a7e7fcf3fa2551d806000b967
Author: Kevin_Zheng <email address hidden>
Date: Mon Nov 28 16:12:51 2016 +0800

    Don't delete neutron port when attach failed

    Currently, when attaching neutron pre-existing
    port to an instance, if the attach failed, it
    will also be deleted in Neutron side due to
    bad judgement of the who created the port by
    reading not up-to-date info_cache.

    The workflow starts at:
    https://github.com/openstack/nova/blob/9ed0d6114/nova/network/neutronv2/api.py#L881
    ordered_ports and preexisting_port_ids are
    the same when attaching a preexisting port
    to an instance and it calls
    https://github.com/openstack/nova/blob/9ed0d6114/nova/network/base_api.py#L246
    which calls back into the neutronv2 api code
    https://github.com/openstack/nova/blob/9ed0d6114/nova/network/neutronv2/api.py#L1274
    and at this point, compute_utils.refresh_info_cache_for_instance(context,
    instance) won't have the newly attached port in it(see
    debug log: http://paste.openstack.org/show/613232/)
    because _build_network_info_model() is going to
    process it. The instance obj in memoryt with old
    info_cache will be used at rollback process and
    causing the miss-judging.

    This patch fixed it by updating instance.info_cache
    to the new ic after it is created.

     Conflicts:
     doc/notification_samples/instance-shutdown-end.json

    Co-Authored-By: <email address hidden>
    Change-Id: Ib323b74d4ea1e874b476ab5addfc6bc79cb7c751
    closes-bug: #1645175
    (cherry picked from commit 115cf068a6d48cdf8b0d20a3c5a779bb8120aa9b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.5 release.