Nova is not marking virtual interfaces as deleted in db on instance delete

[Michael Johnson]

On nova instance delete, the virtual interface records associated with the instance are not getting marked as deleted on instance delete. This leads to a DBDuplicateEntry traceback in n-cond.log when the port is attempted to be used on another instance.

Steps to reproduce:
1. Boot up an instance with a network port.
2. Update the port with "device_id" blank.
3. Delete the nova instance.
4. Attempt to boot a new instance with the port.

Log entry of traceback:
n-cond.log:2016-07-07 20:19:24.587 TRACE nova.db.sqlalchemy.api DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry 'fa:16:3e:45:4c:1a/44aa08c2-85e5-4d70-878d-7723d96484d1-0' for key 'uniq_virtual_interfaces0address0deleted'") [SQL: u'INSERT INTO virtual_interfaces (created_at, updated_at, deleted_at, deleted, address, network_id, instance_uuid, uuid, tag) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(address)s, %(network_id)s, %(instance_uuid)s, %(uuid)s, %(tag)s)'] [parameters: {'instance_uuid': '34ca6eea-8921-4c1e-8c52-c1873e15898d', 'uuid': '44aa08c2-85e5-4d70-878d-7723d96484d1', 'network_id': None, 'created_at': datetime.datetime(2016, 7, 7, 20, 19, 24, 586359), 'updated_at': None, 'deleted': 0, 'tag': None, 'address': u'fa:16:3e:45:4c:1a/44aa08c2-85e5-4d70-878d-7723d96484d1', 'deleted_at': None}]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/341102

[Matt Riedemann (mriedem)]

We should also have a Tempest test added for this, I'll work on that also.

[Matt Riedemann (mriedem)]

Tempest test started for this here: https://review.openstack.org/#/c/341191/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/341197

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/341197
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ae9c58792c7a34f129794fa3f1928199bb859b70
Submitter: Jenkins
Branch: master

commit ae9c58792c7a34f129794fa3f1928199bb859b70
Author: Dan Smith <email address hidden>
Date: Tue Jul 12 14:39:30 2016 -0700

    Add VirtualInterface.destroy()

    This adds a destroy() method for VirtualInterface which has not been
    required before but is now.

    Change-Id: Ie00f52153a816049f8efcc9aa8071371ce0b7e5a
    Related-Bug: #1602357

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/341102
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=92a388a1e34559b2ce69d31fdef996ff029495a6
Submitter: Jenkins
Branch: master

commit 92a388a1e34559b2ce69d31fdef996ff029495a6
Author: Matt Riedemann <email address hidden>
Date: Tue Jul 12 13:17:34 2016 -0400

    neutron: delete VIFs when deallocating networking

    In e2eb6659a426bae05a54b93ebf31ff3247125258 we started creating
    VirtualInterface objects for instances/ports when allocating
    networking in the neutronv2 API code. This is used to support
    virtual device tagging.

    We need to delete the VIFs when deallocating the networking for
    the instance though otherwise we can hit unique constraint failures
    when trying to re-use an existing port from one instance to another
    since the VIF has a unique constraint on the address which is the
    port's MAC and uuid when using neutron.

    Change-Id: I2254bad0df3ccc00cd5c9438fa2684e705442e2d
    Closes-Bug: #1602357

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 14.0.0.0b2 development milestone.

[Matt Riedemann (mriedem)]

We still have a potential bug here. We create the vif object here:

https://github.com/openstack/nova/blob/92a388a1e34559b2ce69d31fdef996ff029495a6/nova/network/neutronv2/api.py#L838

but if updating the port's dns name fails, we aren't deleting the vif objects here:

https://github.com/openstack/nova/blob/92a388a1e34559b2ce69d31fdef996ff029495a6/nova/network/neutronv2/api.py#L847

Before re-raising the exception, or really if any port update fails in that loop because we could do something like:

1. create/update port1, create vif1, ok
2. create/update port2, fails - we deallocate port1 and port2 but not vif1

[Zane Bitter (zaneb)]

Saw what appears to be this issue in stable/rocky - may be related to Matt's comment above:

http://logs.openstack.org/48/638348/1/gate/heat-functional-convg-mysql-lbaasv2-py35/9d70590/logs/screen-n-api.txt.gz?level=ERROR

ERROR nova.api.openstack.wsgi [None req-e05ce059-71c4-437d-91e0-e4bc896acca6 demo demo] Unexpected exception in API method: nova.exception_Remote.VirtualInterfaceCreateException_Remote: Virtual Interface creation failed
pymysql.err.IntegrityError: (1062, "Duplicate entry 'fa:16:3e:9d:18:a6/aac0ca83-b3d2-4b28-ab15-de2d3a3e6e16-0' for key 'uniq_virtual_interfaces0address0deleted'")
oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'fa:16:3e:9d:18:a6/aac0ca83-b3d2-4b28-ab15-de2d3a3e6e16-0' for key 'uniq_virtual_interfaces0address0deleted'") [SQL: 'INSERT INTO virtual_interfaces (created_at, updated_at, deleted_at, deleted, address, network_id, instance_uuid, uuid, tag) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(address)s, %(network_id)s, %(instance_uuid)s, %(uuid)s, %(tag)s)'] [parameters: {'created_at': datetime.datetime(2019, 3, 20, 16, 11, 27, 753079), 'tag': None, 'uuid': 'aac0ca83-b3d2-4b28-ab15-de2d3a3e6e16', 'deleted_at': None, 'deleted': 0, 'address': 'fa:16:3e:9d:18:a6/aac0ca83-b3d2-4b28-ab15-de2d3a3e6e16', 'network_id': None, 'instance_uuid': '890675f9-3a1e-4a07-8bed-8648cea9fbb9', 'updated_at': None}] (Background on this error at: http://sqlalche.me/e/gkpj)

(This sequence of exceptions occurs 3 times, I assume because retrying is normally sufficient to fix a duplicate entry problem.)

The test was heat_integrationtests.functional.test_cancel_update.CancelUpdateTest.test_cancel_update_server_with_port