nova sometimes doesn't clean up neutron ports when VM spawning fails

[Junien F]

Hi,

It appears that sometimes, nova doesn't clean up ports when spawning the instance fails. I'm using SR-IOV so I'm creating ports manually.

Example :
Create port with :
$ neutron port-create <net_id> --name direct --binding:vnic_type direct

Boot instance using said port :
$ nova boot --image <image> --flavor <flavor> --key-name admin_key --nic port-id=<port-id from above> vm_direct

If VM creation fails and the failed VM gets deleted, the port is still bound :
$ neutron port-show direct2
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | xxxxxx |
| binding:profile | {"pci_slot": "0000:04:10.2", "physical_network": "physnet1", "pci_vendor_info": "8086:10ca"} |
| binding:vif_details | {"port_filter": false, "vlan": "1234"} |
| binding:vif_type | hw_veb |
| binding:vnic_type | direct |
| device_id | 2aecc61b-e3c9-4b1f-9e47-574733705a91 |
| device_owner | compute:None |
| dns_assignment | {"hostname": "host-10-190-5-35", "ip_address": "10.190.5.35", "fqdn": "host-10-190-5-35.openstacklocal."} |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "72cfdfed-e614-4add-b880-f4c9d3bb89cc", "ip_address": "10.190.5.35"} |
| id | f34b55c1-f10e-44c9-8326-5a42996c691a |
| mac_address | fa:16:3e:5d:da:16 |
| name | direct |
| network_id | 200d501c-13df-4625-9f46-d7e28ee18dc2 |
| security_groups | feb7b440-450b-4c7b-aa3f-92f498cd2841 |
| status | BUILD |
| tenant_id | 09fae15a5f6f4acf838a97d202786d25 |
+-----------------------+-----------------------------------------------------------------------------------------------------------+

$ nova show 2aecc61b-e3c9-4b1f-9e47-574733705a91
ERROR: No server with a name or ID of '2aecc61b-e3c9-4b1f-9e47-574733705a91' exists.

This is similar to LP#1423845, but I do have the fix from that bug already in my code.

Package versions are : 2:12.0.2-0ubuntu1~cloud0, running on trusty

Thanks

[Junien F (axino)]

nova delete logs from nova-api-os-compute.log (for another instance/port with the same behaviour) :

2016-04-21 00:57:49.671 2210 DEBUG nova.api.openstack.wsgi [req-6af0e69a-6733-47bc-8283-6431bcb7bd30 2e675365e86241bb9072c0d6d1f82357 09fae15a5f6f4acf838a97d202786d25 - - -] Calling method '<bound method Controller.delete of <nova.api.openstack.compute.legacy_v2.servers.Controller object at 0x7fddf66aab90>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:792
2016-04-21 00:57:49.671 2210 DEBUG nova.compute.api [req-6af0e69a-6733-47bc-8283-6431bcb7bd30 2e675365e86241bb9072c0d6d1f82357 09fae15a5f6f4acf838a97d202786d25 - - -] [instance: 6d91af13-0884-4bd3-b170-f627a8321ddf] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2024
2016-04-21 00:57:49.740 2210 DEBUG nova.compute.api [req-6af0e69a-6733-47bc-8283-6431bcb7bd30 2e675365e86241bb9072c0d6d1f82357 09fae15a5f6f4acf838a97d202786d25 - - -] [instance: 6d91af13-0884-4bd3-b170-f627a8321ddf] Going to try to terminate instance delete /usr/lib/python2.7/dist-packages/nova/compute/api.py:1934
2016-04-21 00:57:49.825 2210 DEBUG nova.quota [req-6af0e69a-6733-47bc-8283-6431bcb7bd30 2e675365e86241bb9072c0d6d1f82357 09fae15a5f6f4acf838a97d202786d25 - - -] Attempting to reserve resources for project 09fae15a5f6f4acf838a97d202786d25 and user 2e675365e86241bb9072c0d6d1f82357. Deltas: {'instances': -1, 'ram': -2048, 'cores': -1} reserve /usr/lib/python2.7/dist-packages/nova/quota.py:550
2016-04-21 00:57:49.831 2210 DEBUG nova.quota [req-6af0e69a-6733-47bc-8283-6431bcb7bd30 2e675365e86241bb9072c0d6d1f82357 09fae15a5f6f4acf838a97d202786d25 - - -] Quota limits for project 09fae15a5f6f4acf838a97d202786d25: {'project_id': u'09fae15a5f6f4acf838a97d202786d25'} reserve /usr/lib/python2.7/dist-packages/nova/quota.py:559
2016-04-21 00:57:49.831 2210 DEBUG nova.quota [req-6af0e69a-6733-47bc-8283-6431bcb7bd30 2e675365e86241bb9072c0d6d1f82357 09fae15a5f6f4acf838a97d202786d25 - - -] Getting quotas for project 09fae15a5f6f4acf838a97d202786d25. Resources: ['instances', 'ram', 'cores'] _get_quotas /usr/lib/python2.7/dist-packages/nova/quota.py:409
2016-04-21 00:57:49.838 2210 DEBUG nova.quota [req-6af0e69a-6733-47bc-8283-6431bcb7bd30 2e675365e86241bb9072c0d6d1f82357 09fae15a5f6f4acf838a97d202786d25 - - -] Quotas for project 09fae15a5f6f4acf838a97d202786d25 after resource sync: {'instances': 10, 'ram': 51200, 'cores': 20} reserve /usr/lib/python2.7/dist-packages/nova/quota.py:565
2016-04-21 00:57:49.839 2210 DEBUG nova.quota [req-6af0e69a-6733-47bc-8283-6431bcb7bd30 2e675365e86241bb9072c0d6d1f82357 09fae15a5f6f4acf838a97d202786d25 - - -] Getting quotas for user 2e675365e86241bb9072c0d6d1f82357 and project 09fae15a5f6f4acf838a97d202786d25. Resources: ['instances', 'ram', 'cores'] _get_quotas /usr/lib/python2.7/dist-packages/nova/quota.py:401
2016-04-21 00:57:49.847 2210 DEBUG nova.quota [req-6af0e69a-6733-47bc-8283-6431bcb7bd30 2e675365e86241bb9072c0d6d1f82357 09fae15a5f6f4acf838a97d202786d25 - - -] Quotas for project 09fae15a5f6f4acf838a97d202786d25 and user 2e675365e86241bb9072c0d6d1f82357 after resource sync: {'instances': 10, 'ram': 51200, 'cores': 20} reserve /usr/...

[James Page (james-page)]

This is more of a nova problem than a charm problem, so raising a nova bug task.

[James Page (james-page)]

12.0.2 was a liberty version; I've been testing with SR-IOV ports today on Newton, and I've not observed this issue when VM creation fails; ports drop back to unbound and can be consumed straight away.

So I suspect this is actually fixed.

[Lee Yarwood (lyarwood)]

Marking this as a duplicate of bug 1648840.