Comment 8 for bug 1579037

Instance 2b580249-9467-44d7-9e6b-73693534a5d4 was first scheduled to node-6:

2016-05-26 02:37:34.645 14354 DEBUG nova.compute.manager [req-81751add-78d4-4f53-8042-a04596d0a040 becfc7903e7e4917ab927eae3fb3bb3d 420cb14a42c84b3fbbb8891fc9329816 - - -] [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] Starting instance... _do_build_and_run_instance /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1897

port fa843bf8-01b7-424e-a705-ff8453ec743e created:

2016-05-26 02:37:37.614 14354 DEBUG keystoneauth.session [req-81751add-78d4-4f53-8042-a04596d0a040 becfc7903e7e4917ab927eae3fb3bb3d 420cb14a42c84b3fbbb8891fc9329816 - - -] RESP: [201] Date: Thu, 26 May 2016 02:37:37 GMT Connection: close Content-Type: application/json; charset=UTF-8 Content-Length: 886 X-Openstack-Request-Id: req-c801de97-029d-4927-a983-3d3db3e6f443
RESP BODY: {"port": {"status": "DOWN", "binding:host_id": "node-6.test.domain.local", "description": "", "allowed_address_pairs": [], "extra_dhcp_opts": [], "updated_at": "2016-05-26T02:37:37", "device_owner": "compute:None", "port_security_enabled": true, "binding:profile": {}, "fixed_ips": [{"subnet_id": "b888bc63-4e17-43e1-885f-aa7ba3511d33", "ip_address": "10.100.0.6"}], "id": "fa843bf8-01b7-424e-a705-ff8453ec743e", "security_groups": ["f1cb0412-1db4-4332-911b-1f9a0362f3bc"], "device_id": "2b580249-9467-44d7-9e6b-73693534a5d4", "name": "", "admin_state_up": true, "network_id": "5db2908b-a315-45e2-92ac-37472d4bdf76", "dns_name": null, "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": true}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "tenant_id": "420cb14a42c84b3fbbb8891fc9329816", "mac_address": "fa:16:3e:2c:5a:c4", "created_at": "2016-05-26T02:37:36"}}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-05-26 02:37:37.615 14354 DEBUG nova.network.neutronv2.api [req-81751add-78d4-4f53-8042-a04596d0a040 becfc7903e7e4917ab927eae3fb3bb3d 420cb14a42c84b3fbbb8891fc9329816 - - -] [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] Successfully created port: fa843bf8-01b7-424e-a705-ff8453ec743e _create_port /usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py:261

But eventually instance failed to spawn:

2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [req-81751add-78d4-4f53-8042-a04596d0a040 becfc7903e7e4917ab927eae3fb3bb3d 420cb14a42c84b3fbbb8891fc9329816 - - -] [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] Instance failed to spawn
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] Traceback (most recent call last):
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2218, in _build_resources
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] yield resources
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] block_device_info=block_device_info)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2780, in spawn
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] block_device_info=block_device_info)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4915, in _create_domain_and_network
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] xml, pause=pause, power_on=power_on)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4846, in _create_domain
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] guest.launch(pause=pause)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 142, in launch
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] self._encoded_xml, errors='ignore')
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] self.force_reraise()
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] six.reraise(self.type_, self.value, self.tb)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 137, in launch
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] return self._domain.createWithFlags(flags)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] rv = execute(f, *args, **kwargs)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] six.reraise(c, e, tb)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] rv = meth(*args, **kwargs)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1033, in createWithFlags
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] libvirtError: Unable to read from monitor: Connection reset by peer
2016-05-26 02:37:47.591 14354 ERROR nova.compute.manager [instance: 2b580249-9467-44d7-9e6b-73693534a5d4]
2016-05-26 02:37:47.595 14354 INFO nova.compute.manager [req-81751add-78d4-4f53-8042-a04596d0a040 becfc7903e7e4917ab927eae3fb3bb3d 420cb14a42c84b3fbbb8891fc9329816 - - -] [instance: 2b580249-9467-44d7-9e6b-73693534a5d4] Terminating instance

However nova-compute did not delete the port from neutron during instance termination.
Then instance was rescheduled to node-3, nova-compute created new port in neutron and successfully spawned the instance.

Nova team, please check why nova-compute on node-6 did not clear neutron port.