Comment 10 for bug 1357055

Except the case fixed by https://review.openstack.org/#/c/122986/, there also a race in tempest test.

The analyze is based on:
http://logs.openstack.org/73/122873/1/gate/gate-tempest-dsvm-neutron-full/e5a2bf6/console.html

In this case subnet a404dc60-62ab-4c2c-bf24-2ab33df6f176 can't be deleted.
Port 375d726d-fccd-42e4-8b5f-60a200ed76af is in this subnet, and attach to instance c65d83f5-db85-4adf-b8ae-bba2b81ac4c6

At nova api side, the port attach at:
2014-09-21 05:27:06.453 AUDIT nova.api.openstack.compute.contrib.attach_interfaces [req-d547ba16-49ec-468c-a2d3-00075880f33b TestNetworkBasicOps-1044489211 TestNetworkBasicOps-84164684] [instance: c65d83f5-db85-4adf-b8ae-bba2b81ac4c6] Attach interface

and detach at:
2014-09-21 05:27:10.937 AUDIT nova.api.openstack.compute.contrib.attach_interfaces [req-035307f9-819f-4489-aea0-ce18f00ec2f5 TestNetworkBasicOps-1044489211 TestNetworkBasicOps-84164684] [instance: c65d83f5-db85-4adf-b8ae-bba2b81ac4c6] Detach interface 375d726d-fccd-42e4-8b5f-60a200ed76af

But the detach failed
2014-09-21 05:27:11.047 ERROR oslo.messaging.rpc.dispatcher [req-035307f9-819f-4489-aea0-ce18f00ec2f5 TestNetworkBasicOps-1044489211 TestNetworkBasicOps-84164684] Exception during message handling: Port 375d726d-fccd-42e4-8b5f-60a200ed76af is not attached
2014-09-21 05:27:11.047 4710 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
....

So the reason is detach happend before attach finished.

The failed test is:

    def test_hotplug_nic(self):
        """
        1. create a new network, with no gateway (to prevent overwriting VM's
            gateway)
        2. connect VM to new network
        3. set static ip and bring new nic up
        4. check VM can ping new network dhcp port

        """
        self._setup_network_and_servers()
        self._check_public_network_connectivity(should_connect=True)
        self._create_new_network()
        self._hotplug_server()
        self._check_network_internal_connectivity(network=self.new_net)

self._hotplug_server() attach the interface, but there won't waiting for the interface attachment finished.
and 'self._check_network_internal_connectivity(network=self.new_net)' only check the interface created when boot instance and dhcp interface.

So the cleanup may happened before attach interface finished.

The cleanup is setup at 'self._hotplug_server'

        self.addCleanup(self.delete_wrapper,
                        self.interface_client.delete_interface,
                        server['id'], interface['port_id'])

But this cleanup failed as the log:
2014-09-21 05:27:11.047 ERROR oslo.messaging.rpc.dispatcher [req-035307f9-819f-4489-aea0-ce18f00ec2f5 TestNetworkBasicOps-1044489211 TestNetworkBasicOps-84164684] Exception during message handling: Port 375d726d-fccd-42e4-8b5f-60a200ed76af is not attached

And another cleanup is waiting for the interface deleted,
        self.addCleanup(self.network_client.wait_for_resource_deletion,
                        'port',
                        interface['port_id'])

We also can found this cleanup timeout finally
http://logs.openstack.org/73/122873/1/gate/gate-tempest-dsvm-neutron-full/e5a2bf6/console.html#_2014-09-21_05_41_32_226

So for this case, we should fix the tempest test, to waiting for attach interface down.