Race condition in nova/neutron when booting instance with XenAPI driver

[huan]

1. My environment is:
    Xenserver 6.5
    OpenStack latest master branch
    Neutron network with ML2 plugin, OVS driver

2. When I ran tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops it will always fail.
It failed at finding the new created instance's port before assign a floating ip to this port.

Relevant log file of tempest:
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke]
-------------------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/test.py", line 127, in wrapper
        return f(self, *func_args, **func_kwargs)
      File "tempest/scenario/test_network_basic_ops.py", line 398, in test_network_basic_ops
        self._setup_network_and_servers()
      File "tempest/scenario/test_network_basic_ops.py", line 123, in _setup_network_and_servers
        floating_ip = self.create_floating_ip(server)
      File "tempest/scenario/manager.py", line 774, in create_floating_ip
        port_id, ip4 = self._get_server_port_id_and_ip4(thing)
      File "tempest/scenario/manager.py", line 755, in _get_server_port_id_and_ip4
        % port_map)
      File "/opt/stack/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/opt/stack/tempest/.tox/all/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: 0 != 1: Found multiple IPv4 addresses: []. Unable to determine which port to target.
----------------------------------------------------------------------------------------------------------------------------------------------------

3. This is a failure that can be reproduced each time as long as we use xen and neutron.

4. I tried investagating with this problem, it dues to nova/neutron race condition when booting an instance under xen driver, since xen driver doesn't handle neutron's "network-vif-plugged" notification event.

[Václav Hejral (vhejral)]

Similar issue detected with using OVS DPDK, e.g.
http://intel-openstack-ci-logs.ovh/networking-ci/refs/changes/95/241795/2/tempest/testr_results.html

[Václav Hejral (vhejral)]

And also here: http://logs.openstack.org/76/221876/2/gate/gate-grenade-dsvm-ironic-sideways/bf65114/console.html

[Matt Riedemann (mriedem)]

Is this a duplicate of bug 1514935?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/268258

[Bob Ball (bob-ball)]

Requesting this to be RC potential because it's a long standing race condition in XenAPI driver with a relatively straight forward port of a fix over from the libvirt driver.

[Andrew Laski (alaski)]

I agree that this would be nice to fix in M if possible, but I don't think it should block a release as this is essentially new functionality that the XenAPI driver has never had.

[Andrew Laski (alaski)]

Is there a fix up for this? I see a patch for libvirt+xen but nothing for the xenapi driver.

[Bob Ball (bob-ball)]

Fix is https://review.openstack.org/#/c/241127/ - not sure why launchpad didn't show it up.

The gerrit query from the mitaka-nova-priorities-tracking etherpad did find it though (https://review.openstack.org/#/q/status:open+project:openstack/nova+%28nil+OR+message:%22#1553099%22+OR+message:%22#1552888%22+OR+message:%22#1552303%22+OR+message:%22#1549814%22+OR+message:%22#1536513%22+OR+message:%22#1512955%22+%29,n,z)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/241127
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ecd3eb7c945a7b7939c29617c2000a30a48c55dd
Submitter: Jenkins
Branch: master

commit ecd3eb7c945a7b7939c29617c2000a30a48c55dd
Author: Huan Xie <email address hidden>
Date: Tue Nov 3 06:47:28 2015 +0000

    XenAPI:Resolve Nova/Neutron race condition

    When booting an instance, nova and neutron has race condition because
    nova don't know whether vif(port) is ready in neutron. There is a
    mechenism that letting neutron notify nova when port status changed
    from down to active. This fix is for xen driver to add usage of this
    event notification to avoid race condition

    Closes-Bug: #1512955

    Change-Id: I77be3bb728db72e01701c94ee292fa0f237358ed

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0rc1 release candidate.