py34 unit tests fail randomly for network.test_neutronv2

[Markus Zoeller (markus_z)]

Description
===========
The following unit tests fail **randomly** in the "gate-nova-python34" check queue:
* nova.tests.unit.network.test_neutronv2.TestNeutronv2.test_deallocate_for_instance_2_with_requested
* nova.tests.unit.network.test_neutronv2.TestNeutronv2.test_deallocate_for_instance_2
At least I don't see the root cause for this.

Steps to reproduce
==================
I discovered this with this review https://review.openstack.org/#/c/250907/3

Expected result
===============
The "gate-nova-python34" check should pass the neutron tests as the review doesn't contain any network related changes.

Actual result
=============
b'mox3.mox.ExpectedMethodCallsError: Verify: Expected methods never called:'
b" 0. Client.delete_port('my_portid1') -> None"

and

b'mox3.mox.UnexpectedMethodCallError: Unexpected method call. unexpected:- expected:+'
b"- Client.delete_port('my_portid1') -> None"
b'? ^'
b"+ Client.delete_port('my_portid2') -> None"
b'?

see http://paste.openstack.org/show/480490/

The logstash query shows a few hits since 2015-11-23 (see below)

Environment
===========
* Master code (Mitaka cycle)
* gate-nova-python34
* https://review.openstack.org/#/c/250907/3

Logs
====
* http://logs.openstack.org/07/250907/3/check/gate-nova-python34/d73accd/console.html
* http://logstash.openstack.org/#dashboard/file/logstash.json?query=+message:%5C%22Client.delete_port('my_portid1')%5C%22%20+project:%5C%22openstack/nova%5C%22
  (use the time filter from 2015-11-20 until 2015-12-01)

[Markus Zoeller (markus_z) (mzoeller)]

Elastic recheck query: https://review.openstack.org/#/c/251868/1

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/254979

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/254979
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ba01b11b8632d466ecf6c14bdc7490bfed93ec21
Submitter: Jenkins
Branch: master

commit ba01b11b8632d466ecf6c14bdc7490bfed93ec21
Author: Matt Riedemann <email address hidden>
Date: Tue Dec 8 13:39:33 2015 -0800

    neutron: skip test_deallocate_for_instance_2* in py34 job

    For whatever reason the two tests race fail in the py34 job at a rate of
    about 12 hits per week. These aren't problems in the py27 job.

    Since no one seems to be looking at these, and the unit tests should
    always be passing, let's just skip the tests for now.

    Change-Id: Ia0243d91cb00c610df1b29df983d5c6abe0a89c4
    Related-Bug: #1521599

[Lee Yarwood (lyarwood)]

I've just seen a similar random failure on gate-nova-python27-db as part of the following unrelated change against master :

* Review - https://review.openstack.org/#/c/270289/4/
* Log - http://logs.openstack.org/89/270289/6/check/gate-nova-python27-db/5e67ab7/console.html#_2016-04-27_14_48_09_614
* Logstash -
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Client.delete_port%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22%20AND%20tags%3A%5C%22console%5C%22

I've pushed an updated recheck query here for this bug using the above here :

Update query for bug 1521599
https://review.openstack.org/310750

Do we want to skip these tests on py27 while we continue to look into this?

[Brad Behle (behle)]

I hit this as well yesterday while testing a nova change locally. I looked into it and I think the problem is that two ports are being deleted in the test, but can be deleted in either order. The unit test assumes that one will be deleted before the other, so when they are deleted in the opposite order than the unit test expects, we get the following:

b'mox3.mox.UnexpectedMethodCallError: Unexpected method call. unexpected:- expected:+'
b"- Client.delete_port('my_portid1') -> None"
b'? ^'
b"+ Client.delete_port('my_portid2') -> None"
b'?

The mox unit test assumes that my_portid2 will be deleted first, but instead my_portid1 is being deleted. I think the fix for this is just to modify the unit test so it allows deletion in any order

[Markus Zoeller (markus_z) (mzoeller)]

It can be reproduced locally by executing it multiple times, for example:

#!/usr/bin/env bash

for i in `seq 1 100`;
do
    echo "The counter is $i"
    .tox/py27/bin/python2.7 -m testtools.run nova.tests.unit.network.test_neutronv2.TestNeutronv2.test_deallocate_for_instance_2_with_requested
done

This raised the error 10 times in my local test setup.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/313574

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/313574
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=94b40cf43ca3ab7d85fd15c8fda9baf8ed07f8a1
Submitter: Jenkins
Branch: master

commit 94b40cf43ca3ab7d85fd15c8fda9baf8ed07f8a1
Author: Markus Zoeller <email address hidden>
Date: Fri May 6 16:16:54 2016 +0200

    Fix TestNeutronv2.test_deallocate_for_instance_2* race failures

    The unit tests
        test_deallocate_for_instance_2_with_requested
        test_deallocate_for_instance_2
    in
        nova.tests.unit.network.test_neutronv2.TestNeutronv2
    fail randomly. An example error looks like this:

        b'mox3.mox.UnexpectedMethodCallError: Unexpected method call.
        unexpected:- expected:+'
        b"- Client.delete_port('my_portid1') -> None"
        b'? ^'
        b"+ Client.delete_port('my_portid2') -> None"
        b'?

    This is due to the mox record with expected the deletion of multiple
    ports on an instance in a specific order. This order is not important
    though and also not the same in each test case execution. To make it
    explicit in the test case, that the order is *not* important, I've
    chosen to define that in the mock records with "InAnyOrder()".

    Another solution could have been to ensure the order of the port IDs
    through the unit tests, but that could be interpreted as a precondition
    for the functional code to work correctly, which is a wrong assumption
    and therefore not chosen for this fix.

    Closes-Bug: 1521599

    Change-Id: I6ee4512d221aed48ee56f6cca19744f20f70cbe0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/314671

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314671
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3d3fb49cd1589d726925b758e8eaaa2f8a1b22d0
Submitter: Jenkins
Branch: stable/mitaka

commit 3d3fb49cd1589d726925b758e8eaaa2f8a1b22d0
Author: Markus Zoeller <email address hidden>
Date: Fri May 6 16:16:54 2016 +0200

    Fix TestNeutronv2.test_deallocate_for_instance_2* race failures

    The unit tests
        test_deallocate_for_instance_2_with_requested
        test_deallocate_for_instance_2
    in
        nova.tests.unit.network.test_neutronv2.TestNeutronv2
    fail randomly. An example error looks like this:

        b'mox3.mox.UnexpectedMethodCallError: Unexpected method call.
        unexpected:- expected:+'
        b"- Client.delete_port('my_portid1') -> None"
        b'? ^'
        b"+ Client.delete_port('my_portid2') -> None"
        b'?

    This is due to the mox record with expected the deletion of multiple
    ports on an instance in a specific order. This order is not important
    though and also not the same in each test case execution. To make it
    explicit in the test case, that the order is *not* important, I've
    chosen to define that in the mock records with "InAnyOrder()".

    Another solution could have been to ensure the order of the port IDs
    through the unit tests, but that could be interpreted as a precondition
    for the functional code to work correctly, which is a wrong assumption
    and therefore not chosen for this fix.

    Closes-Bug: 1521599

    Change-Id: I6ee4512d221aed48ee56f6cca19744f20f70cbe0

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 13.1.0 release.