test_describe_instances_with_filters_tags is non deterministic

[Sean Dague]

test_describe_instances_with_filters_tags is often failing in the gate, through some non determinism in the test or the ec2 code itself. However, because all compares are done with assertJsonEqual, exactly what the data structures look like during fail isn't easy to deduce.

The following patch https://review.openstack.org/#/c/207403/ will dump more details about what's gone wrong, then hopefully we can get to the bottom of it.

[Sean Dague (sdague)]

This is our major unit test race at the moment, so marking High as I believe we should clean this up soon.

[Matt Riedemann (mriedem)]

24 hits in the last 7 days, more spiky in the last 24 hours:

http://goo.gl/yB0I71

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/207403
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=858e6bcb579717a42e0e3c0cf1f04fbc6cc62ef2
Submitter: Jenkins
Branch: master

commit 858e6bcb579717a42e0e3c0cf1f04fbc6cc62ef2
Author: Sean Dague <email address hidden>
Date: Thu Jul 30 07:03:14 2015 -0400

    return more details on assertJsonEqual fail

    assertJsonEqual does a recursive traverse of a data structure, using
    assertEqual strategically along the way. The problem with this is that
    it can lead to really cryptic messages like

      Failed '3' != '4'

    When some very deep list was the wrong number of elements. It doesn't
    tell you which list was the wrong number of elements, and so it's
    completely opaque to get to the bottom of. Especially if this doesn't
    happen on every run, just some.

    This creates a version which catches an inner match issue, and
    constructs a new failure which includes information about the 2 data
    structures being compared at the top level. This should help debugging
    racey unit test in the gate like
    test_describe_instances_with_filters_tags.

    Related-Bug: #1479780

    Change-Id: I6f6e961a3c63f9f86fe9b21ce6c16a2e634e9ce5

[Sergey Nikitin (snikitin)]

It's a problem connected with PYTHONHASHSEED.

When PYTHONHASHSEED=2772952699 or PYTHONHASHSEED=1948061410 test is failed

[Matt Riedemann (mriedem)]

Things are slightly better looking in the output:

http://logs.openstack.org/80/193380/16/check/gate-nova-python27/04522e5/console.html#_2015-07-31_01_41_42_616

2015-07-31 01:41:42.616 | {6} nova.tests.unit.api.ec2.test_cloud.CloudTestCase.test_describe_instances_with_filters_tags [1.260158s] ... FAILED
2015-07-31 01:41:42.616 |
2015-07-31 01:41:42.617 | Captured traceback:
2015-07-31 01:41:42.617 | ~~~~~~~~~~~~~~~~~~~
2015-07-31 01:41:42.617 | Traceback (most recent call last):
2015-07-31 01:41:42.617 | File "nova/tests/unit/api/ec2/test_cloud.py", line 1145, in test_describe_instances_with_filters_tags
2015-07-31 01:41:42.617 | [inst1_ret, inst2_ret]})
2015-07-31 01:41:42.617 | File "nova/test.py", line 328, in assertJsonEqual
2015-07-31 01:41:42.617 | inner(expected, observed)
2015-07-31 01:41:42.617 | File "nova/test.py", line 314, in inner
2015-07-31 01:41:42.618 | inner(expected[key], observed[key])
2015-07-31 01:41:42.618 | File "nova/test.py", line 324, in inner
2015-07-31 01:41:42.618 | next(observed_values_iter))
2015-07-31 01:41:42.618 | File "nova/test.py", line 314, in inner
2015-07-31 01:41:42.618 | inner(expected[key], observed[key])
2015-07-31 01:41:42.618 | File "nova/test.py", line 326, in inner
2015-07-31 01:41:42.618 | self.assertEqual(expected, observed)
2015-07-31 01:41:42.619 | File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
2015-07-31 01:41:42.619 | self.assertThat(observed, matcher, message)
2015-07-31 01:41:42.619 | File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
2015-07-31 01:41:42.619 | raise mismatch_error
2015-07-31 01:41:42.619 | testtools.matchers._impl.MismatchError: u'a' != u'b'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/208253

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/208253
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=33180322248edc2ff6c6fed3caf6d5cc8ad758f7
Submitter: Jenkins
Branch: master

commit 33180322248edc2ff6c6fed3caf6d5cc8ad758f7
Author: Sergey Nikitin <email address hidden>
Date: Tue Aug 4 16:15:11 2015 +0300

    Fixed random failing of test_describe_instances_with_filters_tags

    In some cases (when PYTHONHASHSEED=2772952699)
    test test_describe_instances_with_filters_tags fails.

    This happens because order of dictionaries in list
    depends on PYTHONHASHSEED. We can't sort these dictionaries
    by method sort() because dictionaries are uncomparable
    in Python 3.4.

    Added method _normalize_reservation_set to the test.
    This method sorts elements order of which could change.

    Closes-Bug: #1479780

    Co-Authored-By: Timofey Durakov <email address hidden>
    Co-Authored-By: Pavel Kholkin <email address hidden>

    Change-Id: I3af94fa8f96e239dfc64537895d06a09eae380b5