test_objects.test_versions fails with different hash but code is unchanged

[Matt Riedemann]

After rebasing with community I'm seeing this failure, but I haven't changed the Network object or any of it's base classes.

======================================================================
FAIL: nova.tests.objects.test_objects.TestObjectVersions.test_versions
tags: worker-0
----------------------------------------------------------------------
Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
INFO [migrate.versioning.api] 215 -> 216...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 216 -> 217...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 217 -> 218...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 218 -> 219...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 219 -> 220...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 220 -> 221...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 221 -> 222...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 222 -> 223...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 223 -> 224...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 224 -> 225...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 225 -> 226...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 226 -> 227...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 227 -> 228...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 228 -> 229...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 229 -> 230...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 230 -> 231...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 231 -> 232...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 232 -> 233...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 233 -> 234...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 234 -> 235...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 235 -> 236...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 236 -> 237...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 237 -> 238...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 238 -> 239...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 239 -> 240...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 240 -> 241...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 241 -> 242...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 242 -> 243...
INFO [migrate.versioning.api] done
INFO [migrate.versioning.api] 243 -> 244...
INFO [migrate.versioning.api] done
}}}

Traceback (most recent call last):
  File "nova/tests/objects/test_objects.py", line 941, in test_versions
    self._test_versions_cls(obj_name)
  File "nova/tests/objects/test_objects.py", line 937, in _test_versions_cls
    'has been bumped, and then update this hash') % obj_name)
  File "/opt/stack/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 321, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 406, in assertThat
    raise mismatch_error
MismatchError: !=:
reference = '1.1-faba26d0290395456f9a040584c4364b'
actual = '1.1-6d5f3c575cfc4b25db53ee5f071207ce'
: Network object has changed; please make sure the version has been bumped, and then update this hash
======================================================================
FAIL: process-returncode
tags: worker-0
----------------------------------------------------------------------
Binary content:
  traceback (test/plain; charset="utf8")
Ran 2 tests in 3.262s (+2.164s)
FAILED (id=100, failures=2)
error: testr failed (1)

Looks like it's also happening in the check queue:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTmV0d29yayBvYmplY3QgaGFzIGNoYW5nZWQ7IHBsZWFzZSBtYWtlIHN1cmUgdGhlIHZlcnNpb24gaGFzIGJlZW4gYnVtcGVkLCBhbmQgdGhlbiB1cGRhdGUgdGhpcyBoYXNoXCIgQU5EIChidWlsZF9uYW1lOmdhdGUtbm92YS1weXRob24yNiBPUiBidWlsZF9uYW1lOmdhdGUtbm92YS1weXRob24yNykgQU5EIHRhZ3M6Y29uc29sZSIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwMTU2MjMwMzg3Mn0=

10 hits in 48 hours, all failures, check queue only but different changes.

[Matt Riedemann (mriedem)]

e-r query here: https://review.openstack.org/#/c/97045/

[Matt Riedemann (mriedem)]

Looking at the changes in the check queue that are failing on this, they are these network-related changes from vishy:

https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/better-support-for-multiple-networks,n,z

But they aren't all failing, and they don't all have changes to nova.objects.network, so I still think there is probably some other issue here.

[Corey Wright (coreywright)]

https://review.openstack.org/97071

That's my attempt to remove the variability of dicts from the version test, but as I haven't seen the test fail (except when an API did change without updating the version+hash string), I can't really test it and I'm depending upon someone who has experienced the anomally.

Worse case: this doesn't solve the current failure (as it is due to something else entirely), but does prevent potential future failures.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97071
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9047165b7fed691eb89ef7322b7f4fe9b453d184
Submitter: Jenkins
Branch: master

commit 9047165b7fed691eb89ef7322b7f4fe9b453d184
Author: Corey Wright <email address hidden>
Date: Sat May 31 19:39:37 2014 -0500

    Remove variability from object change detection unit test

    Insure hashes of object representations are consistent by normalizing
    data using sorted lists (as compared to dictionaries whose keys and
    values are listed in an arbitrary, variable, dependent order:
    https://docs.python.org/2/library/stdtypes.html#dict.items).

    Closes-Bug: #1325246

    Change-Id: I1344a820263683a024818d181ec6ee08e5e18984

[Sean Dague (sdague)]

Dropped from ER tracking