Many tests fail when run with random hash seed value

[Daniel Berrange]

When runing the test suite with a random hash seed, eg

PYTHONHASHSEED=666 ./run_tests.sh

many tests will fail as iterators over hash keys/values change their ordering. For example there are many failures in libvirt like

======================================================================
FAIL: nova.tests.virt.libvirt.test_driver.LibvirtConnTestCase.test_xml_and_uri_rescue_no_ramdisk
----------------------------------------------------------------------
Traceback (most recent call last):
_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
INFO [nova.network.driver] Loading network driver 'nova.network.linux_net'
WARNING [nova.virt.libvirt.firewall] Libvirt module could not be loaded. NWFilterFirewall will not work correctly.
INFO [nova.network.driver] Loading network driver 'nova.network.linux_net'
WARNING [nova.virt.libvirt.driver] Cannot update service status on host: fake-mini,since it is not registered.
}}}

Traceback (most recent call last):
  File "/home/berrange/src/cloud/nova/nova/tests/virt/libvirt/test_driver.py", line 2580, in test_xml_and_uri_rescue_no_ramdisk
    expect_ramdisk=False, rescue=instance_data)
  File "/home/berrange/src/cloud/nova/nova/tests/virt/libvirt/test_driver.py", line 4028, in _check_xml_and_uri
    (check(tree), expected_result, i))
  File "/home/berrange/src/cloud/nova/.venv/lib/python2.7/site-packages/testtools/testcase.py", line 321, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/berrange/src/cloud/nova/.venv/lib/python2.7/site-packages/testtools/testcase.py", line 406, in assertThat
    raise mismatch_error
MismatchError: '8388608' != '2097152': 8388608 != 2097152 failed common check 1

As well as failures across many other tests suites.

This bug is to track resolution of all problems in Nova seen when setting PYTHONHASHSEED to a non-zero value.

[Daniel Berrange (berrange)]

NB a workaround was applied in this review to force the hash seed to 0 for tox, until the proper fixes are done. https://review.openstack.org/#/c/109700/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/110605

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/110606

[Daniel Berrange (berrange)]

After the two fixes above, I see failures from the following 87 test cases which need resolving

nova.tests.api.ec2.test_cloud.CloudTestCase
    test_describe_instances_with_filters_tags FAIL
nova.tests.api.openstack.compute.contrib.test_availability_zone.AvailabilityZoneApiTest
    test_availability_zone_detail FAIL
nova.tests.api.openstack.compute.contrib.test_flavor_access.FlavorAccessSerializerTest
    test_serializer FAIL
nova.tests.api.openstack.compute.plugins.v3.test_availability_zone.AvailabilityZoneApiTest
    test_availability_zone_detail FAIL
nova.tests.api.openstack.compute.plugins.v3.test_keypairs.KeypairsTest
    test_keypair_create_with_empty_name FAIL
nova.tests.api.openstack.compute.test_versions.VersionsTest
    test_multi_choice_image FAIL
    test_multi_choice_server FAIL
nova.tests.api.openstack.test_xmlutil.SelectorTest
    test_empty_selector FAIL
nova.tests.api.openstack.compute.test_versions.VersionsTest
    test_multi_choice_image_xml FAIL
nova.tests.api.openstack.test_xmlutil.MiscellaneousXMLUtilTests
    test_make_flat_dict FAIL
nova.tests.image.test_glance.TestGlanceSerializer
    test_serialize FAIL
nova.tests.integrated.test_api_samples.AggregatesSampleXmlTest
    test_add_host FAIL
    test_add_metadata FAIL
    test_aggregate_get FAIL
    test_remove_host FAIL
    test_update_aggregate FAIL
nova.tests.integrated.test_api_samples.DiskConfigXmlTest
    test_get_image FAIL
nova.tests.integrated.test_api_samples.DiskConfigJsonTest
    test_get_image FAIL
nova.tests.integrated.test_api_samples.ImageSizeSampleXmlTests
    test_show FAIL
nova.tests.integrated.test_api_samples.ImagesSampleJsonTest
    test_image_get FAIL
nova.tests.integrated.test_api_samples.ImagesSampleXmlTest
    test_image_metadata_post FAIL
nova.tests.integrated.test_api_samples.ImageSizeSampleJsonTests
    test_show FAIL
nova.tests.integrated.test_api_samples.ImagesSampleJsonTest
    test_image_metadata_get FAIL
    test_image_metadata_post FAIL
nova.tests.integrated.test_api_samples.ImagesSampleXmlTest
    test_image_get FAIL
    test_image_meta...

[Clark Boylan (cboylan)]

This bug may be a duplicate of https://bugs.launchpad.net/keystone/+bug/1348818 however tracking nova specifics here is probably easier than on the mega thread bug. So I won't mark this as a duplicate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/112470

[Christopher Yeoh (cyeoh-0)]

https://review.openstack.org/112470 addresses the test_keypair_create_with_empty_name failure

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/112546

[Christopher Yeoh (cyeoh-0)]

https://review.openstack.org/112546 addresses the test_versions related failures

btw not intending to take the whole bug report, just working through the API related ones as I have the time.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Daniel Berrange (<email address hidden>) on branch: master
Review: https://review.openstack.org/110605
Reason: No agreement on need of this

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/112546
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8ee037a71fd0e37c94243beeda0273eacc3ba42d
Submitter: Jenkins
Branch: master

commit 8ee037a71fd0e37c94243beeda0273eacc3ba42d
Author: Chris Yeoh <email address hidden>
Date: Thu Aug 7 16:20:03 2014 +0930

    Makes versions API output deterministic

    Makes the order of available versions that the API version
    information is returned to clients deterministic. This is
    necessary to ensure that with PYTHONHASHSEED to fixed to 0 the
    unittests will consistently pass.

    Change-Id: I5b7f6b7f061424adc2263c485562e72c7cc29255
    Partial-Bug: #1350287

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/112470
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e7840303a683d91b01a8c73328f93419c41d7815
Submitter: Jenkins
Branch: master

commit e7840303a683d91b01a8c73328f93419c41d7815
Author: Chris Yeoh <email address hidden>
Date: Thu Aug 7 14:08:13 2014 +0930

    Make API name validation failure deterministic

    Makes the testing of a valid name in the API deterministic
    with respect to failing when an empty name is supplied
    by the user. Formerly it could fail either of two constraints
    - must be at least one character long
    - does not match the valid regexp which requires one character

    The lack of deterministic behaviour was causing issues with
    some unittests when PYTHONHASHSEED is not set to 0 because
    of the two failure modes. The second constraint for validation
    of names is now modified so it allows empty strings but overall the
    behaviour remains the same because of the first constraint.

    Change-Id: Iea89bb329a8939af5168926717b5c13bdf2a7f62
    Partial-Bug: 1350287

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/110606
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6589a36fbbdd4b0953683d0ba1a34679eb0a540d
Submitter: Jenkins
Branch: master

commit 6589a36fbbdd4b0953683d0ba1a34679eb0a540d
Author: Daniel P. Berrange <email address hidden>
Date: Wed Jul 30 12:02:27 2014 +0100

    tests: fix test_compute to have predictable service list

    The _create_service_entries() method in test_compute.py accepted
    a dict for the list of services. The tests would then pull out
    the first item in the list of keys of the dict. When the python
    hash seed is set to a random value the list of keys has no
    predictable ordering, causing the test suite to break. Changing
    the data structure to use a list of lists instead of a dict
    ensures strict ordering.

    Related-bug: #1350287
    Change-Id: I01a0d4cfb086eb94b49d25b65daf08e55e2087b4

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/144379

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144778
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b8885fc6ec3160dc858dce7a21f83ea1d7458118
Submitter: Jenkins
Branch: master

commit b8885fc6ec3160dc858dce7a21f83ea1d7458118
Author: Marian Horban <email address hidden>
Date: Fri Jan 2 10:33:02 2015 -0500

    Expansion of matching XML strings logic

    Extended class XMLMatcher with following features:
    1. Ability to ignore ordering of XML nodes;
    2. Matching version and encoding of XMLs;
    3. Ability to skip empty text nodes(nodes which contain white spaces only)

    Related-Bug: #1350287
    Change-Id: I74b0660d2b09b3c6251ec0eabf37978fa5d03004

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kevin L. Mitchell (<email address hidden>) on branch: master
Review: https://review.openstack.org/144379
Reason: At my suggestion, XMLMatches was updated to include the desired functionality, but it appears this change didn't get abandoned when the other change went in…