TestRemoteInstanceObject randomly failing

[Matt Riedemann]

I started hitting this kind of failure locally, but intermittently:

nova.tests.unit.objects.test_instance.TestRemoteInstanceObject.test_migrate_flavor
----------------------------------------------------------------------------------

Captured traceback:
˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜
    Traceback (most recent call last):
      File "nova/tests/unit/objects/test_instance.py", line 1027, in test_migrate_flavor
        self.assertNotIn('instance_type_id', inst.system_metadata)
      File "/home/mriedem/git/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 392, in assertNotIn
        self.assertThat(haystack, matcher, message)
      File "/home/mriedem/git/nova/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 433, in assertThat
        raise mismatch_error
    MismatchError: {u'old_instance_type_flavorid': u'2', u'old_instance_type_vcpus': u'1', u'instance_type_name': u'm1.small', u'instance_type_extra_hw:numa_cpus.1': u'123', u'old_instance_type_vcpu_weight': u'0', u'instance_type_ephemeral_gb': u'0', u'old_instance_type_swap': u'0', u'old_instance_type_id': u'5', u'old_instance_type_ephemeral_gb': u'0', u'old_instance_type_rxtx_factor': u'1.0', u'instance_type_vcpu_weight': u'0', u'instance_type_root_gb': u'20', u'instance_type_id': u'5', u'old_instance_type_root_gb': u'20', u'instance_type_rxtx_factor': u'1.0', u'instance_type_vcpus': u'1', u'instance_type_memory_mb': u'2048', u'instance_type_swap': u'0', u'old_instance_type_memory_mb': u'2048', u'old_instance_type_name': u'm1.small', u'old_instance_type_extra_hw:numa_cpus.1': u'1', u'instance_type_flavorid': u'2'} matches Contains('instance_type_id')
    Traceback (most recent call last):
    _StringException: Empty attachments:
      stderr
      stdout

I'm also seeing it in the community jenkins test runs (31 hits on the check queue going back to 1/30):

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOm5vdmEudGVzdHMudW5pdC5vYmplY3RzLnRlc3RfaW5zdGFuY2UuVGVzdFJlbW90ZUluc3RhbmNlT2JqZWN0KiBBTkQgbWVzc2FnZTpcIkZBSUxFRFwiIEFORCB0YWdzOlwiY29uc29sZVwiIEFORCBidWlsZF9uYW1lOlwiZ2F0ZS1ub3ZhLXB5dGhvbjI3XCIgIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiJjdXN0b20iLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsiZnJvbSI6IjIwMTUtMDEtMjBUMTc6MjQ6NDcrMDA6MDAiLCJ0byI6IjIwMTUtMDItMDNUMTc6MjQ6NDcrMDA6MDAiLCJ1c2VyX2ludGVydmFsIjoiMCJ9LCJzdGFtcCI6MTQyMjk4NDQ0MjIzNywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

I believe it coincides with merging this:

https://review.openstack.org/#/c/135700/

The failure rates aren't high, but I think it's an issue.

The failures appear to only be in this remote instance objects test, so maybe something goofing up with rpc/db interactions being slowed down.

[Davanum Srinivas (DIMS) (dims-v)]

tox with --concurrency 1 will recreate this every single time!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/152731

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/152742

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/152731
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=20e7c0b63fff0e7d6f939709bdca4f1040053130
Submitter: Jenkins
Branch: master

commit 20e7c0b63fff0e7d6f939709bdca4f1040053130
Author: Dan Smith <email address hidden>
Date: Tue Feb 3 15:38:53 2015 -0800

    Fix bad mocking of methods on Instance

    Since the objects come and go from the registry during tests, mocking
    a staticmethod via path results in us not seeing the mock on the remote
    side.

    Change-Id: I046c30c5852c17012c8938b31a7d1d6bacb9e7f2
    Partial-Bug: #1417678

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/152742
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c40ae6183494d5df019de61412412b388a87d9e4
Submitter: Jenkins
Branch: master

commit c40ae6183494d5df019de61412412b388a87d9e4
Author: Dan Smith <email address hidden>
Date: Tue Feb 3 16:29:55 2015 -0800

    Fix corrupting the object repository with test instance objects

    In order to test older instance versions in the remote instance
    tests, we were creating an OldInstance object masquerading as a
    proper Instance so we could override the version. That had the
    potential to confuse other instance object tests because from that
    point on, it was the first Instance object in the repository,
    breaking the ordering requirement. Since we don't need it to be
    called Instance to test what we want to test, this patch just
    removes the masquerading bit.

    Change-Id: Iabb77ea3084c3701bf67f051ace494a1efb12917
    Closes-Bug: #1417678