nova.tests.unit.conductor.test_conductor.ConductorTaskAPITestCase.test_rebuild_instance_with_scheduler_group_failure race fails in the gate

[Matt Riedemann]

http://logs.openstack.org/30/164330/2/gate/gate-nova-python27/19869e2/console.html#_2015-03-19_15_30_04_664

2015-03-19 15:30:04.664 | {5} nova.tests.unit.conductor.test_conductor.ConductorTaskAPITestCase.test_rebuild_instance_with_scheduler_group_failure [0.396073s] ... FAILED
2015-03-19 15:30:04.664 |
2015-03-19 15:30:04.664 | Captured traceback:
2015-03-19 15:30:04.664 | ~~~~~~~~~~~~~~~~~~~
2015-03-19 15:30:04.664 | Traceback (most recent call last):
2015-03-19 15:30:04.664 | File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/mock.py", line 1201, in patched
2015-03-19 15:30:04.664 | return func(*args, **keywargs)
2015-03-19 15:30:04.664 | File "nova/tests/unit/conductor/test_conductor.py", line 1746, in test_rebuild_instance_with_scheduler_group_failure
2015-03-19 15:30:04.664 | exception, request_spec)
2015-03-19 15:30:04.664 | File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/mock.py", line 845, in assert_called_once_with
2015-03-19 15:30:04.665 | raise AssertionError(msg)
2015-03-19 15:30:04.665 | AssertionError: Expected to be called once. Called 2 times.

Seeing it on two changes so far:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwibm92YS50ZXN0cy51bml0LmNvbmR1Y3Rvci50ZXN0X2NvbmR1Y3Rvci5Db25kdWN0b3JUYXNrQVBJVGVzdENhc2UudGVzdF9yZWJ1aWxkX2luc3RhbmNlX3dpdGhfc2NoZWR1bGVyX2dyb3VwX2ZhaWx1cmVcIiBBTkQgbWVzc2FnZTpcIkZBSUxFRFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI2NzgxMzA5MDYwfQ==

[Matt Riedemann (mriedem)]

Note that the rebuild_instance test has _set_vm_state_and_notify mocked, but we see that when it fails, something else is calling _set_vm_state_and_notify from build_instances:

http://logs.openstack.org/30/164330/2/gate/gate-nova-python27/19869e2/console.html#_2015-03-19_15_30_04_669

So there must be another test doing the same mock (or not doing it right) and that's screwing this up, which is probably this test:

test_build_instances_scheduler_group_failure

That does the same mocking.

[Hans Lindgren (hanlind)]

Sound similar to something I tried to fix some time ago. See https://review.openstack.org/122726 which describes a possible cause.

[Matt Riedemann (mriedem)]

Thanks Hans, I'm going to try kind of a hybrid of that.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/166389

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/166389
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=97d049b977d18ab277e7dfeae49062032a320ccd
Submitter: Jenkins
Branch: master

commit 97d049b977d18ab277e7dfeae49062032a320ccd
Author: Matt Riedemann <email address hidden>
Date: Fri Mar 20 14:41:11 2015 -0700

    mock out build_instances/rebuild_instance when not used

    Trying to close the gap on what could be causing issues
    with races in the conductor tests that hit build/rebuild
    instance.

    Change-Id: If36ffc21192493c940df9f60beef9476fbd31f84
    Partial-Bug: #1434161

[Matt Riedemann (mriedem)]

We haven't seen this in a long time so marking it as fixed (since Kilo).