IronicDriverConsoleTestCase.test__get_node_console_with_reset_wait_timeout intermittently fails since 4/18

[Matt Riedemann]

Since this change adjusted the timeout value in the test from .1 to 1 second:

https://review.openstack.org/#/c/457188/

This test is intermittently failing:

http://logs.openstack.org/38/457338/2/check/gate-nova-python35/21ee8c6/console.html#_2017-04-18_19_18_28_126076

Because it's not timing out as expected, which actually makes the test itself timeout after 160 seconds:

nova.tests.unit.virt.ironic.test_driver.IronicDriverConsoleTestCase.test__get_node_console_with_reset_wait_timeout 160.006

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Dynamic%20backoff%20interval%20looping%20call%20'nova.virt.ironic.driver.IronicDriver._get_node_console_with_reset%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

[Markus Zoeller (markus_z) (mzoeller)]

Can't we just raise the "loopingcall.LoopingCallTimeOut" at https://github.com/openstack/nova/blob/fe8415060ca452990d7019a03eaaa4b92aadfe8b/nova/virt/ironic/driver.py#L1473-L1477 with a mocked "timer" object?

[Markus Zoeller (markus_z) (mzoeller)]

Nevermind, I don't get that test.

[Jay Pipes (jaypipes)]

Yeah, I noticed this as well working on some patches yesterday. :(

[Matt Riedemann (mriedem)]

Markus, I thought about just removing all of the stub methods that get called by the looping call code, and just mock the looping call to raise the timeout exception to see that it's handled, since otherwise the test is very tightly coupled to how the loopcall code works, but that might defeat the purpose of the test.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/458161

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/458161
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5f2b187ae449fe03a5eac3e3493a6b267d1ed95a
Submitter: Jenkins
Branch: master

commit 5f2b187ae449fe03a5eac3e3493a6b267d1ed95a
Author: melanie witt <email address hidden>
Date: Wed Apr 19 16:29:05 2017 +0000

    Mock timeout in test__get_node_console_with_reset_wait_timeout

    This test is failing intermittently because it's waiting for a real
    timeout from BackOffLoopingCall which does an exponential backoff
    each interval. Sometimes the test times out entirely and fails.

    This changes the test to mock BackOffLoopingCall and raise the timeout
    exception to simulate a timeout. The only thing not covered by this is
    calling of the _wait_state() function, but that is covered by other
    unit tests in the file.

    Closes-Bug: #1683953

    Change-Id: Ie3ff6b0020dca8a5f5292b327aa2491920e338ef

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0b2 development milestone.