test_heatclient failures due to possible config race

Bug #1273126 reported by Steven Hardy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Steven Hardy

Bug Description

As noticed by stevebaker, we seem to have some spurious failures happening in test_heatclient:

http://logs.openstack.org/38/69238/1/check/gate-heat-python27/9899837/testr_results.html.gz

The exact cause is unclear, but it would appear it could be a race when tests are run in parallel due to the cfg.CONF.set_override method used to override the config defaults (due to the globale cfg.CONF in oslo.config)

It should be possible to work around the issue by mocking the calls to cfg.CONF in heat_keystoneclient.py, instead of using set_override.

Steven Hardy (shardy)
Changed in heat:
assignee: nobody → Steven Hardy (shardy)
milestone: none → icehouse-3
status: New → In Progress
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/69339

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/69339
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9544b8008f716d720299adcb1a94837ae093d801
Submitter: Jenkins
Branch: master

commit 9544b8008f716d720299adcb1a94837ae093d801
Author: Steven Hardy <email address hidden>
Date: Mon Jan 27 12:11:44 2014 +0000

    stub cfg.CONF in heat_keystoneclient

    It seems that sometimes tests can fail when reading the config
    and the suspected cause is a race caused by running the tests in
    parallel when using the global set_override on oslo.config.

    Stubbing all calls to the config should solve this issue (although
    it is a bit cumbersome in the test..)

    Change-Id: Ib1ee97f116544e069123253002d4a408503ac82e
    Closes-Bug: #1273126

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
Steven Hardy (shardy) wrote :

So as discussed here with Clint, the fix merged here turned out to be the wrong one, tl;dr we didn't need the additional mocking, so I'm going to partially revert the patch and add the missing option group import:

https://review.openstack.org/#/c/65576/5/heat/tests/test_engine_service.py

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/73977

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master)

Reviewed: https://review.openstack.org/73977
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=4f21429e589cdd79bbc0491ccdc53afb62fb7bc2
Submitter: Jenkins
Branch: master

commit 4f21429e589cdd79bbc0491ccdc53afb62fb7bc2
Author: Steven Hardy <email address hidden>
Date: Thu Feb 13 17:03:08 2014 +0000

    Revert config mocking to cfg.CONF.set_override

    After moving to a different mocking strategy, it was pointed out that
    the issue was actually a missing config group import, so partially
    revert the change made under 9544b80 to simplify the test and add
    the missing config group import.

    Related-Bug: #1273126
    Change-Id: I3a8b65ab16845be060d6df4d94ae08f2d7d8255a

Thierry Carrez (ttx)
Changed in heat:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.