update_restrict intermittent failure when convergence is enabled

[Rabi Mishra]

It seems update_restrict functional tests failing with convergence gate job at times with the following error.

2016-03-08 06:00:35.400 | 2016-03-08 06:00:35.383 | ~~~~~~~~~~~~~~~~~~~
2016-03-08 06:00:35.402 | 2016-03-08 06:00:35.386 | Traceback (most recent call last):
2016-03-08 06:00:35.404 | 2016-03-08 06:00:35.388 | File "/opt/stack/new/heat/heat_integrationtests/functional/test_update_restricted.py", line 77, in test_update
2016-03-08 06:00:35.429 | 2016-03-08 06:00:35.389 | expected_status='UPDATE_COMPLETE')
2016-03-08 06:00:35.429 | 2016-03-08 06:00:35.391 | File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 400, in update_stack
2016-03-08 06:00:35.430 | 2016-03-08 06:00:35.395 | self._wait_for_stack_status(**kwargs)
2016-03-08 06:00:35.430 | 2016-03-08 06:00:35.397 | File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 338, in _wait_for_stack_status
2016-03-08 06:00:35.430 | 2016-03-08 06:00:35.399 | raise exceptions.TimeoutException(message)
2016-03-08 06:00:35.430 | 2016-03-08 06:00:35.401 | heat_integrationtests.common.exceptions.TimeoutException: Request timed out
2016-03-08 06:00:35.430 | 2016-03-08 06:00:35.403 | Details: Stack UpdateRestrictedStackTest-318585966/5f390d65-f3fe-475c-be35-6a577aad4d91 failed to reach UPDATE_COMPLETE status within the required time (1200 s).

I could see the following in the log.

f525dbdd1fd64a279717e1e78d2ec824 NOTIFY exchange 'heat' topic 'notifications.error' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:438
2016-03-08 05:40:29.600 16283 INFO heat.engine.stack [req-039f714a-71e3-4aad-87f5-e44b22feda68 demo demo - default default] Stack UPDATE FAILED (UpdateRestrictedStackTest-318585966): Resource CREATE failed: ResourceActionRestricted: resources.bar: update is restricted for resource.

This message is pretty confusing. It seems the stack is CREATED along with the resource and goes to CREATE_COMPLETE[1] and then when updating it with a changed env(that only restricts update), something is going wrong and it's trying to CREATE the resource and the error message is confusing.

[1] http://logs.openstack.org/45/280045/8/gate/gate-heat-dsvm-functional-convg-mysql-lbaasv1/0e5b06b/logs/screen-h-eng.txt.gz#_2016-03-08_05_40_29_600

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/329433
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=c85cae58bf2acfd32b6b745d7e218ba8fe05e604
Submitter: Jenkins
Branch: master

commit c85cae58bf2acfd32b6b745d7e218ba8fe05e604
Author: Zane Bitter <email address hidden>
Date: Tue Jun 14 15:09:19 2016 +0200

    Fix test_update_restricted timeouts due to timestamp resolution

    Updating a stack where the only resource needs replacement but is
    restricted from being replaced is *very* fast. Since the test framework
    relies on seeing a change in the updated_time timestamp, which has only 1s
    resolution, it can easily be missed causing the test to timeout. This adds
    a 1s sleep between updates that have the same expected status to ensure
    that the timestamp changes.

    Change-Id: Ief845a21108e925325b1b3e8f090038ccdf73b8e
    Closes-Bug: #1554375
    Related-Bug: #1583630

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/335970

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/335970
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=50e638b328c0c5ac3a4b809c3359db13e7918548
Submitter: Jenkins
Branch: master

commit 50e638b328c0c5ac3a4b809c3359db13e7918548
Author: Thomas Herve <email address hidden>
Date: Thu Jun 30 13:51:01 2016 +0200

    Fix another test race in test_update_restricted

    We fixed 2 tests that were intermittently failing because of timestamp
    resolution, but it can happen on the 3rd one as well.

    Change-Id: Ic916b79a81000684b04c585eef1227255a949172
    Related-Bug: #1554375

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/heat 7.0.0.0b2 development milestone.