convergence functional tests sometimes timeout

Bug #1486281 reported by Angus Salkeld
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Rakesh H S

Bug Description

Different tests seem to randomly fail with timeouts.

An example (I have seen others, just can't find them right now):

heat_integrationtests.functional.test_create_update.UpdateStackTest.test_stack_update_rollback
 - http://logs.openstack.org/71/213971/1/check/gate-heat-dsvm-functional-convg-mysql/220af93/console.html#_2015-08-18_03_03_29_427

Angus Salkeld (asalkeld)
tags: added: functional-tests
tags: added: gate-failure
removed: functional-tests
Revision history for this message
Angus Salkeld (asalkeld) wrote :
Changed in heat:
status: New → Triaged
importance: Undecided → High
Angus Salkeld (asalkeld)
tags: added: convergence-bugs
Anant Patil (ananta)
Changed in heat:
assignee: nobody → Anant Patil (ananta)
Changed in heat:
assignee: Anant Patil (ananta) → Rakesh H S (rh-s)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/219699
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=139014f094d35610816718ba4024fce8cc5a6ef1
Submitter: Jenkins
Branch: master

commit 139014f094d35610816718ba4024fce8cc5a6ef1
Author: Rakesh H S <email address hidden>
Date: Wed Sep 2 19:13:03 2015 +0530

    Convergence: Resolve attributes during rollback

    Fixes failing convergence scenario tests
    - Added condition to resolve attributes during Rollback also.
    - When new template is assigned, reset dependencies and resources.

    Change-Id: I84762de87032168f97cdd19533649851073b4e81
    Partial-Bug: 1486281
    Co-Authored-By: Anant Patil <email address hidden>
    Co-Authored-By: Ishant Tyagi <email address hidden>

Revision history for this message
Rakesh H S (rh-s) wrote :

After investigation found out that timeout issues occur in following scenario:

Assume a CREATE/UPDATE of stack is in progress.
Now if a resource is stuck in [CREATE|UPDATE]_IN_PROGRESS status, and another update/delete is triggered, this new traversal will wait for the IN_PROGRESS resource to fail/complete, which will lead to timeout.

Seems like we need to bring back thread group manager, to stop the job, at least while delete. Why do we have to wait for the previous resource traversal to complete in case of delete.

tags: added: liberty-rc-potential
Changed in heat:
milestone: none → mitaka-1
tags: added: liberty-backport-potential
removed: liberty-rc-potential
Changed in heat:
milestone: mitaka-1 → mitaka-2
Revision history for this message
Rabi Mishra (rabi) wrote :

Not sure if it's the same issue or not. However, I see the following in the logs.

1. Stack DELETE completes successfully.

http://logs.openstack.org/28/256328/3/check/gate-heat-dsvm-functional-convg-mysql/32330af/logs/screen-h-eng.txt.gz#_2015-12-16_06_17_25_888

2. We get an error in the logs "The Stack (UpdateStackTest-1494054015) could not be found".

http://logs.openstack.org/28/256328/3/check/gate-heat-dsvm-functional-convg-mysql/32330af/logs/screen-h-eng.txt.gz#_2015-12-16_06_17_29_830

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/261929

Revision history for this message
Rakesh H S (rh-s) wrote :

This bug is fixed with multiple commits, we no longer see timeout issues in gate.

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
Sergey Kraynev (skraynev) wrote :

Change Fix Committed to Fix Released. Now the target/final status of bug should be Fix Released.

Changed in heat:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.