Conductor HA: fail on heat stack create loop

Bug #1198990 reported by Timur Nurlygayanov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Murano
Fix Released
Critical
Unassigned

Bug Description

Moved to https://bugs.launchpad.net/murano/+bug/1198990

*Steps To Reproduce:*
1. Run a few instances of Murano Conductor and configure all instances for one rabbitMQ user and vhost
2. Start to deploy environment with a few services (AD with 5 instances and one IIS service)
3. Shutdown conductor service on node, which started to create heat stack (we can monitor this activity by conductor logs)

*Observed Result:*
Other nodes does not works and does not continue to deploy this environment. Open Stack Heat can create stack for a few seconds (10+ seconds or a few minutes) - we can easily lose a few servers in HA mode during this time period - and we should fix heat stack polling mechanism to avoid this situation.

*Note:*
Now conductor uses the following algorithm:
1. Create Heat template for environment
2. Start to deploy Heat template and check status of created stack (this step is black hole for HA)
3. Start to deploy workflow tasks
Need to improve step 2 to track Heat stack status by all conductor instances, we can do this if conductor will save information about stacks in rabbitMQ - and if one conductor instance will fail, other instances will take this task from rabbitMQ queue.

Tags: conductor
Changed in murano:
importance: Critical → Medium
Changed in murano:
importance: Medium → High
milestone: none → 0.3
Changed in murano:
milestone: 0.3 → none
milestone: none → 0.2.1
description: updated
Changed in murano:
importance: High → Critical
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.