Conductor HA: fail on heat stack create loop
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Murano |
Fix Released
|
Critical
|
Unassigned |
Bug Description
Moved to https:/
*Steps To Reproduce:*
1. Run a few instances of Murano Conductor and configure all instances for one rabbitMQ user and vhost
2. Start to deploy environment with a few services (AD with 5 instances and one IIS service)
3. Shutdown conductor service on node, which started to create heat stack (we can monitor this activity by conductor logs)
*Observed Result:*
Other nodes does not works and does not continue to deploy this environment. Open Stack Heat can create stack for a few seconds (10+ seconds or a few minutes) - we can easily lose a few servers in HA mode during this time period - and we should fix heat stack polling mechanism to avoid this situation.
*Note:*
Now conductor uses the following algorithm:
1. Create Heat template for environment
2. Start to deploy Heat template and check status of created stack (this step is black hole for HA)
3. Start to deploy workflow tasks
Need to improve step 2 to track Heat stack status by all conductor instances, we can do this if conductor will save information about stacks in rabbitMQ - and if one conductor instance will fail, other instances will take this task from rabbitMQ queue.
Changed in murano: | |
importance: | Critical → Medium |
Changed in murano: | |
importance: | Medium → High |
milestone: | none → 0.3 |
Changed in murano: | |
milestone: | 0.3 → none |
milestone: | none → 0.2.1 |
description: | updated |
Changed in murano: | |
importance: | High → Critical |
status: | New → Fix Released |