Description of problem:
It has been observed that it’s not possible to have more than one simultaneous overcloud update. The issue is not Heat but Mistral-related.
###########
2019-11-05 16:45:36Z [AllNodesDeploySteps]: UPDATE_COMPLETE state changed
2019-11-05 16:45:38Z [edge0-compute]: UPDATE_COMPLETE Stack UPDATE completed successfully
Stack edge0-compute/d03b5b8d-e26c-472e-9c57-dd4bd8b348eb UPDATE_COMPLETE
Deploying overcloud configuration
Enabling ssh admin (tripleo-admin) for hosts:
192.168.111.213
Using ssh user heat-admin for initial connection.
Using ssh key at /home/stack/.ssh/id_rsa for initial connection.
Inserting TripleO short term key for 192.168.111.213
Warning: Permanently added '192.168.111.213' (ECDSA) to the list of known hosts.
Starting ssh admin enablement workflow
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - COMPLETE.
Removing TripleO short term key from 192.168.111.213
Warning: Permanently added '192.168.111.213' (ECDSA) to the list of known hosts.
Removing short term keys locally
Enabling ssh admin - COMPLETE.
Waiting for messages on queue 'tripleo' with no timeout.
Deployment already in progress with execution 535264f4-d735-4491-bc4f-37356b249622
Overcloud configuration failed.
###########
This issue poses a serious limitation in the usefulness of a single director managing multiple stacks. There have already been customers asking for about 30/40 Edge Locations managed by a single Control Plane (hence a single director). In such a context concurrent life-cycle operation could potentially happen.
Version-Release number of selected component (if applicable):
OSP15
How reproducible:
Deploy central zone, and then two edge zones in parallel.
Steps to Reproduce:
1.
2.
3.
Actual results:
Mistral fails to run config-download
Expected results:
Mistral runs two config-download in parallel
Alternative approach:
To solve this problem, the deployment workflow looks differently.
- Mistral is executed to only create, and potentially update, the plan and stack. Mistral, in turn, calls Heat. Once it has ended the workflow is completed.
- Then Ansible - Config-Download - is directly executed without Mistral interactions.
Fix proposed to branch: master
Review: https:/