Description
===========
Description of problem:
- CU is using RHEL 9.4 with OSP 17.1.2 Wallaby
- CU faced failure in Overcloud Deployment with error "Message queue for ephemeral heat not created in time" [1]
- Cu edited the heat_launcher.py and increased retry time to ephemeral Heat message queue with a higher value; 360 and 3. but the issue still exists and currently CU restored the default code after the test.
- CU also try running several times "openstack tripleo launch heat" before running the deployment but still issue remains
- CU has enough resources on director node as below:
undercloud) [stack@director ~]$ free -m
total used free shared buff/cache available
Mem: 42127 5224 34985 42 2522 36903
Swap: 0 0 0
Steps to reproduce
==================
Perform overcloud deployment.
Expected result
===============
Deployment would pass.
Actual result
=============
2024-05-27 15:42:50.638 87661 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden
2024-05-27 15:42:50.638 87661 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Using ephemeral heat for stack operation
2024-05-27 15:42:52.058 87661 INFO tripleoclient.heat_launcher [-] Skipping container image pull.
2024-05-27 15:42:52.075 87661 INFO tripleoclient.heat_launcher [-] Checking that database is up
2024-05-27 15:42:52.631 87661 INFO tripleoclient.heat_launcher [-] Checking that message bus (rabbitmq) is up
2024-05-27 10:12:57.433 2 INFO migrate.versioning.api [-] 72 -> 73...
2024-05-27 10:12:57.882 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.883 2 INFO migrate.versioning.api [-] 73 -> 74...
2024-05-27 10:12:57.893 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.894 2 INFO migrate.versioning.api [-] 74 -> 75...
2024-05-27 10:12:57.905 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.905 2 INFO migrate.versioning.api [-] 75 -> 76...
2024-05-27 10:12:57.916 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.916 2 INFO migrate.versioning.api [-] 76 -> 77...
2024-05-27 10:12:57.927 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.928 2 INFO migrate.versioning.api [-] 77 -> 78...
2024-05-27 10:12:57.938 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.938 2 INFO migrate.versioning.api [-] 78 -> 79...
2024-05-27 10:12:58.115 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.115 2 INFO migrate.versioning.api [-] 79 -> 80...
2024-05-27 10:12:58.226 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.226 2 INFO migrate.versioning.api [-] 80 -> 81...
2024-05-27 10:12:58.236 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.237 2 INFO migrate.versioning.api [-] 81 -> 82...
2024-05-27 10:12:58.247 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.247 2 INFO migrate.versioning.api [-] 82 -> 83...
2024-05-27 10:12:58.256 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.257 2 INFO migrate.versioning.api [-] 83 -> 84...
2024-05-27 10:12:58.267 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.267 2 INFO migrate.versioning.api [-] 84 -> 85...
2024-05-27 10:12:58.277 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.278 2 INFO migrate.versioning.api [-] 85 -> 86...
2024-05-27 10:12:58.386 2 INFO migrate.versioning.api [-] done
Pod:
c85d07f67416c5ef8bc59eb65873a4424e3c32643904850becdf1e233b1f6509
Containers:
4497376ae129b80661e03ea0825a5c284a022137cf80c7f49f071fb3d34230a0
6e3a7646b84fa2c50b84be318167894f9050c1cecbf823a14e98e6ddbf5140ab
2024-05-27 15:48:01.827 87661 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: tripleoclient.exceptions.HeatPodMessageQueueException: Message queue for ephemeral heat not created in time.
2024-05-27 15:48:01.830 87661 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat.
2024-05-27 15:48:02.053 87661 INFO tripleoclient.heat_launcher [-] Killing pod: ephemeral-heat
c85d07f67416c5ef8bc59eb65873a4424e3c32643904850becdf1e233b1f6509
2024-05-27 15:48:02.547 87661 INFO tripleoclient.heat_launcher [-] Killed pod: ephemeral-heat
2024-05-27 15:48:03.165 87661 INFO tripleoclient.heat_launcher [-] Starting back up of heat db
2024-05-27 15:48:03.839 87661 INFO tripleoclient.heat_launcher [-] Created tarfile /home/stack/overcloud-deploy/overcloud/heat-launcher/heat-db.sql-1716804770.6390433.tar.bzip2
2024-05-27 15:48:03.840 87661 INFO tripleoclient.heat_launcher [-] Deleting /home/stack/overcloud-deploy/overcloud/heat-launcher/heat-db.sql
2024-05-27 15:48:04.783 87661 INFO tripleoclient.heat_launcher [-] Removing pod: ephemeral-heat
c85d07f67416c5ef8bc59eb65873a4424e3c32643904850becdf1e233b1f6509
2024-05-27 15:48:05.144 87661 INFO tripleoclient.heat_launcher [-] Created tarfile /home/stack/overcloud-deploy/overcloud/heat-launcher/log/heat-1716804770.6390433.log-1716804770.6390433.tar.bzip2
2024-05-27 15:48:05.145 87661 INFO tripleoclient.heat_launcher [-] Deleting /home/stack/overcloud-deploy/overcloud/heat-launcher/log/heat-1716804770.6390433.log
2024-05-27 15:48:05.146 87661 ERROR openstack [-] Message queue for ephemeral heat not created in time.: tripleoclient.exceptions.HeatPodMessageQueueException: Message queue for ephemeral heat not created in time.
2024-05-27 15:48:05.147 87661 INFO osc_lib.shell [-] END return value: 1
Environment
===========
- RHOSP 17.1.2
- RHEL 9.4
2. Which storage type did you use?
Ceph
3. Which networking type did you use?
OVS
Description
===========
Description of problem:
- CU is using RHEL 9.4 with OSP 17.1.2 Wallaby
- CU faced failure in Overcloud Deployment with error "Message queue for ephemeral heat not created in time" [1]
- Cu edited the heat_launcher.py and increased retry time to ephemeral Heat message queue with a higher value; 360 and 3. but the issue still exists and currently CU restored the default code after the test.
- CU also try running several times "openstack tripleo launch heat" before running the deployment but still issue remains
- CU has enough resources on director node as below:
undercloud) [stack@director ~]$ free -m
total used free shared buff/cache available
Mem: 42127 5224 34985 42 2522 36903
Swap: 0 0 0
Steps to reproduce
==================
Perform overcloud deployment.
Expected result
===============
Deployment would pass.
Actual result utils.safe_ write [-] The output file /home/stack/ overcloud- deploy/ overcloud/ overcloud- deployment_ status. yaml will be overriden v1.overcloud_ deploy. DeployOvercloud [-] Using ephemeral heat for stack operation heat_launcher [-] Skipping container image pull. heat_launcher [-] Checking that database is up heat_launcher [-] Checking that message bus (rabbitmq) is up versioning. api [-] 72 -> 73... versioning. api [-] done versioning. api [-] 73 -> 74... versioning. api [-] done versioning. api [-] 74 -> 75... versioning. api [-] done versioning. api [-] 75 -> 76... versioning. api [-] done versioning. api [-] 76 -> 77... versioning. api [-] done versioning. api [-] 77 -> 78... versioning. api [-] done versioning. api [-] 78 -> 79... versioning. api [-] done versioning. api [-] 79 -> 80... versioning. api [-] done versioning. api [-] 80 -> 81... versioning. api [-] done versioning. api [-] 81 -> 82... versioning. api [-] done versioning. api [-] 82 -> 83... versioning. api [-] done versioning. api [-] 83 -> 84... versioning. api [-] done versioning. api [-] 84 -> 85... versioning. api [-] done versioning. api [-] 85 -> 86... versioning. api [-] done f8bc59eb65873a4 424e3c326439048 50becdf1e233b1f 6509 661e03ea0825a5c 284a022137cf80c 7f49f071fb3d342 30a0 50b84be31816789 4f9050c1cecbf82 3a14e98e6ddbf51 40ab
=============
2024-05-27 15:42:50.638 87661 WARNING tripleoclient.
2024-05-27 15:42:50.638 87661 INFO tripleoclient.
2024-05-27 15:42:52.058 87661 INFO tripleoclient.
2024-05-27 15:42:52.075 87661 INFO tripleoclient.
2024-05-27 15:42:52.631 87661 INFO tripleoclient.
2024-05-27 10:12:57.433 2 INFO migrate.
2024-05-27 10:12:57.882 2 INFO migrate.
2024-05-27 10:12:57.883 2 INFO migrate.
2024-05-27 10:12:57.893 2 INFO migrate.
2024-05-27 10:12:57.894 2 INFO migrate.
2024-05-27 10:12:57.905 2 INFO migrate.
2024-05-27 10:12:57.905 2 INFO migrate.
2024-05-27 10:12:57.916 2 INFO migrate.
2024-05-27 10:12:57.916 2 INFO migrate.
2024-05-27 10:12:57.927 2 INFO migrate.
2024-05-27 10:12:57.928 2 INFO migrate.
2024-05-27 10:12:57.938 2 INFO migrate.
2024-05-27 10:12:57.938 2 INFO migrate.
2024-05-27 10:12:58.115 2 INFO migrate.
2024-05-27 10:12:58.115 2 INFO migrate.
2024-05-27 10:12:58.226 2 INFO migrate.
2024-05-27 10:12:58.226 2 INFO migrate.
2024-05-27 10:12:58.236 2 INFO migrate.
2024-05-27 10:12:58.237 2 INFO migrate.
2024-05-27 10:12:58.247 2 INFO migrate.
2024-05-27 10:12:58.247 2 INFO migrate.
2024-05-27 10:12:58.256 2 INFO migrate.
2024-05-27 10:12:58.257 2 INFO migrate.
2024-05-27 10:12:58.267 2 INFO migrate.
2024-05-27 10:12:58.267 2 INFO migrate.
2024-05-27 10:12:58.277 2 INFO migrate.
2024-05-27 10:12:58.278 2 INFO migrate.
2024-05-27 10:12:58.386 2 INFO migrate.
Pod:
c85d07f67416c5e
Containers:
4497376ae129b80
6e3a7646b84fa2c
2024-05-27 15:48:01.827 87661 WARNING tripleoclient. utils.safe_ write [-] The output file /home/stack/ overcloud- deploy/ overcloud/ overcloud- deployment_ status. yaml will be overriden: tripleoclient. exceptions. HeatPodMessageQ ueueException: Message queue for ephemeral heat not created in time. v1.overcloud_ deploy. DeployOvercloud [-] Stopping ephemeral heat. heat_launcher [-] Killing pod: ephemeral-heat f8bc59eb65873a4 424e3c326439048 50becdf1e233b1f 6509 heat_launcher [-] Killed pod: ephemeral-heat heat_launcher [-] Starting back up of heat db heat_launcher [-] Created tarfile /home/stack/ overcloud- deploy/ overcloud/ heat-launcher/ heat-db. sql-1716804770. 6390433. tar.bzip2 heat_launcher [-] Deleting /home/stack/ overcloud- deploy/ overcloud/ heat-launcher/ heat-db. sql heat_launcher [-] Removing pod: ephemeral-heat f8bc59eb65873a4 424e3c326439048 50becdf1e233b1f 6509 heat_launcher [-] Created tarfile /home/stack/ overcloud- deploy/ overcloud/ heat-launcher/ log/heat- 1716804770. 6390433. log-1716804770. 6390433. tar.bzip2 heat_launcher [-] Deleting /home/stack/ overcloud- deploy/ overcloud/ heat-launcher/ log/heat- 1716804770. 6390433. log exceptions. HeatPodMessageQ ueueException: Message queue for ephemeral heat not created in time.
2024-05-27 15:48:01.830 87661 INFO tripleoclient.
2024-05-27 15:48:02.053 87661 INFO tripleoclient.
c85d07f67416c5e
2024-05-27 15:48:02.547 87661 INFO tripleoclient.
2024-05-27 15:48:03.165 87661 INFO tripleoclient.
2024-05-27 15:48:03.839 87661 INFO tripleoclient.
2024-05-27 15:48:03.840 87661 INFO tripleoclient.
2024-05-27 15:48:04.783 87661 INFO tripleoclient.
c85d07f67416c5e
2024-05-27 15:48:05.144 87661 INFO tripleoclient.
2024-05-27 15:48:05.145 87661 INFO tripleoclient.
2024-05-27 15:48:05.146 87661 ERROR openstack [-] Message queue for ephemeral heat not created in time.: tripleoclient.
2024-05-27 15:48:05.147 87661 INFO osc_lib.shell [-] END return value: 1
Environment
===========
- RHOSP 17.1.2
- RHEL 9.4
2. Which storage type did you use?
Ceph
3. Which networking type did you use?
OVS