[RDO17.1] Overcloud Deployment Is Getting Failed with error "Message queue for ephemeral heat not created in time"

[James Bagwell]

Description
===========
Description of problem:
- CU is using RHEL 9.4 with openstack 17.1.2 Wallaby
- CU faced failure in Overcloud Deployment with error "Message queue for ephemeral heat not created in time" [1]
- Cu edited the heat_launcher.py and increased retry time to ephemeral Heat message queue with a higher value; 360 and 3. but the issue still exists and currently CU restored the default code after the test.
- CU also try running several times "openstack tripleo launch heat" before running the deployment but still issue remains
- CU has enough resources on director node as below:

undercloud) [stack@director ~]$ free -m
total used free shared buff/cache available
Mem: 42127 5224 34985 42 2522 36903
Swap: 0 0 0

Steps to reproduce
==================
Perform overcloud deployment.
Expected result
===============
Deployment would pass.

Actual result
=============
2024-05-27 15:42:50.638 87661 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden
2024-05-27 15:42:50.638 87661 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Using ephemeral heat for stack operation
2024-05-27 15:42:52.058 87661 INFO tripleoclient.heat_launcher [-] Skipping container image pull.
2024-05-27 15:42:52.075 87661 INFO tripleoclient.heat_launcher [-] Checking that database is up
2024-05-27 15:42:52.631 87661 INFO tripleoclient.heat_launcher [-] Checking that message bus (rabbitmq) is up
2024-05-27 10:12:57.433 2 INFO migrate.versioning.api [-] 72 -> 73...
2024-05-27 10:12:57.882 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.883 2 INFO migrate.versioning.api [-] 73 -> 74...
2024-05-27 10:12:57.893 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.894 2 INFO migrate.versioning.api [-] 74 -> 75...
2024-05-27 10:12:57.905 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.905 2 INFO migrate.versioning.api [-] 75 -> 76...
2024-05-27 10:12:57.916 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.916 2 INFO migrate.versioning.api [-] 76 -> 77...
2024-05-27 10:12:57.927 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.928 2 INFO migrate.versioning.api [-] 77 -> 78...
2024-05-27 10:12:57.938 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:57.938 2 INFO migrate.versioning.api [-] 78 -> 79...
2024-05-27 10:12:58.115 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.115 2 INFO migrate.versioning.api [-] 79 -> 80...
2024-05-27 10:12:58.226 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.226 2 INFO migrate.versioning.api [-] 80 -> 81...
2024-05-27 10:12:58.236 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.237 2 INFO migrate.versioning.api [-] 81 -> 82...
2024-05-27 10:12:58.247 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.247 2 INFO migrate.versioning.api [-] 82 -> 83...
2024-05-27 10:12:58.256 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.257 2 INFO migrate.versioning.api [-] 83 -> 84...
2024-05-27 10:12:58.267 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.267 2 INFO migrate.versioning.api [-] 84 -> 85...
2024-05-27 10:12:58.277 2 INFO migrate.versioning.api [-] done
2024-05-27 10:12:58.278 2 INFO migrate.versioning.api [-] 85 -> 86...
2024-05-27 10:12:58.386 2 INFO migrate.versioning.api [-] done
Pod:
c85d07f67416c5ef8bc59eb65873a4424e3c32643904850becdf1e233b1f6509
Containers:
4497376ae129b80661e03ea0825a5c284a022137cf80c7f49f071fb3d34230a0
6e3a7646b84fa2c50b84be318167894f9050c1cecbf823a14e98e6ddbf5140ab

2024-05-27 15:48:01.827 87661 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: tripleoclient.exceptions.HeatPodMessageQueueException: Message queue for ephemeral heat not created in time.
2024-05-27 15:48:01.830 87661 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat.
2024-05-27 15:48:02.053 87661 INFO tripleoclient.heat_launcher [-] Killing pod: ephemeral-heat
c85d07f67416c5ef8bc59eb65873a4424e3c32643904850becdf1e233b1f6509
2024-05-27 15:48:02.547 87661 INFO tripleoclient.heat_launcher [-] Killed pod: ephemeral-heat
2024-05-27 15:48:03.165 87661 INFO tripleoclient.heat_launcher [-] Starting back up of heat db
2024-05-27 15:48:03.839 87661 INFO tripleoclient.heat_launcher [-] Created tarfile /home/stack/overcloud-deploy/overcloud/heat-launcher/heat-db.sql-1716804770.6390433.tar.bzip2
2024-05-27 15:48:03.840 87661 INFO tripleoclient.heat_launcher [-] Deleting /home/stack/overcloud-deploy/overcloud/heat-launcher/heat-db.sql
2024-05-27 15:48:04.783 87661 INFO tripleoclient.heat_launcher [-] Removing pod: ephemeral-heat
c85d07f67416c5ef8bc59eb65873a4424e3c32643904850becdf1e233b1f6509
2024-05-27 15:48:05.144 87661 INFO tripleoclient.heat_launcher [-] Created tarfile /home/stack/overcloud-deploy/overcloud/heat-launcher/log/heat-1716804770.6390433.log-1716804770.6390433.tar.bzip2
2024-05-27 15:48:05.145 87661 INFO tripleoclient.heat_launcher [-] Deleting /home/stack/overcloud-deploy/overcloud/heat-launcher/log/heat-1716804770.6390433.log
2024-05-27 15:48:05.146 87661 ERROR openstack [-] Message queue for ephemeral heat not created in time.: tripleoclient.exceptions.HeatPodMessageQueueException: Message queue for ephemeral heat not created in time.
2024-05-27 15:48:05.147 87661 INFO osc_lib.shell [-] END return value: 1

Environment
===========
- Openstack Wallaby 17.1.2
- RHEL 9.4

2. Which storage type did you use?
Ceph
3. Which networking type did you use?
OVS

[James Bagwell (jimbagwell)]

Hi - can someone from the community project please have a look on this? A month has passed and no activity