duplicate horizon, keystone containers getting launced during overcloud deploy

[Shyam]

During overcloud deploy command execution, duplicate containers getting launched for horizon and keystone.

Steps to reproduce:

1. Run "openstack overcloud deploy..." command, overcloud stack create completes but deployment fails at ansible step4
2. Run "openstack overcloud deploy --config-download-only..." command to just apply ansible configuration
3. Above command creates duplicate horizon and keystone containers with different name trying to listen on same port. So one horizon container keeps restarting with error "address already in use"

[root@overcloud-controller-0 ~]# docker ps -a | grep horizon
5d89f5fd3faf 172.172.3.151:8787/rhosp14/openstack-horizon:14.0-124 "kolla_start" About an hour ago Exited (1) 18 minutes ago horizon-eilkqsif
f07170d17117 172.172.3.151:8787/rhosp14/openstack-horizon:14.0-124 "kolla_start" 3 hours ago Exited (1) 2 hours ago horizon-slbfkwmf
aeeea02c6125 172.172.3.151:8787/rhosp14/openstack-horizon:14.0-124 "kolla_start" 3 hours ago Up 3 hours horizon
c9b3020ee979 172.172.3.151:8787/rhosp14/openstack-horizon:14.0-124 "/bin/bash -c 'tou..." 16 hours ago Exited (0) 16 hours ago horizon_fix_perms

[root@overcloud-controller-0 heat-admin]# docker ps -a | grep keystone
98be6a894c3e 172.172.3.151:8787/rhosp14/openstack-keystone:14.0-119 "/bin/bash -c '/us..." About an hour ago Up About an hour keystone_cron-u3l5opvw
277bc5a9c47b 172.172.3.151:8787/rhosp14/openstack-keystone:14.0-119 "/bin/bash -c '/us..." About an hour ago Up About an hour keystone_cron
0bb4a84baac9 172.172.3.151:8787/rhosp14/openstack-keystone:14.0-119 "kolla_start" 3 hours ago Exited (1) 2 hours ago keystone-8qsm5jpl
1852c3e4d116 172.172.3.151:8787/rhosp14/openstack-keystone:14.0-119 "kolla_start" 3 hours ago Up 3 hours (healthy) keystone
1e61d427c852 172.172.3.151:8787/rhosp14/openstack-keystone:14.0-119 "/usr/bin/bootstra..." 3 hours ago Exited (1) 3 hours ago keystone_db_sync-lzkfykbg
287314b2f2fa 172.172.3.151:8787/rhosp14/openstack-keystone:14.0-119 "/usr/bin/bootstra..." 3 hours ago Exited (0) 3 hours ago keystone_db_sync
e4242f3af966 172.172.3.151:8787/rhosp14/openstack-keystone:14.0-119 "/bin/bash -c 'cho..." 15 hours ago Exited (0) 15 hours ago keystone_init_log

Expected Results:
Openstack overcloud deploy command should not create duplicate containers of the same service listing on same port.

Actual Result:
openstack overcloud deploy command creating duplicate containers for same service listening on the same port

Environment:
RHOSP14

Run command:

openstack overcloud deploy --templates \
--libvirt-type qemu \
--ntp-server 172.172.3.150 \
-e /home/stack/templates/node-info.yaml \
-e /home/stack/containers-prepare-parameter.yaml \
--log-file overcloud_deployment.log

openstack overcloud deploy --config-download-only --templates \
--libvirt-type qemu \
--ntp-server 172.172.3.150 \
-e /home/stack/templates/node-info.yaml \
-e /home/stack/containers-prepare-parameter.yaml \
--log-file overcloud_deployment.log

(undercloud) [stack@undercloud14 ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID | Stack Name | Project | Stack Status | Creation Time | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| f5f82cdf-2a78-47c7-9551-88b5660d84e2 | overcloud | 6a5b1ea180f945ef945288a2fea4c307 | CREATE_COMPLETE | 2019-12-16T11:32:31Z | None |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+

(undercloud) [stack@undercloud14 ~]$ openstack overcloud status --plan overcloud
+-----------+---------------------+---------------------+-------------------+
| Plan Name | Created | Updated | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2019-12-17 08:22:03 | 2019-12-17 08:22:03 | DEPLOY_FAILED |
+-----------+---------------------+---------------------+-------------------+

[wes hayutin (weshayutin)]

@Luke please have a look

[Luke Short (ekultails)]

RHOSP 14 went end-of-life at the beginning of this year so no more updates will be rolled into that release. Are you preparing for an upgrade to RHOSP 15?

You can manually apply this patch on your nodes to see if this helps: https://review.opendev.org/#/c/704655/

I will do further testing in my own lab as well.