openstack overcloud status shows DEPLOY_SUCCESS when the stack status is CREATE_FAILED

[James Slagle]

Description of problem:
Our deployment failed with error "503 Service Unavailable".
Apparently the heat service is down:
(undercloud) [stack@undercloud-0 ~]$ sudo docker ps |grep heat
2db1dff00df6 192.168.24.1:8787/rhosp14/openstack-heat-api-cfn:2018-10-08.4 "kolla_start" 5 hours ago Up 5 hours (unhealthy)
                    heat_api_cfn
35b191b639f2 192.168.24.1:8787/rhosp14/openstack-heat-api:2018-10-08.4 "kolla_start" 5 hours ago Up 5 hours
                    heat_api_cron
c804c8eb85aa 192.168.24.1:8787/rhosp14/openstack-heat-engine:2018-10-08.4 "kolla_start" 5 hours ago Up 5 hours (healthy)
                    heat_engine
33157896efc6 192.168.24.1:8787/rhosp14/openstack-heat-api:2018-10-08.4 "kolla_start" 5 hours ago Up 5 hours (unhealthy)
                    heat_api
(undercloud) [stack@undercloud-0 ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| ID | Stack Name | Project | Stack Status | Creation Time | Updated Time |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| f1517388-13c5-48c5-b2cc-5fc7a2468502 | overcloud | 57c16282a45444fe9c4fe49c8ae4206e | CREATE_FAILED | 2018-10-10T09:09:58Z | None |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
(undercloud) [stack@undercloud-0 ~]$ openstack overcloud status

+-----------+---------------------+---------------------+-------------------+
| Plan Name | Created | Updated | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-10-10 09:05:58 | 2018-10-10 09:05:58 | DEPLOY_SUCCESS |
+-----------+---------------------+---------------------+-------------------+

And this issue confuses the GUI. The GUI is showed: Deployment success.

Version-Release number of selected component (if applicable):
openstack-tripleo-common-container-base-9.3.1-0.20180923215327.d22cb3e.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Install OSP14 and deploy overcloud
2. Stop the heat service shortly after the deployment started
3.

Actual results:

Expected results:

Additional info:
SOSReport will be attached

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/611168

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/611168
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=de06f9373aa3f26739deaae7b57901679cf62373
Submitter: Zuul
Branch: master

commit de06f9373aa3f26739deaae7b57901679cf62373
Author: James Slagle <email address hidden>
Date: Tue Oct 16 16:28:15 2018 -0400

    Sync state if needed during retrieval

    Update the deployment status if needed when retrieving the status. We do
    this since tripleoclient does not yet use a single API for overcloud
    deployment. Since there is no long running process to make sure the
    status is updated, we instead update the status if needed when we get it
    with this action.

    The logic to update the status is detailed in a comment in the code.

    Also as part of this commit the status is kept as deploying in
    deploy_play when not triggering config_download, since the client will
    trigger config_download on it's own. It makes more sense to keep the
    status as deploying in that scenario.

    Change-Id: I6d329e974965edf28d6f5b12e6854319cfb683f4
    Closes-Bug: #1798193

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/617555

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/617555
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=79c687cd32f324f3ad207f83e39fa20752f4147f
Submitter: Zuul
Branch: stable/rocky

commit 79c687cd32f324f3ad207f83e39fa20752f4147f
Author: James Slagle <email address hidden>
Date: Tue Oct 16 16:28:15 2018 -0400

    Sync state if needed during retrieval

    Update the deployment status if needed when retrieving the status. We do
    this since tripleoclient does not yet use a single API for overcloud
    deployment. Since there is no long running process to make sure the
    status is updated, we instead update the status if needed when we get it
    with this action.

    The logic to update the status is detailed in a comment in the code.

    Also as part of this commit the status is kept as deploying in
    deploy_play when not triggering config_download, since the client will
    trigger config_download on it's own. It makes more sense to keep the
    status as deploying in that scenario.

    Change-Id: I6d329e974965edf28d6f5b12e6854319cfb683f4
    Closes-Bug: #1798193
    (cherry picked from commit de06f9373aa3f26739deaae7b57901679cf62373)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 9.5.0 release.