deploying a stack fails after running a major upgrade on a previous stack

[Damien Ciabrini]

I'm trying o exercise the major upgrade workflow on a minimal one-node HA, containerized Queens overcloud.
(I only enabled a few services on that overcloud: docker, pacemaker, galera, tripleo-packages).

./overcloud-deploy.sh --no-cleanup -e ~/tripleo-heat-templates/environments/config-debug.yaml -e ~/tripleo-heat-templates/environments/disable-telemetry.yaml -e ~/tripleo-heat-templates/environments/docker.yaml -e ~/tripleo-heat-templates/environments/docker-ha.yaml -e /home/stack/containers-default-parameters.yaml -e ~/custom_stuff.yaml -r ~/custom_roles.yaml

openstack overcloud upgrade --init-update --templates tripleo-heat-templates -r ~/custom_roles.yaml --container-registry-file containers-default-parameters.yaml

openstack overcloud upgrade --nodes Controller

The ansible upgrade tasks run just fine on my controller nodes, and the result is as expected.

I then delete my stack and try to redeploy another stack with the same properties.
When the new deploy is finished, it seems I have mo services at all in controller nodes.

Looking at the deploy logs, it looks like all the ControllerDeployment_Step{xxx} all take less than a second, which whould indicate that it never ever tried to deploy any service?

2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step1]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step1]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step1]: CREATE_COMPLETE state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step1]: CREATE_COMPLETE state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step2]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step2]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step2]: CREATE_COMPLETE state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step2]: CREATE_COMPLETE state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step3]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step3]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step3]: CREATE_COMPLETE state changed
2018-02-22 09:58:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step3]: CREATE_COMPLETE state changed
2018-02-22 09:58:35Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step4]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:35Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step4]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:35Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step4]: CREATE_COMPLETE state changed
2018-02-22 09:58:35Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step4]: CREATE_COMPLETE state changed
2018-02-22 09:58:35Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:35Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:35Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_COMPLETE state changed
2018-02-22 09:58:35Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5]: CREATE_COMPLETE state changed
2018-02-22 09:58:35Z [overcloud.AllNodesDeploySteps.ComputeExtraConfigPost]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:35Z [overcloud.AllNodesDeploySteps.ControllerExtraConfigPost]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:36Z [overcloud.AllNodesDeploySteps.ComputeExtraConfigPost]: CREATE_COMPLETE state changed
2018-02-22 09:58:38Z [overcloud.AllNodesDeploySteps.ControllerExtraConfigPost]: CREATE_COMPLETE state changed
2018-02-22 09:58:38Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:38Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_IN_PROGRESS state changed
2018-02-22 09:58:38Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_COMPLETE state changed
2018-02-22 09:58:38Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_COMPLETE state changed

When I log in to the deployed controller nodes, docker is not started, neither is pacemaker, no docker-puppet tasks have been run.

[Marios Andreou (marios-b)]

13:37 < marios> dciabrin: hmm sounds like config is still 'no-oped' somehow? (perhaps it has persisted the
                https://github.com/openstack/tripleo-heat-templates/blob/master/environments/noop-deploy-steps.yaml which is currently set like
                https://github.com/openstack/tripleo-common/blob/9f40ac8745db4c8589cd73556fc34db27f09f299/tripleo_common/actions/package_update.py#L7213:38 < marios> dciabrin: we are currently reworking a lot of the cli there, fwiw (though i haven't tried or hit that particular issue personally)
13:40 < marios> dciabrin: see jistr patches at https://review.openstack.org/#/c/546242 (there are 6) plus https://review.openstack.org/#/c/545026/ and
                https://review.openstack.org/#/c/545365/ ontop (I will update the https://review.openstack.org/535859 with the application of those for copy/paste later...
                but i'm updating those top 2 patches at the moment
13:40 < marios> dciabrin: but the cli will be more like https://review.openstack.org/#/c/545026/6//COMMIT_MSG

[Damien Ciabrini (dciabrin)]

Apparently when applying https://review.openstack.org/#/c/546168/2, my problem goes away.
Thanks Marios and Jiri!