Ocata to Pike upgrade: cannot rerun major-upgrade-composable-steps-docker.yaml after the pacemaker resources have been deleted

[Marius Cornea]

Description of problem:
Ocata -> Pike containerized upgrade: cannot rerun major-upgrade-composable-steps-docker.yaml after the pacemaker resources have been deleted. The overcloud deploy command fails with:

2017-08-30 10:21:03Z [overcloud]: UPDATE_FAILED resources.AllNodesDeploySteps: resources.ControllerUpgrade_Step2: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2

 Stack overcloud UPDATE_FAILED

overcloud.AllNodesDeploySteps.ControllerUpgrade_Step2.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 5ab172b9-7096-43f0-8688-3820207267d3
  status: UPDATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
    TASK [set is_bootstrap_node fact] **********************************************
    ok: [localhost]

    TASK [Disable the haproxy cluster resource.] ***********************************
    fatal: [localhost]: FAILED! => {"changed": false, "error": "Error: resource/clone/master/group/bundle 'haproxy' does not exist\n", "failed": true, "msg": "Failed, to set the resource haproxy to the statedisable", "output": "", "rc": 1}
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/7c63dab0-fd92-4f7f-9097-5aff99a62126_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=7 changed=4 unreachable=0 failed=1

    (truncated, view all with --long)

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11
2. Run major-upgrade-composable-steps-docker.yaml step to upgrade to OSP12
3. Fail after the pacemaker resources have been deleted
4. Fix the failure cause
5. Rerun the major-upgrade-composable-steps-docker.yaml step

Actual results:
Upgrade fails with error": "Error: resource/clone/master/group/bundle 'haproxy' does not exist because the haproxy pacemaker resource has been deleted.

Expected results:
major-upgrade-composable-steps-docker.yaml should be idempotent. If the resource has already been stopped/deleted then it should proceed to the next step.

Additional info:

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/501734
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=64d7be1e3d4552e06cbc53f788572e530cc5c3bb
Submitter: Jenkins
Branch: master

commit 64d7be1e3d4552e06cbc53f788572e530cc5c3bb
Author: Marius Cornea <email address hidden>
Date: Thu Sep 7 15:38:54 2017 +0200

    One time delete pacemaker resources during upgrade to containers

    This change allows running the major upgrade composable docker
    steps multiple times by not trying to delete the pacemaker resources
    if they're not reported as started or in master state.

    Closes-bug: 1716031
    Depends-On: I8da03f5c4a6d442617b81be5793a9724cc8842bf
    Change-Id: Ifcf9de8c82550a90a9fb118052d43fdbcdc6ca7e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/504420

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/504420
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=dc9d9893717b819cf65893573199db8b473d284f
Submitter: Jenkins
Branch: stable/pike

commit dc9d9893717b819cf65893573199db8b473d284f
Author: Marius Cornea <email address hidden>
Date: Thu Sep 7 15:38:54 2017 +0200

    One time delete pacemaker resources during upgrade to containers

    This change allows running the major upgrade composable docker
    steps multiple times by not trying to delete the pacemaker resources
    if they're not reported as started or in master state.

    Closes-bug: 1716031
    Depends-On: I8da03f5c4a6d442617b81be5793a9724cc8842bf
    Change-Id: Ifcf9de8c82550a90a9fb118052d43fdbcdc6ca7e
    (cherry picked from commit 64d7be1e3d4552e06cbc53f788572e530cc5c3bb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b1 development milestone.