FFU: openstack overcloud ffwd-upgrade prepare fails with: u'message': u'Init Minor update failed'

[Carlos Camacho]

Description of problem:

FFU: openstack overcloud ffwd-upgrade prepare fails with: u'message': u'Init Minor update failed':

2018-06-01 02:47:45Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.ControllerPrePuppet]: DELETE_IN_PROGRESS state changed
2018-06-01 02:47:46Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya-ControllerPrePuppet-6chp5nf3gnro]: DELETE_IN_PROGRESS Stack DELETE started
2018-06-01 02:47:46Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya-ControllerPrePuppet-6chp5nf3gnro.ControllerPrePuppetMaintenanceModeDeployment]: DELETE_IN_PROGRESS state changed
2018-06-01 02:47:46Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.CephStorageConfig]: DELETE_IN_PROGRESS state changed
2018-06-01 02:47:46Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya-CephStorageConfig-6anashbrrrau]: DELETE_IN_PROGRESS Stack DELETE started
2018-06-01 02:47:46Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya-CephStorageConfig-6anashbrrrau.CephStoragePuppetConfigImpl]: DELETE_IN_PROGRESS state changed
2018-06-01 02:47:47Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.ControllerConfig]: DELETE_COMPLETE state changed
2018-06-01 02:47:47Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.ComputeConfig]: DELETE_COMPLETE state changed
2018-06-01 02:47:47Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.CephStorageConfig]: DELETE_COMPLETE state changed
2018-06-01 02:47:51Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.ControllerPrePuppet]: DELETE_COMPLETE state changed
2018-06-01 02:47:51Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya]: UPDATE_COMPLETE Stack UPDATE completed successfully
2018-06-01 02:47:52Z [AllNodesDeploySteps]: UPDATE_COMPLETE state changed
2018-06-01 02:47:55Z [overcloud]: UPDATE_COMPLETE Stack UPDATE completed successfully

 Stack overcloud UPDATE_COMPLETE

Started Mistral Workflow tripleo.package_update.v1.get_config. Execution ID: e0327a73-d609-4182-9b81-894aa4759c55
Waiting for messages on queue 'tripleo' with no timeout.
{u'execution': {u'created_at': u'2018-06-01 02:47:58',
                u'id': u'e0327a73-d609-4182-9b81-894aa4759c55',
                u'input': {u'container': u'overcloud',
                           u'queue_name': u'tripleo'},
                u'name': u'tripleo.package_update.v1.get_config',
                u'params': {u'namespace': u''},
                u'spec': {u'input': [u'container',
                                     {u'queue_name': u'tripleo'}],
                          u'name': u'get_config',
                          u'tags': [u'tripleo-common-managed'],
                          u'tasks': {u'get_config': {u'action': u'tripleo.config.get_overcloud_config container=<% $.container %>',
                                                     u'name': u'get_config',
                                                     u'on-complete': u'send_message',
                                                     u'publish': {u'message': u'<% task().result %>',
                                                                  u'status': u'SUCCESS'},
                                                     u'publish-on-error': {u'message': u'Init Minor update failed',
                                                                           u'status': u'FAILED'},
                                                     u'type': u'direct',
                                                     u'version': u'2.0'},
                                     u'send_message': {u'action': u'zaqar.queue_post',
                                                       u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>',
                                                                                                       u'message': u"<% $.get('message', '') %>",
                                                                                                       u'status': u"<% $.get('status', 'SUCCESS') %>"},
                                                                                          u'type': u'tripleo.package_update.v1.package_update_plan'}},
                                                                  u'queue_name': u'<% $.queue_name %>'},
                                                       u'name': u'send_message',
                                                       u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}],
                                                       u'type': u'direct',
                                                       u'version': u'2.0'}},
                          u'version': u'2.0'},
                u'updated_at': u'2018-06-01 02:47:58'},
 u'message': u'Init Minor update failed',
 u'status': u'FAILED'}

Version-Release number of selected component (if applicable):
python-tripleoclient-9.2.1-12.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy newton with 3 controller + 2 compute nodes + 3 ceph OSD nodes
2. Upgrade undercloud to ocata/pike/queens
3. Convert deprecated parameters to their new correspondents according to https://doc-stage.usersys.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/fast_forward_upgrades/#preparing_for_deprecated_parameters

Parameters converted: OvercloudControlFlavor, NovaComputeExtraConfig, NovaComputeSchedulerHints, NovaComputeIPs
4. Run openstack overcloud ffwd-upgrade prepare:

openstack overcloud ffwd-upgrade prepare --templates /usr/share/openstack-tripleo-heat-templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-management.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/enable-cpu-pinning.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/disk-layout.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/scheduler_hints_env.yaml \
-e ~/openstack_deployment/environments/ips-from-pool-all.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \
-e ~/openstack_deployment/environments/custom_hiera.yaml \
-e $HOME/ffu_repos.yaml \
--container-registry-file /home/stack/virt/docker-images.yaml \

Actual results:
Fails.

Expected results:
Doesn't fail.

Additional info:
Attaching sosreport and ffwd prepare output.

[Carlos Camacho (ccamacho)]

Marius already tested the fix.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571765
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=8f7f5d2c6a62515e63dd2a11e61d2734ef31d1af
Submitter: Zuul
Branch: master

commit 8f7f5d2c6a62515e63dd2a11e61d2734ef31d1af
Author: Mathieu Bultel <email address hidden>
Date: Fri Jun 1 17:25:16 2018 +0200

    Get deployment_stack_id via deployment_ref for config download

    When the deployment is a software_resource, trying to get the
    deployment_stack_id will failed with a http stack not found.
    We need to get the deployment_stack_id with the dpeloyment
    links.

    Closes-Bug: 1774961
    Resolves: rhbz#1584951
    Change-Id: I7f51a1d89ac6518d160b037615501dda745cb6e7

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571983
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=726c603fb742c2d89e6dba820dcd8a47094d7347
Submitter: Zuul
Branch: stable/queens

commit 726c603fb742c2d89e6dba820dcd8a47094d7347
Author: Mathieu Bultel <email address hidden>
Date: Fri Jun 1 17:25:16 2018 +0200

    Get deployment_stack_id via deployment_ref for config download

    When the deployment is a software_resource, trying to get the
    deployment_stack_id will failed with a http stack not found.
    We need to get the deployment_stack_id with the dpeloyment
    links.
    Cherry pick from 8f7f5d2c6a62515e63dd2a11e61d2734ef31d1af

    Closes-Bug: 1774961
    Resolves: rhbz#1584951
    Change-Id: I7f51a1d89ac6518d160b037615501dda745cb6e7

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 9.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 8.6.3 release.