Just got the following error during an M/N upgrade:
+ openstack overcloud deploy --templates --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --timeout 75 --ntp-server clock.redha
t.com --control-scale 3 --neutron-network-type vxlan --neutron-tunnel-types vxlan -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/opensta
ck-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.ya
ml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml -e /home/st
ack/overcloud-repo.yaml --ntp-server clock.redhat.com -e /tmp/deploy_env.yaml
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
WARNING: openstackclient.common.exceptions is deprecated and will be removed after Jun 2017. Please use osc_lib.exceptions
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2016-09-06 08:25:36 [overcloud]: UPDATE_IN_PROGRESS Stack UPDATE started
2016-09-06 08:25:48 [ServiceNetMap]: CREATE_IN_PROGRESS state changed
...snip..
2016-09-06 08:29:55 [overcloud-allNodesConfig-swx4hduwudr3]: UPDATE_COMPLETE Stack UPDATE completed successfully
2016-09-06 08:29:55 [1]: SIGNAL_COMPLETE Unknown
2016-09-06 08:29:57 [ControllerDeployment]: SIGNAL_COMPLETE Unknown
2016-09-06 08:29:59 [allNodesConfig]: UPDATE_COMPLETE state changed
2016-09-06 08:30:00 [ControllerSwiftDeployment]: UPDATE_FAILED UPDATE aborted
2016-09-06 08:30:00 [1]: SIGNAL_COMPLETE Unknown
2016-09-06 08:30:00 [UpdateWorkflow]: UPDATE_FAILED UPDATE aborted
2016-09-06 08:30:00 [overcloud]: UPDATE_FAILED .enabled_services.list_join: Incorrect arguments to "list_join" should be: "list_join" : [ " ", [ "str1", "str2"]]
2016-09-06 08:30:00 [0]: CREATE_IN_PROGRESS state changed
2016-09-06 08:30:01 [1]: UPDATE_FAILED UPDATE aborted
2016-09-06 08:30:01 [0]: SIGNAL_IN_PROGRESS Signal: deployment 669d746d-932c-4bd2-9099-cd781127d05a succeeded
2016-09-06 08:30:01 [ComputeDeliverUpgradeScriptDeployment]: CREATE_FAILED CREATE aborted
2016-09-06 08:30:01 [UpgradeInitComputeDeployment]: CREATE_FAILED CREATE aborted
2016-09-06 08:30:01 [0]: UPDATE_FAILED UPDATE aborted
2016-09-06 08:30:01 [UpgradeInitControllerDeployment]: CREATE_FAILED CREATE aborted
2016-09-06 08:30:01 [2]: UPDATE_FAILED UPDATE aborted
2016-09-06 08:30:01 [overcloud-UpdateWorkflow-xrcqsywmwkrr]: UPDATE_FAILED Operation cancelled
2016-09-06 08:30:01 [overcloud-ControllerSwiftDeployment-3zkecnhcjnc4]: UPDATE_FAILED Operation cancelled
2016-09-06 08:30:02 [2]: CREATE_IN_PROGRESS state changed
2016-09-06 08:30:06 [2]: SIGNAL_COMPLETE Unknown
Stack overcloud UPDATE_FAILED
Heat Stack update failed.
This is during the major-upgrade-pacemaker-init.yaml step. The repos I used to upgrade:
# undercloud
export CURRENT_VERSION=mitaka
export NEW_VERSION=newton
mkdir /home/stack/REPOBACKUP
sudo mv /etc/yum.repos.d/delorean* /home/stack/REPOBACKUP/
sudo curl -o /etc/yum.repos.d/delorean-$NEW_VERSION.repo http://trunk.rdoproject.org/centos7-$NEW_VERSION/current/delorean.repo
sudo curl -o /etc/yum.repos.d/delorean-deps-$NEW_VERSION.repo http://trunk.rdoproject.org/centos7-$NEW_VERSION/delorean-deps.repo
sudo yum clean all
sudo yum repolist
# overcloud
cat > ~/overcloud-repo.yaml <<EOF
parameter_defaults:
UpgradeInitCommand: |
set -e
curl -L -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos7-newton/delorean-deps.repo
curl -L -o /etc/yum.repos.d/delorean.repo http://trunk.rdoproject.org/centos7-newton/current-passed-ci/delorean.repo
yum clean all
EOF
The heat-engine logs are a bit unhelpful given that "debug = True" in the conf:
heat-engine.log-20160906:2016-09-06 08:30:00.543 539 INFO heat.engine.stack [req-062cb50d-439b-4fdc-90cb-4c73b5155041 b829ee63ca3d4597abf9327135e30510 d26412647be54c849cb1d30a4e4b3998 - - -] Stack UPDATE FAILED (overcloud): .enabled_services.list_join: Incorrect arguments to "list_join" should be: "list_join" : [ " ", [ "str1", "str2"]]
rpm versions: heat-9. 2.0-0.201609010 72004.4d7b5be. el7.centos. noarch heat-templates- 0.0.1-0. 20160905224105. ac2db55. el7.centos. noarch tripleo- puppet- elements- 5.0.0-0. 20160902162220. 01fb147. el7.centos. noarch heatclient- 1.4.0-0. 20160831084943. fb7802e. el7.centos. noarch heat-tests- 7.0.0-0. 20160906023509. 8a2f4dd. el7.centos. noarch heat-common- 7.0.0-0. 20160906023509. 8a2f4dd. el7.centos. noarch tripleo- image-elements- 5.0.0-0. 20160811131857. 98b9c6a. el7.centos. noarch heat-api- 7.0.0-0. 20160906023509. 8a2f4dd. el7.centos. noarch tripleo- common- 5.0.1-0. 20160905143814. 6c39473. el7.centos. noarch tripleo- 0.0.1-0. 20160831005944. 15f0afe. el7.centos. noarch heat-engine- 7.0.0-0. 20160906023509. 8a2f4dd. el7.centos. noarch tripleoclient- 5.0.0-0. 20160905171224. b0d7ce7. el7.centos. noarch tripleo- 5.0.0-0. 20160905160927. 8b0e161. el7.centos. noarch tripleo- heat-templates- 5.0.0-0. 20160906091634. 4488b0f. el7.centos. noarch heat-api- cfn-7.0. 0-0.20160906023 509.8a2f4dd. el7.centos. noarch
[root@undercloud heat]# rpm -qa |grep -E "heat|tripleo"
puppet-
openstack-
openstack-
python2-
python-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
python-
puppet-
openstack-
openstack-