overcloud upgrade fails with error: Timed out waiting for messages from Execution
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
High
|
Jose Luis Franco |
Bug Description
Description
===========
When trying to upgrade an overcloud from Ocata to Pike (with containers) I am always obtaining the same error:
(undercloud) [stack@undercloud ~]$ openstack overcloud deploy --templates $THT --libvirt-type qemu --ntp-server pool.ntp.org -e $THT/overcloud-
Started Mistral Workflow tripleo.
Waiting for messages on queue '91e97a86-
Removing the current plan files
Uploading new plan files
Started Mistral Workflow tripleo.
Plan updated.
Processing templates in the directory /tmp/tripleocli
Started Mistral Workflow tripleo.
Timed out waiting for messages from Execution (ID: 0d0f53a6-
Steps to reproduce
==================
Follow this instructions: http://
1. Deploy Pike undercloud and overcloud using TripleO-quickstart:
bash ./tripleo-
export VIRTHOST=127.0.0.2
export CONFIG=
bash ./tripleo-
2. Delete overcloud:
source ~/stackrc
openstack stack delete overcloud
3. Delete images and load new ones
4. Download tht for ocata and deploy Ocata overcloud (non-HA)
git clone -b stable/ocata https:/
openstack overcloud deploy \
--libvirt-type qemu \
--ntp-server pool.ntp.org \
--templates /home/stack/
-e /home/stack/
5. Once deployed, download tht-master and load docker images:
git clone https:/
openstack overcloud container image upload --config-file /usr/share/
6. Perform overcloud upgrade:
source ~/stackrc
export THT=/home/
openstack overcloud deploy
--templates $THT
--libvirt-type qemu
--ntp-server pool.ntp.org
-e $THT/overcloud-
-e $THT/environmen
-e upgrade_repos.yaml
-e $THT/environmen
-e $THT/environmen
-e $THT/environmen
-e docker_
Expected result
===============
Successful upgrade to Pike with containerized services
Actual result
=============
Removing the current plan files
Uploading new plan files
Started Mistral Workflow tripleo.
Plan updated.
Processing templates in the directory /tmp/tripleocli
Started Mistral Workflow tripleo.
Timed out waiting for messages from Execution (ID: 0d0f53a6-
Environment
===========
Dell PowerEdge R720
Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
32059 MB memory, 135026 GB disk space
Changed in tripleo: | |
milestone: | none → pike-rc1 |
importance: | Undecided → High |
When increasing the timeout in wait_for_messages in:
https:/ /github. com/openstack/ python- tripleoclient/ blob/master/ tripleoclient/ workflows/ parameters. py#L117
from 60 seconds to 180, the upgrade continues with no errors:
(undercloud) [stack@undercloud ~]$ openstack overcloud deploy --templates $THT --libvirt-type qemu --ntp-server pool.ntp.org -e $THT/overcloud- resource- registry- puppet. yaml -e $THT/environmen ts/major- upgrade- composable- steps.yaml -e upgrade_repos.yaml -e $THT/environmen ts/docker. yaml -e $THT/environmen ts/major- upgrade- composable- steps-docker. yaml -e $THT/environmen ts/docker- centos- tripleoupstream .yaml -e docker_ registry. yaml validations. v1.check_ pre_deployment_ validations. Execution ID: 050fb650- f1e3-4756- a9a1-1c4d0d9bfc ab f36c-4d33- a097-72ab9460fd 0a' with no timeout. plan_management .v1.update_ deployment_ plan. Execution ID: 9a1fd37f- 608f-4c7e- 9c77-db9a5a2cae 9b ent-cCvJFU/ tripleo- heat-templates plan_management .v1.get_ deprecated_ parameters. Execution ID: 67fdb863- afdb-4e0b- b13b-6ce70f06cb 3d ent-cCvJFU/ tripleo- heat-templates deployment. v1.deploy_ plan. Execution ID: 89d3d292- fe00-44d0- b135-5bbda70f5f d4 ServiceNetMap- 74n4ytnllkaf] : UPDATE_IN_PROGRESS Stack UPDATE started erBlacklistDict ]: CREATE_IN_PROGRESS state changed ServiceNetMap- 74n4ytnllkaf. ServiceNetMapVa lue]: UPDATE_IN_PROGRESS state changed ServiceNetMap- 74n4ytnllkaf. ServiceNetMapVa lue]: UPDATE_COMPLETE state changed ServiceNetMap- 74n4ytnllkaf] : UPDATE_COMPLETE Stack UPDATE completed successfully ord]: UPDATE_IN_PROGRESS state changed Networks- qfmqqouxf3rl] : UPDATE_IN_PROGRESS Stack UPDATE started
Started Mistral Workflow tripleo.
Waiting for messages on queue 'e7f83872-
Removing the current plan files
Uploading new plan files
Started Mistral Workflow tripleo.
Plan updated.
Processing templates in the directory /tmp/tripleocli
Started Mistral Workflow tripleo.
Deploying templates in the directory /tmp/tripleocli
Started Mistral Workflow tripleo.
2017-08-08 10:49:57Z [ServiceNetMap]: UPDATE_IN_PROGRESS state changed
2017-08-08 10:49:58Z [overcloud-
2017-08-08 10:49:58Z [DeploymentServ
2017-08-08 10:49:59Z [overcloud-
2017-08-08 10:49:59Z [Networks]: UPDATE_IN_PROGRESS state changed
2017-08-08 10:49:59Z [overcloud-
2017-08-08 10:49:59Z [overcloud-
2017-08-08 10:50:00Z [HorizonSecret]: UPDATE_IN_PROGRESS state changed
2017-08-08 10:50:00Z [PcsdPassword]: UPDATE_IN_PROGRESS state changed
2017-08-08 10:50:00Z [MysqlRootPassw
2017-08-08 10:50:00Z [overcloud-
2017-08-08 10:50:00Z [RabbitCookie]: UPDATE_IN_PROGRESS state changed