OVB periodic jobs fail in overcloud deploy task with timeout waiting for messages

[Alfredo Moralejo]

Both periodic-tripleo-ci-centos-7-ovb-ha-oooq and periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 have failed on periodic job with similar error when deploying overcloud:

2017-08-02 07:58:01 | + openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates --libvirt-type qemu --timeout 80 -e /home/jenkins/cloud-names.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /home/jenkins/neutronl3ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation-v6.yaml -e /opt/stack/new/tripleo-ci/test-environments/ipv6-network-templates/network-environment.yaml -e /opt/stack/new/tripleo-ci/test-environments/net-iso.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml --validation-errors-nonfatal --ceph-storage-scale 1 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/ci/environments/ceph-min-osds.yaml --ntp-server pool.ntp.org
2017-08-02 07:58:09 | Waiting for messages on queue 'ddac5d2b-b8c1-4527-a8f7-4fdd9f049973' with no timeout.
2017-08-02 08:06:32 | Timed out waiting for messages from Execution (ID: 0cd8b935-d268-4297-b2e7-f6c8d6bf45ac, State: RUNNING). The WebSocket timed out before the Workflow completed.
2017-08-02 08:06:33 |
2017-08-02 08:06:33 | Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: 9369718a-6682-4e0a-889e-94239474b691
2017-08-02 08:06:33 | Removing the current plan files
2017-08-02 08:06:33 | Uploading new plan files
2017-08-02 08:06:33 | Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: 47ef58b2-03fd-4894-8f58-41baafcdb057
2017-08-02 08:06:33 | Plan updated.
2017-08-02 08:06:33 | Processing templates in the directory /tmp/tripleoclient-OQN54R/tripleo-heat-templates
2017-08-02 08:06:33 | Started Mistral Workflow tripleo.plan_management.v1.get_deprecated_parameters. Execution ID: cc1026a1-49bb-4c2b-86a7-b2588d90a1db
2017-08-02 08:06:33 | WARNING: Following parameters are deprecated and still defined. Deprecated parameters will be removed soon!
2017-08-02 08:06:33 | NeutronL3HA
2017-08-02 08:06:33 | Deploying templates in the directory /tmp/tripleoclient-OQN54R/tripleo-heat-templates
2017-08-02 08:06:33 | Started Mistral Workflow tripleo.deployment.v1.deploy_plan. Execution ID: 0cd8b935-d268-4297-b2e7-f6c8d6bf45ac

Logs in:

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024/e7e67f0/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha-oooq/13f3169/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz

[Thomas Herve (therve)]

So it appears to be Heat's fault.

See http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024/e7e67f0/logs/undercloud/var/log/heat/heat-engine.log.txt.gz#_2017-08-02_08_01_36_329

It makes a 7 minutes pause. That causes the websocket to timeout.

[Thomas Herve (therve)]

It has been introduced by https://review.openstack.org/#/c/477046/

[Dougal Matthews (d0ugal)]

I have proposed the revert and am now waiting for the results of "check experimental". https://review.openstack.org/490075

[Attila Darazs (adarazs)]

https://review.openstack.org/477047 should actually fix this issue. Running the periodic jobs with this change included currently.

[Alfredo Moralejo (amoralej)]

OVB jobs are passing with https://review.openstack.org/477047 so it seems that the issue is fixed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Thomas Herve (<email address hidden>) on branch: master
Review: https://review.openstack.org/490438

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dougal Matthews (<email address hidden>) on branch: master
Review: https://review.openstack.org/490075