[master/Rocky]Fs035 job fails in promotion because of heat stack timeout

[Sagi (Sergey) Shnaidman]

fs035 OVB job fails in master promotion jobs because of heat timeout (3 times in a row)

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/7631ec2/

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/7631ec2/logs/undercloud/var/log/containers/heat/heat-engine.log.1.gz

ERROR heat.engine.resource Traceback (most recent call last):
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 924, in _action_recorder
ERROR heat.engine.resource yield
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1037, in _do_action
ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper
ERROR heat.engine.resource step = next(subtask)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 986, in action_handler_task
ERROR heat.engine.resource done = check(handler_data)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 409, in check_create_complete
ERROR heat.engine.resource return self._check_status_complete(self.CREATE)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 463, in _check_status_complete
ERROR heat.engine.resource action=action)
ERROR heat.engine.resource ResourceFailure: resources[0]: Stack CREATE cancelled

[Rabi Mishra (rabi)]

I guess these nova/ironic errors are relevant here..

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/7631ec2/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz?level=ERROR

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/7631ec2/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz?level=ERROR#_2018-11-04_07_35_45_879

Something wrong with rdocloud?

[chandan kumar (chkumar246)]

We are seeing the same issue on rocky also
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-rocky/1774d66/logs/undercloud/var/log/containers/heat/heat-engine.log.txt.gz?level=ERROR

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/3c5f881/logs/undercloud/var/log/containers/heat/heat-engine.log.txt.gz?level=ERROR

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/204561b/logs/undercloud/var/log/containers/heat/heat-engine.log.txt.gz?level=ERROR

[chandan kumar (chkumar246)]

Here is more timed out https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/309c423/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz?level=ERROR

[chandan kumar (chkumar246)]

fs002 is also failing with overcloud image prepare during introspection timed out https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/3f89efa/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz?level=ERROR

[chandan kumar (chkumar246)]

Some more logs from fs02 https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/cc058e9/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz?level=ERROR

[chandan kumar (chkumar246)]

InstanceDeployFailure: Failed to provision instance 32014403-2029-42a1-ad83-aa60e40222d0: Timeout reached while waiting for callback for node c2eb86f0-03e6-423d-8185-9299e64bde1c https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-rocky-upload/9d18325/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz?level=ERROR#_2018-11-25_13_54_39_585

[Marios Andreou (marios-b)]

ovb has been upgraded hopefully will address the bug (from a call i am in just now thanks)

[Kieran Forde (kieran-forde)]

I've seen the OVB jobs are passing again but it is un-related to the upgrade of rdo-cloud (sort of).

The OVS upgrade should tackle the instability we've seen when rdo-cloud is under load.

BUT what I found to fix the OVB issue was in the heat.conf (auth_uri).

What I noticed when looking at the failures was there was a inability to reach http://172.16.0.9:5000.

This is the internal keystone URL, however my understanding is that any instances that heat creates and heat itself need to reach this URL.

So I changed it to the public URL [https://phx2.cloud.rdoproject.org:13000 and restarted heat-engine on the controllers.

Since then things have been passing.

I will look at where this was set and how to fix the value.

[Sorin Sbarnea (ssbarnea)]

Let create a a logstash query + elastic-recheck entry on this to measure recurrence.

[wes hayutin (weshayutin)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-rocky