[master] container build job failed while deploying undercloud with HTTPInternalServerError: ERROR: Internal Error

[chandan kumar]

master periodic container build is failing while deploying undercloud returns HTTPInternalServerError: ERROR: Internal Error during heat stack create.
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/840c38b/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-11-27_15_00_59

2018-11-27 15:00:58 | ** Performing Heat stack create.. **
2018-11-27 15:00:59 | Exception: ERROR: Internal Error
2018-11-27 15:00:59 | Traceback (most recent call last):
2018-11-27 15:00:59 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1150, in _standalone_deploy
2018-11-27 15:00:59 | parsed_args)
2018-11-27 15:00:59 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 760, in _deploy_tripleo_heat_templates
2018-11-27 15:00:59 | stack = orchestration_client.stacks.create(**stack_args)
2018-11-27 15:00:59 | File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 171, in create
2018-11-27 15:00:59 | data=kwargs, headers=headers)
2018-11-27 15:00:59 | File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 292, in post
2018-11-27 15:00:59 | return self.client_request("POST", url, **kwargs)
2018-11-27 15:00:59 | File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 282, in client_request
2018-11-27 15:00:59 | resp, body = self.json_request(method, url, **kwargs)
2018-11-27 15:00:59 | File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 271, in json_request
2018-11-27 15:00:59 | resp = self._http_request(url, method, **kwargs)
2018-11-27 15:00:59 | File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 234, in _http_request
2018-11-27 15:00:59 | raise exc.from_response(resp)
2018-11-27 15:00:59 | HTTPInternalServerError:

Here is the tracback from the heat launcher:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/840c38b/logs/undercloud/var/log/heat-launcher/undercloud_deploy-itinp1/heat.log.txt.gz?level=ERROR

I still need to check from where internal error is coming.

[chandan kumar (chkumar246)]

https://review.openstack.org/611680 might be the culprit reverting it.

[Zane Bitter (zaneb)]

I believe the next patch, https://review.openstack.org/610103 was actually the culprit.

Revert proposed here: https://review.openstack.org/620660

[Marios Andreou (marios-b)]

o/ was about to file a new bug for this as I see it happening still on master promition, examples here:

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/30126eb/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/8e6e3bc/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

the real error is here " http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/30126eb/logs/undercloud/var/log/heat-launcher/undercloud_deploy-QM5zx9/heat.log.txt.gz

2018-11-30 04:58:35.535 10386 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/engine/hot/functions.py", line 71, in result
2018-11-30 04:58:35.535 10386 ERROR oslo_messaging.rpc.server assert self.parameters is not None, "No stack definition in Function"
2018-11-30 04:58:35.535 10386 ERROR oslo_messaging.rpc.server AssertionError: No stack definition in Function

[Zane Bitter (zaneb)]

The fix has now merged. Let me know if you're still seeing issues, but the experimental job passed and Alex confirmed.

[Zane Bitter (zaneb)]

I started working on some additional regression tests to try to make sure we don't have another failure in the same area: https://review.openstack.org/622063

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 12.0.0.0rc1 release candidate.