tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 jobs are failing overcloud deployment with 'KeyError: u'type''

[Ronelle Landy]

featureset039 jobs are failing in check and promotion. We are seeing errors in overcloud-prep-images (introspection errors - which may be cloud related) and overcloud deploy errors:

2019-04-12 17:31:32.066 8 ERROR heat.engine.check_resource [req-a799f5c3-5a74-4b9e-9c2e-e36f5cc7dea4 - admin - default default] Unexpected exception in resource check.: KeyError: u'type'

 [overcloud.ComputeServiceChain.ServiceServerMetadataHook]: CREATE_FAILED resources.ServiceServerMetadataHook: u'type'

2019-04-12 16:31:11 | 2019-04-12 16:30:49Z [ServiceServerMetadataHook.IncomingMetadataSettings]: CREATE_COMPLException occured while running the command

Full logs linked below:
(stein promotion)
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-stein/415c446/logs/undercloud/var/log/containers/heat/heat-engine.log.txt.gz#_2019-04-12_17_31_32_066

(master check)tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 jobs are failing overcloud deployme
https://logs.rdoproject.org/39/651239/10/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/ae8f974/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

The only passing fs039 jobs we see are on rocky - all other branches show failures

[wes hayutin (weshayutin)]

just a guess, but maybe something that merged from https://review.openstack.org/#/q/topic:containers-common+(status:open+OR+status:merged) ?

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039&branch=master&result=FAILURE shows failing jobs since 04/11

[Ronelle Landy (rlandy)]

possible culprits:

<ykarel> <weshay> rlandy|ruck or https://review.openstack.org/#/c/639119/
...
<ykarel> that patch should have missed some tls config

[Ronelle Landy (rlandy)]

see results from revert: https://review.openstack.org/#/c/652140/

[Dan Prince (dan-prince)]

I think I see the issue here. Its because I swapped out a hiera data lookup in t-h-t for OVNSouthboundServerPort in that commit. Let me see if I can post a fix here shortly.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/652437

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/652445

[Dan Prince (dan-prince)]

2 options there ^^, maybe we can use one of them to fix this.

[yatin (yatinkarel)]

<< I think I see the issue here. Its because I swapped out a hiera data lookup in t-h-t for OVNSouthboundServerPort in that commit. Let me see if I can post a fix here shortly.

Not sure if ^^ is the actual issue considering only tls(fs039) job is affected. I mean if this(String/Integer) is the cause why only tls job affected?
The issue i think is the change https://review.openstack.org/#/c/639119/16/deployment/ovn/ovn-metadata-container-puppet.yaml@284, looking at code it seems it looks for .type at when metadata_settings is not null(currently {}) http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/extraconfig/nova_metadata/krb-service-principals.j2.yaml#n68, i can fire up a patch to confirm this meanwhile.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/652499

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/652499
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=23352806b64f4b50a1050426844ad4ce54da6248
Submitter: Zuul
Branch: master

commit 23352806b64f4b50a1050426844ad4ce54da6248
Author: yatin <email address hidden>
Date: Mon Apr 15 08:02:27 2019 +0530

    Fix OVN yaql KeyError: type error

    https://review.openstack.org/#/c/639119/ changed
    the 'metadata_settings' RoleData for ovn-metadata
    to {} which caused a TypeError, this patch reverts
    the metadata_settings to original to fix the issue.

    Closes-Bug: #1824579
    Change-Id: I09ed9c99065c946760c5319086ff0dbccd29a85c

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dan Prince (<email address hidden>) on branch: master
Review: https://review.openstack.org/652437

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dan Prince (<email address hidden>) on branch: master
Review: https://review.openstack.org/652445

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.5.0 release.