[fs035][master] multinode periodic job failed at overcloud deploy giving haproxy_public_tls_inject_short_bootstrap_node_name is undefined

[chandan kumar]

FS035 multinode periodic job failed at overcloud deploy giving following error from the logs:
https://logs.rdoproject.org/48/17348/5/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/24f7424/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-21_10_37_37

2018-11-21 10:37:37 | TASK [set is_bootstrap_node fact] **********************************************
2018-11-21 10:37:37 | Wednesday 21 November 2018 10:37:33 +0000 (0:00:00.353) 0:03:19.573 ****
2018-11-21 10:37:37 | fatal: [overcloud-controller-2]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'haproxy_public_tls_inject_short_bootstrap_node_name' is undefined\n\nThe error appears to have been in '/var/lib/mistral/overcloud/Controller/host_prep_tasks.yaml': line 171, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n path: '{{cert_path}}'\n - name: set is_bootstrap_node fact\n ^ here\n"}
2018-11-21 10:37:37 | fatal: [overcloud-controller-1]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'haproxy_public_tls_inject_short_bootstrap_node_name' is undefined\n\nThe error appears to have been in '/var/lib/mistral/overcloud/Controller/host_prep_tasks.yaml': line 171, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n path: '{{cert_path}}'\n - name: set is_bootstrap_node fact\n ^ here\n"}
2018-11-21 10:37:37 | fatal: [overcloud-controller-0]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'haproxy_public_tls_inject_short_bootstrap_node_name' is undefined\n\nThe error appears to have been in '/var/lib/mistral/overcloud/Controller/host_prep_tasks.yaml': line 171, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n path: '{{cert_path}}'\n - name: set is_bootstrap_node fact\n ^ here\n"}
2018-11-21 10:37:37 | skipping: [overcloud-novacompute-0] => {"changed": false, "skip_reason": "Conditional result was False"}

from rpm-qa list: https://logs.rdoproject.org/48/17348/5/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/24f7424/logs/undercloud/var/log/extra/rpm-list.txt.gz

openstack-tripleo-heat-templates-10.0.1-0.20181121051529.07241f3.el7.noarch is used point to this review: https://review.openstack.org/#/c/611800/ -> Revert "Revert "Convert *tasks from bootstrap_nodeid to short_bootstrap_node_name""

It might caused the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/619215

[Steven Hardy (shardy)]

Ok so I investigated, and this is because puppet/services/haproxy-public-tls-inject.yaml is not included via composable services, so it doesn't end up in the EnabledServices list calculated in overcloud.j2.yaml.

This is a special-case where we directly instantiate OS::TripleO::Services::HAProxyPublicTLS inside ./puppet/services/haproxy.yaml, which is an anti-pattern in terms of composable services, but the fix I proposed should work (since these aren't really "services" but additional configuration for haproxy).

If CI passes I'll removed the WIP and we can go ahead with changing to haproxy_short_bootstrap_node_name.

[chandan kumar (chkumar246)]

Looks like fs01 also got affected https://logs.rdoproject.org/39/17339/7/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/e5381fc/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-21_12_10_35

[Steven Hardy (shardy)]

> Looks like fs01 also got affected

We'll hit this problem anywhere TLS is enabled I think.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619215
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fd088e2fe70c598a3d5e40147e25b0c2f6d6dadc
Submitter: Zuul
Branch: master

commit fd088e2fe70c598a3d5e40147e25b0c2f6d6dadc
Author: Steven Hardy <email address hidden>
Date: Wed Nov 21 11:31:39 2018 +0000

    Adjust haproxy-public-tls-inject.yaml bootstrap variable

    In RDO CI we're seeing this undefined, but haproxy_short_bootstrap_node_name
    is defined, which proves https://review.openstack.org/#/c/605046/ is included
    and working.

    The root cause is that the haproxy_public_tls_inject_service is actually
    created via the haproxy template as a nested stack, so we need to use
    haproxy_short_bootstrap_node_name instead

    Change-Id: I870825140b8947a1845307b5bec1bcff387c15c0
    Closes-Bug: #1804433

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.2.0 release.