Puppet fails with "Error while evaluating a Resource Statement" on CentOS 9 master during undercloud install

[Jakob Meng]

Our TripleO CI jobs
* periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master-validation [1]
* periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master [2]
* periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-tripleo-master [3]
* periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master [4]
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master [5]

fail during undercloud install on CentOS 9 master with:

  <13>Aug 28 20:33:32 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Apache::Vhost[swift-proxy-api-proxy]: parameter 'port' expects a value of type Undef, Array[Stdlib::Port = Integer[0, 65535]], or Stdlib::Port = Integer[0, 65535], got String (file: /etc/puppet/modules/tripleo/manifests/tls_proxy.pp, line: 56) on node standalone-0.ooo.test

or

  <13>Aug 28 20:11:36 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Apache::Vhost[ipxe_vhost]: parameter 'port' expects a value of type Undef, Array[Stdlib::Port = Integer[0, 65535]], or Stdlib::Port = Integer[0, 65535], got String (file: /etc/puppet/modules/ironic/manifests/pxe.pp, line: 325) on node undercloud.localdomain

or

  <13>Aug 28 20:03:47 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Apache::Vhost[ironic_wsgi]: parameter 'port' expects a value of type Undef, Array[Stdlib::Port = Integer[0, 65535]], or Stdlib::Port = Integer[0, 65535], got String (file: /etc/puppet/modules/openstacklib/manifests/wsgi/apache.pp, line: 353) on node undercloud.localdomain

etc.

[1] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master-validation/16fc597/logs/undercloud/home/zuul/undercloud_install.log.txt.gz
[2] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master/bafd4bf/logs/undercloud/home/zuul/undercloud_install.log.txt.gz
[3] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-tripleo-master/cce7f79/logs/undercloud/home/zuul/undercloud_install.log.txt.gz
[4] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/a7e1de0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
[5] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/662ebdc/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

[Jakob Meng (jm1337)]

[29.08.22 22:05] <slagle> jm1: it's the same as this bug: https://bugs.launchpad.net/puppet-aodh/+bug/1983300
[29.08.22 22:05] <slagle> need to figure out where those port values are getting passed in, and make sure they are integers
[29.08.22 22:06] <slagle> that's probably coming all the way from tripleo-heat-templates, where we can't really cast things to int. so we should probably have a check in puppet-tripleo, to just convert everything to int

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/855106

[Takashi Kajinami (kajinamit)]

$ grep -r port deployment/ | grep wsgi
$

I hope nothing remains now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/855129

[Takashi Kajinami (kajinamit)]

The proposed patch fixes only the issue with ironic_wsgi vhost. We need to fix the other two vhosts as well...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/855155

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/855106
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/8646f99d60f4e6048435e03a301a3e5395f7e771
Submitter: "Zuul (22348)"
Branch: master

commit 8646f99d60f4e6048435e03a301a3e5395f7e771
Author: Takashi Kajinami <email address hidden>
Date: Tue Aug 30 10:37:32 2022 +0900

    Ironic: Remove incorrect override of api port

    The port defined in EndpointMap determines the frontend port for
    the internal endpoint instead of the backend port used by individual
    ironic-api instances.

    Current usage results in passing a string value to the port parameter,
    which is no longer accepted by puppetlabs-apache since 8.0.0.

    Closes-Bug: #1988053
    Change-Id: I08da4e0725d833f87bab88d27da4958076efef9b

[Jakob Meng (jm1337)]

Reopening because:

> The proposed patch fixes only the issue with ironic_wsgi vhost.
> We need to fix the other two vhosts as well..."

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/855155
Committed: https://opendev.org/openstack/puppet-tripleo/commit/136f7a16c5f2526731ec9bfab8e97f2eda80e9d2
Submitter: "Zuul (22348)"
Branch: master

commit 136f7a16c5f2526731ec9bfab8e97f2eda80e9d2
Author: Takashi Kajinami <email address hidden>
Date: Tue Aug 30 18:29:16 2022 +0900

    tls_proxy: Make sure port is an integer

    The apache::vhost::port parameter no longer accepts a string value and
    now an integer value is required. This change makes sure the port is
    converted to an integer value even if a string value is passed.

    Partial-Bug: #1988053
    Change-Id: I52f099b5ab198c8cd02bbe73a4fe491b86c6d442

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/855129
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/63a08d22968d08338d07226f8dc21ff6e8be076a
Submitter: "Zuul (22348)"
Branch: master

commit 63a08d22968d08338d07226f8dc21ff6e8be076a
Author: Takashi Kajinami <email address hidden>
Date: Tue Aug 30 17:39:28 2022 +0900

    Ironic: Require number for IronicIPXEPort

    This parameter determines the tcp port used by PXE server, thus its
    value should be a number rather than a string.

    This also fixes the issue caused by puppetlabs-apache 8.0.0 which
    requires an integer value for apache::vhost::port.

    The logic to generate http_url is refactored using the str_replace
    template function.

    Partial-Bug: #1988053
    Change-Id: I2a47db5a91400d5321ad6f3750cf75281cb89d08

[Takashi Kajinami (kajinamit)]

I believe all the errors mentioned in this bug have been addressed.
We'll see whether there is any remaining issue based on the result from next periodic run.
(In case there is any then we are likely to need a new bug)

[Jakob Meng (jm1337)]

Thank you very much, Takashi!

tripleo component status=Green last_promotion=2022-09-01 00:09:28
Hash_under_test=https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_detail.html?commit_hash=0c501542e0dcb06744b92c5929d74cee91bc2217&distro_hash=f225635ca807e557b30c52786b9b330ce5f54bab
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Jobs which passed: ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master │
│ periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master-validation │
│ periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master │
│ periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-tripl… │
│ periodic-tripleo-ci-centos-9-scenario004-standalone-tripleo-master │
│ periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master │
│ periodic-tripleo-ci-centos-9-standalone-tripleo-master │
│ periodic-tripleo-ci-centos-9-standalone-tripleo-master-validation │
└──────────────────────────────────────────────────────────────────────────────────┘

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 17.0.0 release.