centos8 undercloud-minion job fails - puppet Error while evaluating a Function Call

[Marios Andreou]

At [1][2][3] the tripleo-ci-centos-8-containers-undercloud-minion fails during the minion installation with trace like:

 <13>Jul 30 06:59:52 puppet-user: Error: Evaluation Error: Error while evaluating a Function Call, Please set password for Ironic API service user (file: /etc/puppet/modules/ironic/manifests/api/authtoken.pp, line: 233, column: 5) on node minion.localdomain
 + rc=1
 + '[' false = false ']'
 + set +x
 2021-07-30 06:59:56.362141 | | WARNING | ERROR: Can't run container container-puppet-ironic
 stderr: + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,ironic_config"' /etc/config.pp
 + logger -s -t puppet-user
 <13>Jul 30 06:59:49 puppet-user: Warning: Found multiple default providers for service: base, swiftinit; using base
 <13>Jul 30 06:59:52 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
 <13>Jul 30 06:59:52 puppet-user: (file: /etc/puppet/hiera.yaml)
 <13>Jul 30 06:59:52 puppet-user: Warning: Undefined variable '::deploy_config_name';
 <13>Jul 30 06:59:52 puppet-user: (file & line not available)
 <13>Jul 30 06:59:52 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.8/deprecated_language.html
 <13>Jul 30 06:59:52 puppet-user: (file & line not available)
 <13>Jul 30 06:59:52 puppet-user: Warning: Scope(Class[Ironic]): The database_connection parameter is deprecated and will be \
 <13>Jul 30 06:59:52 puppet-user: removed in a future realse. Use ironic::db::database_connection instead
 <13>Jul 30 06:59:52 puppet-user: Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/profile/base/ironic/authtoken.pp", 61]:["/etc/puppet/modules/tripleo/manifests/profile/base/ironic/conductor.pp", 44]
 <13>Jul 30 06:59:52 puppet-user: (location: /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:36:in `deprecation')
 <13>Jul 30 06:59:52 puppet-user: Error: Evaluation Error: Error while evaluating a Function Call, Please set password for Ironic API service user (file: /etc/puppet/modules/ironic/manifests/api/authtoken.pp, line: 233, column: 5) on node minion.localdomain
 + rc=1

It looks like it was caused by the commit at [4] - this is a gate blocker.

[1] https://de19e57dd79c29d6fa65-9c39e1b31aead2b89889cdc7bed43508.ssl.cf2.rackcdn.com/802537/1/gate/tripleo-ci-centos-8-containers-undercloud-minion/9593bcf/logs/subnode-1/home/zuul/undercloud_minion_install.log
[2] https://ae24a376f86905d600d6-62d0735aef3f9ef87737b188a9096b9f.ssl.cf1.rackcdn.com/802915/1/check/tripleo-ci-centos-8-containers-undercloud-minion/f955d3e/logs/subnode-1/home/zuul/undercloud_minion_install.log
[3] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e90/802742/2/check/tripleo-ci-centos-8-containers-undercloud-minion/e901972/logs/subnode-1/home/zuul/undercloud_minion_install.log
[4] https://opendev.org/openstack/puppet-tripleo/commit/8a847689261e4c97026e2bdeb8ed533260a581f0

[Marios Andreou (marios-b)]

Revert posted 13:54 < opendevreview> Marios Andreou proposed openstack/puppet-tripleo master: Revert "Configure json_rpc/authtoken sections for ironic" https://review.opendev.org/c/openstack/puppet-tripleo/+/802947

unless there are better suggestions :)

[Rabi Mishra (rabi)]

https://review.opendev.org/c/openstack/tripleo-heat-templates/+/802823 should fix it. We should either drop the undercloud minion jobs from master of repurpose them as it is still running installed heat[1] which won't work with ephemeral heat on the main undercloud.

https://b716d73cd64f8bb80e80-894cabf02f3412ccfe3808b1322402eb.ssl.cf2.rackcdn.com/802915/1/check/tripleo-ci-centos-8-containers-undercloud-minion/721e7b7/logs/undercloud/var/log/containers/heat/heat-engine.log

[Marios Andreou (marios-b)]

14:53 < ykarel> ramishra, marios|ruck testing fix with depends-on in
                https://review.rdoproject.org/r/c/rdo-jobs/+/34765

[Rabi Mishra (rabi)]

correct link for minion running installed heat https://b716d73cd64f8bb80e80-894cabf02f3412ccfe3808b1322402eb.ssl.cf2.rackcdn.com/802915/1/check/tripleo-ci-centos-8-containers-undercloud-minion/721e7b7/logs/subnode-1/var/log/containers/heat/heat-engine.log

[Alex Schultz (alex-schultz)]

https://review.opendev.org/c/openstack/tripleo-ci/+/803043 to drop the minion for ephemeral heat versions (e.g. >=wallaby)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/803043
Committed: https://opendev.org/openstack/tripleo-ci/commit/d52d282afe2548f58b05bca936dbbd17f37c3cf9
Submitter: "Zuul (22348)"
Branch: master

commit d52d282afe2548f58b05bca936dbbd17f37c3cf9
Author: Alex Schultz <email address hidden>
Date: Fri Jul 30 08:32:37 2021 -0600

    Stop running minion for Wallaby+

    With the switch to ephemeral heat for the overcloud, the minion is no
    longer viable. This change updates the branches to only run T->V where
    the minion should work and was supported.

    Related-Bug: #1938539
    Change-Id: I5855de77b700124be0d0332ebe6a3f80524c521e

[Marios Andreou (marios-b)]

moving fix released I believe we are done here

if you disagree please reach out to us on oftc #tripleo and move the bug back in progress