master jobs are failing undercloud install - Error: Facter: error while resolving custom fact "os_[x]": undefined method `[]' for nil:NilClass

[Ronelle Landy]

Multiple jobs including: tripleo-ci-centos-9-containers-multinode are failing undercloud install with:

stderr: time="2022-06-27T11:25:46Z" level=info msg="podman filtering at log level info"
time="2022-06-27T11:25:46Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
time="2022-06-27T11:25:46Z" level=info msg="Setting parallel job count to 25"
time="2022-06-27T11:25:46Z" level=info msg="Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host"
time="2022-06-27T11:25:46Z" level=info msg="User mount overriding libpod mount at \"/etc/hosts\""
time="2022-06-27T11:25:46Z" level=info msg="Running conmon under slice machine.slice and unitName libpod-conmon-de9d5ff22f08f0c72fa766f8ea605c0100b7aa5a08948720384c1f89bd5f185b.scope"
time="2022-06-27T11:25:46Z" level=info msg="Got Conmon PID as 29979"
time="2022-06-27T11:25:47Z" level=info msg="Received shutdown.Stop(), terminating!" PID=29874
+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,ironic_config"' /etc/config.pp
+ logger -s -t puppet-user
<13>Jun 27 11:25:47 puppet-user: Error: Facter: error while resolving custom fact "os_package_type": undefined method `[]' for nil:NilClass
<13>Jun 27 11:25:51 puppet-user: Error: Facter: error while resolving custom fact "os_workers": undefined method `key?' for nil:NilClass
<13>Jun 27 11:25:51 puppet-user: Error: Facter: error while resolving custom fact "os_workers_heat_engine": undefined method `key?' for nil:NilClass
<13>Jun 27 11:25:51 puppet-user: Error: Facter: error while resolving custom fact "os_workers_keystone": undefined method `key?' for nil:NilClass
<13>Jun 27 11:25:51 puppet-user: Error: Facter: error while resolving custom fact "os_workers_large": undefined method `key?' for nil:NilClass
<13>Jun 27 11:25:51 puppet-user: Error: Facter: error while resolving custom fact "os_workers_small": undefined method `key?' for nil:NilClass
<13>Jun 27 11:25:51 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
<13>Jun 27 11:25:51 puppet-user: (file: /etc/puppet/hiera.yaml)
<13>Jun 27 11:25:51 puppet-user: Warning: Undefined variable '::deploy_config_name';
<13>Jun 27 11:25:51 puppet-user: (file & line not available)
<13>Jun 27 11:25:51 puppet-user: Warning: Unknown variable: '::os'. (file: /etc/puppet/modules/openstacklib/manifests/defaults.pp, line: 9, column: 7)
<13>Jun 27 11:25:51 puppet-user: Error: Evaluation Error: Operator '[]' is not applicable to an Undef Value. (file: /etc/puppet/modules/openstacklib/manifests/defaults.pp, line: 9, column: 7) on node undercloud.localdomain
+ rc=1

and various repetitions of that.

See logs:

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e62/841609/8/check/tripleo-ci-centos-9-containers-multinode/e620bb7/logs/undercloud/home/zuul/undercloud_install.log

https://logserver.rdoproject.org/54/36254/125/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/1b5536d/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

[Rabi Mishra (rabi)]

Sounds like it's factor 4.2 bump.

[Ronelle Landy (rlandy)]

yep

facter.x86_64 1:3.14.19-1.el9s @delorean-master-testing

vs

facter.noarch 1:4.2.10-1.el9s @delorean-master-testing

[Ronelle Landy (rlandy)]

[delorean-master-testing]
name=dlrn-master-testing
baseurl=http://mirror.iad3.inmotion.opendev.org:8080/rdo/centos9-master/deps/latest/
enabled=1
gpgcheck=0
module_hotfixes=1

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/r/c/rdoinfo/+/42273 - Update puppet and facter to last versions for CS9 in Zed - caused the issue

[Takashi Kajinami (kajinamit)]

I've checked the content provider job and confirmed that facter 3, instead of 4 was pulled during image build.

However the subsequent tripleo job pulled facter 4 during package installation.

This inconsistency caused the problem with loading facter cache in containers because the cache data is generated by facter 4 in host while it is loaded by facter 3 inside container.
AFAIK there was a change made in data structure in facter and the generated data is not compatible between facter 3 and 4.

We'd need to wait until the updated package is fully synced to all mirrors.