CentOS 9 standalone jobs fail because of failing container build

[Takashi Kajinami]

Description
===========
We are currently replacing CentOS 8 TripleO jobs in puppet repos by CentOS 9 TripleO jobs.

The base change in p-o-i:
 https://review.opendev.org/c/openstack/puppet-openstack-integration/+/828525

Actual job update in individual repos:
 https://review.opendev.org/c/openstack/puppet-cinder/+/828527

During migration we noticed CentOS 9 jobs are frequently (>80%) failing, because of a failure with container build process.

https://zuul.opendev.org/t/openstack/build/31ca890878af455bbed038b9f489f2d0

https://03aa714b37a788fb6c9f-73eca520d44e7bdb73532e4ac34bedaa.ssl.cf1.rackcdn.com/828527/1/check/puppet-cinder-tripleo-standalone/31ca890/logs/undercloud/home/zuul/container-builds/b1d67c4a-4fb3-47ce-b7e4-ea9539d16516/base/base-build.log
~~~
...
Failed to set locale, defaulting to C.UTF-8
warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.

This system is not registered with an entitlement server. You can use subscription-manager to register.

warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
Last metadata expiration check: 0:00:14 ago on Thu Feb 10 07:33:18 2022.
Dependencies resolved.
==========================================================================================================
 Package Arch Version Repository Size
==========================================================================================================
Installing:
 openstack-tripleo-common-containers noarch 16.3.1-0.20220203092303.3eae6a0.el9 delorean-current 67 k

Transaction Summary
==========================================================================================================
Install 1 Package

Total download size: 67 k
Installed size: 120 k
Downloading Packages:
openstack-tripleo-common-containers-16.3.1-0.20 17 MB/s | 67 kB 00:00
--------------------------------------------------------------------------------
Total 6.5 MB/s | 67 kB 00:00
Running transaction check
warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
RPM: warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
RPM: error: cannot open Packages index using bdb_ro - Operation not permitted (1)
RPM: error: cannot open Packages database in /var/lib/rpm
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Could not run transaction.
~~~

Looking at logs from tcib it seems the build process is using ubi8, which seems incorrect.

https://03aa714b37a788fb6c9f-73eca520d44e7bdb73532e4ac34bedaa.ssl.cf1.rackcdn.com/828527/1/check/puppet-cinder-tripleo-standalone/31ca890/logs/undercloud/home/zuul/container_image_build.log
~~~
START with options: tripleo container image build --base ubi8 --debug --distro centos --exclude neutron-mlnx-agent --namespace tripleomastercentos9 --prefix openstack --push --registry 127.0.0.1:5001 --tag 72664b3968141cf014fa161d4a69f0a6 --volume /etc/yum.repos.d:/etc/distro.repos.d:z --volume /etc/pki/rpm-gpg:/etc/pki/rpm-gpg:z --volume /etc/dnf/vars:/etc/dnf/vars:z --volume /opt/gating_repo:/opt/gating_repo:z --work-dir /home/zuul/container-builds --tcib-extras tcib_release=9 --tcib-extras tcib_python_version=3.9
~~~

It might be a problem caused by wrong job definition but I've not yet found what is wrong.
I'd appreciate any help to dig into this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/828736