RPM: error: cannot open Packages index using bdb_ro - Operation not permitted

[Ananya Banerjee]

Jobs in common component pipeline (master, wallaby c9 wallaby c8) are failing container build with
RPM: warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
RPM: error: cannot open Packages index using bdb_ro - Operation not permitted (1)
RPM: error: cannot open Packages database in /var/lib/rpm
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Could not run transaction.
time="2022-08-02T06:05:22-04:00" level=debug msg="\"/var/tmp/buildah860633471/mnt/buildah-bind-target-13\" is apparently not really mounted, skipping"
time="2022-08-02T06:05:22-04:00" level=debug msg="\"/var/tmp/buildah860633471/mnt/buildah-bind-target-12\" is apparently not really mounted, skipping"
time="2022-08-02T06:05:22-04:00" level=debug msg="\"/var/tmp/buildah860633471/mnt/buildah-bind-target-8\" is apparently not really mounted, skipping"
time="2022-08-02T06:05:22-04:00" level=debug msg="\"/var/tmp/buildah860633471/mnt/buildah-bind-target-6\" is apparently not really mounted, skipping"
time="2022-08-02T06:05:22-04:00" level=debug msg="\"/var/tmp/buildah860633471/mnt/rootfs\" is apparently not really mounted, skipping"
time="2022-08-02T06:05:22-04:00" level=debug msg="\"/var/tmp/buildah860633471/mnt\" is apparently not really mounted, skipping"
time="2022-08-02T06:05:22-04:00" level=debug msg="error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci LANG=en_US.UTF-8] Command:run Args:[dnf install -y openstack-tripleo-common-containers] Flags:[] Attrs:map[] Message:RUN dnf install -y openstack-tripleo-common-containers Original:RUN dnf install -y openstack-tripleo-common-containers}: error while running runtime: exit status 1"

Logs:

https://logserver.rdoproject.org/42/44342/2/check/periodic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby/cda9e66/logs/undercloud/home/zuul/container-builds/876f3388-1ef2-4a6e-8dce-4bff6b47441f/base/base-build.log.txt.gz
https://logserver.rdoproject.org/42/44342/2/check/periodic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby/cda9e66/logs/undercloud/home/zuul/container_image_build.log.txt.gz

[Ronelle Landy (rlandy)]

The same issue reproduces in the integration line if you try build containers in standalone ...

periodic-tripleo-ci-centos-9-scenario003-standalone-master

 openstack-tripleo-common-containers
 RPM: warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
 RPM: error: cannot open Packages index using bdb_ro - Operation not permitted (1)
 RPM: error: cannot open Packages database in /var/lib/rpm
 The downloaded packages were saved in cache until the next successful transaction.
 You can remove cached packages by executing 'dnf clean packages'.
 Error: Could not run transaction.

See comments in: https://bugzilla.redhat.com/show_bug.cgi?id=1960991 and compare to output in:

https://logserver.rdoproject.org/65/31165/71/check/periodic-tripleo-ci-centos-9-scenario003-standalone-master/2c80c9e/logs/undercloud/home/zuul/container-builds/3be4ae8e-8b51-4a4a-9828-4eaec368e679/base/base-build.log.txt.gz

[Ronelle Landy (rlandy)]

The issues is with

openstack-tripleo-common-containers noarch 16.4.1-0.20220729033412.6939ec8.el9 delorean-current 67 k

which is installed from:

openstack-selinux.noarch 0.8.34-0.20220711202841.a241718.el9 @delorean-component-common
openstack-tripleo-common.noarch 16.4.1-0.20220729033412.6939ec8.el9 @delorean-current
openstack-tripleo-common-containers.noarch 16.4.1-0.20220729033412.6939ec8.el9 @delorean-current

[Ronelle Landy (rlandy)]

The above only happens when container images are built without specifying the right base container - registry.access.redhat.com/ubi9 - in this case - otherwise it defaults to ubi8.

Without building containers, there is still the error:

: line 121: syntax error near unexpected token `;;'",

When building containers with the correct base image defined the base container builds.

The deploy still fails with:

022-08-02 18:06:04.597268 | fa163e65-6417-1e82-9729-00000000022d | FATAL | Ensure packages are actually well installed | standalone | error={"changed": true, "cmd": ["rpm", "-V", "lvm2", "jq", "openvswitch", "openstack-heat-agents", "openstack-selinux", "os-net-config", "python3-libselinux", "python3-pyyaml", "puppet-tripleo", "rsync", "tmpwatch", "sysstat"], "delta": "0:00:01.106041", "end": "2022-08-02 18:06:04.566316", "failed_when_result": true, "msg": "non-zero return code", "rc": 2, "start": "2022-08-02 18:06:03.460275", "stderr": "error: %verify(openstack-selinux-0.8.34-0.20220727174553.c3061f5.el9.noarch) scriptlet failed, exit status 2", "stderr_lines": ["error: %verify(openstack-selinux-0.8.34-0.20220727174553.c3061f5.el9.noarch) scriptlet failed, exit status 2"], "stdout": "package openvswitch is not installed\n/usr/share/openstack-selinux/0.8.34/local_settings.sh: line 121: syntax error near unexpected token `;;'\n/usr/share/openstack-selinux/0.8.34/local_settings.sh: line 121: ` if [ $? -le 2 ];; then'", "stdout_lines": ["package openvswitch is not installed", "/usr/share/openstack-selinux/0.8.34/local_settings.sh: line 121: syntax error near unexpected token `;;'", "/usr/share/openstack-selinux/0.8.34/local_settings.sh: line 121: ` if [ $? -le 2 ];; then'"]}

Rekicked the line to pick up latest changes - looks to be fixed now.

[Ananya Banerjee (frenzyfriday)]

the component promoted today - seems to be solved