Standalone deployment fails with error "Found 28 missing module(s)."

[Tom Weininger]

Description
===========
This seems to be somewhat related to https://bugs.launchpad.net/tripleo/+bug/1981353, but I am seeing this today when doing a fresh standalone deployment from master. The error I get is:

ESC[0;31m2022-07-25 04:56:32.444144 | fa163e55-c1c2-0c35-d1a0-000000000239 | FATAL | Ensure packages are actually well installed | tripleo | error={"changed": true, "cmd": ["rpm", "-V", "lvm2", "jq", "openvswitch", "openstack-heat-agents", "openstack-selinux", "os-net-config", "python3-libselinux", "python3-pyyaml", "puppet-tripleo", "rsync", "tmpwatch", "sysstat"], "delta": "0:00:00.631653", "end": "2022-07-25 04:56:32.421745", "failed_when_result": true, "msg": "non-zero return code", "rc": 2, "start": "2022-07-25 04:56:31.790092", "stderr": "error: %verify(openstack-selinux-0.8.34-0.20220711202841.a241718.el9.noarch) scriptlet failed, exit status 1", "stderr_lines": ["error: %verify(openstack-selinux-0.8.34-0.20220711202841.a241718.el9.noarch) scriptlet failed, exit status 1"], "stdout": "package openvswitch is not installed\nMissing os-ovs!\nMissing os-swift!\nMissing os-nova!\nMissing os-neutron!\nMissing os-mysql!\nMissing os-glance!\nMissing os-rsync!\nMissing os-rabbitmq!\nMissing os-keepalived!\nMissing os-keystone!\nMissing os-haproxy!\nMissing os-mongodb!\nMissing os-ipxe!\nMissing os-redis!\nMissing os-cin
der!\nMissing os-httpd!\nMissing os-gnocchi!\nMissing os-collectd!\nMissing os-virt!\nMissing os-dnsmasq!\nMissing os-octavia!\nMissing os-podman!\nMissing os-rsyslog!\nMissing os-pbis!\nMis
sing os-barbican!\nMissing os-logrotate!\nMissing os-certmonger!\nMissing os-timemaster!\nFound 28 missing module(s).", "stdout_lines": ["package openvswitch is not installed", "Missing os-o
vs!", "Missing os-swift!", "Missing os-nova!", "Missing os-neutron!", "Missing os-mysql!", "Missing os-glance!", "Missing os-rsync!", "Missing os-rabbitmq!", "Missing os-keepalived!", "Missi
ng os-keystone!", "Missing os-haproxy!", "Missing os-mongodb!", "Missing os-ipxe!", "Missing os-redis!", "Missing os-cinder!", "Missing os-httpd!", "Missing os-gnocchi!", "Missing os-collect
d!", "Missing os-virt!", "Missing os-dnsmasq!", "Missing os-octavia!", "Missing os-podman!", "Missing os-rsyslog!", "Missing os-pbis!", "Missing os-barbican!", "Missing os-logrotate!", "Miss
ing os-certmonger!", "Missing os-timemaster!", "Found 28 missing module(s)."]}ESC[0m
ESC[1;30m2022-07-25 04:56:32.445116 | fa163e55-c1c2-0c35-d1a0-000000000239 | TIMING | tripleo_bootstrap : Ensure packages are actually well installed | tripleo | 0:00:12.551346 | 0.81sES
ESC[0m

Steps to reproduce
==================
Run standalone deployment using master branch.

Expected result
===============
no errors.

Actual result
=============
Error. See above.

[John Fulton (jfulton-org)]

Introduced by [1] and an attempt to fix was made by [2].

However, my standalone deploys no longer works as I hit this bug.

You can modify the command [3] to make the rpm command not return an error code [4]. The modifications are:

A. switch openvswitch for rdo-openvswitch because of [5]
B. use 'rpm -q' instead of 'rpm -V' for openstack-selinux because of [6]

The following works [7] for my env:

sudo rpm -q lvm2 jq rdo-openvswitch openstack-heat-agents os-net-config python3-libselinux python3-pyyaml puppet-tripleo rsync tmpwatch sysstat openstack-selinux

and seems to achieve the desired effect; i.e. if the above returned a non-zero exit code we could stop the playbook. I'll submit a patch for review.

[1] https://github.com/openstack/tripleo-ansible/commit/2f29bfb7e01d94e42e5e6fcc9305f339d8d5efe4

[2] https://github.com/openstack/tripleo-ansible/commit/a89edcd010c321d4e05aa068a8fb31cb8cded344

[3] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_bootstrap/tasks/packages.yml#L65

[4]
[stack@standalone zed]$ sudo rpm -V lvm2 jq rdo-openvswitch openstack-heat-agents os-net-config python3-libselinux python3-pyyaml puppet-tripleo rsync tmpwatch sysstat
[stack@standalone zed]$ echo $?
0
[stack@standalone zed]$

[5]
[stack@standalone zed]$ sudo rpm -V openvswitch
package openvswitch is not installed
[stack@standalone zed]$ sudo dnf install openvswitch
Repository baseos is listed more than once in the configuration
Repository appstream is listed more than once in the configuration
Last metadata expiration check: 1:52:30 ago on Mon 25 Jul 2022 03:09:15 PM EDT.
Package rdo-openvswitch-2:2.17-2.el9s.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[stack@standalone zed]$

[6]
[stack@standalone zed]$ sudo rpm -V openstack-selinux
Missing os-ovs!
Missing os-swift!
Missing os-nova!
Missing os-neutron!
Missing os-mysql!
Missing os-glance!
Missing os-rsync!
Missing os-rabbitmq!
Missing os-keepalived!
Missing os-keystone!
Missing os-haproxy!
Missing os-mongodb!
Missing os-ipxe!
Missing os-redis!
Missing os-cinder!
Missing os-httpd!
Missing os-gnocchi!
Missing os-collectd!
Missing os-virt!
Missing os-dnsmasq!
Missing os-octavia!
Missing os-podman!
Missing os-rsyslog!
Missing os-pbis!
Missing os-barbican!
Missing os-logrotate!
Missing os-certmonger!
Missing os-timemaster!
Found 28 missing module(s).
error: %verify(openstack-selinux-0.8.34-0.20220711202841.a241718.el9.noarch) scriptlet failed, exit status 1
[stack@standalone zed]$ rpm -q openstack-selinux
openstack-selinux-0.8.34-0.20220711202841.a241718.el9.noarch
[stack@standalone zed]$

[7]
[stack@standalone zed]$ sudo rpm -q lvm2 jq rdo-openvswitch openstack-heat-agents os-net-config python3-libselinux python3-pyyaml puppet-tripleo rsync tmpwatch sysstat openstack-selinux
lvm2-2.03.16-1.el9.x86_64
jq-1.6-9.el9.x86_64
rdo-openvswitch-2.17-2.el9s.noarch
openstack-heat-agents-4.1.0-0.20220708025946.deaefdd.el9.noarch
os-net-config-15.2.1-0.20220629114404.6505f24.el9.noarch
python3-libselinux-3.4-2.el9.x86_64
python3-pyyaml-5.4.1-6.el9.x86_64
puppet-tripleo-16.1.1-0.20220721192904.1d53e71.el9.noarch
rsync-3.2.3-11.el9.x86_64
tmpwatch-2.11-...

[John Fulton (jfulton-org)]

After further inspection of the original patch [1] I have reason to think that I'm not failing because of the rdo-openvswitch vs openvswitch issue. Also, switching to 'rpm -q' would not achieve the original goal of [1] since we want to use the verify option of RPM (we already know the package is installed from [2]). If openvswitch is not causing the failure, then I think the root cause is likely in the verify script [3] of the openstack-selinux package it self.

Perhaps this should be an RDO bug for openstack-selinux-distgit.

[1] https://github.com/openstack/tripleo-ansible/commit/2f29bfb7e01d94e42e5e6fcc9305f339d8d5efe4

[2] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_bootstrap/tasks/packages.yml#L46-L50

[3] https://review.rdoproject.org/r/c/openstack/openstack-selinux-distgit/+/43578

[John Fulton (jfulton-org)]

Reproducer: `sudo rpm -V openstack-selinux` fails on centos-stream9 with latest updates as of today.

[stack@standalone tasks]$ sudo rpm -V -vvv openstack-selinux
ufdio: 1 reads, 17654 total bytes in 0.000003 secs
D: PRAGMA secure_delete = OFF: 0
D: PRAGMA case_sensitive_like = ON: 0
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
D: loading keyring from rpmdb
D: read h# 1356
Header SHA256 digest: OK
Header SHA1 digest: OK
D: Plugin: calling hook init in systemd_inhibit plugin
D: ========== +++ openstack-selinux-0.8.34-0.20220711202841.a241718.el9 noarch/linux 0x0
D: read h# 527
Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
Header SHA256 digest: OK
Header SHA1 digest: OK
D: Requires: /bin/sh YES (db files)
D: Requires: /bin/sh YES (cached)
D: read h# 527
Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
Header SHA256 digest: OK
Header SHA1 digest: OK
D: Requires: /usr/bin/bash YES (db files)
D: read h# 500
Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
Header SHA256 digest: OK
Header SHA1 digest: OK
D: Requires: container-selinux YES (db provides)
D: read h# 686
Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
Header SHA256 digest: OK
Header SHA1 digest: OK
D: Requires: policycoreutils YES (db provides)
D: Requires: policycoreutils YES (cached)
D: Requires: rpmlib(CompressedFileNames) <= 3.0.4-1 YES (rpmlib provides)
D: Requires: rpmlib(FileDigests) <= 4.6.0-1 YES (rpmlib provides)
D: Requires: rpmlib(PayloadFilesHavePrefix) <= 4.0-1 YES (rpmlib provides)
D: Requires: rpmlib(PayloadIsZstd) <= 5.4.18-1 YES (rpmlib provides)
......... /usr/share/licenses/openstack-selinux
ufdio: 2 reads, 17987 total bytes in 0.000009 secs
......... l /usr/share/licenses/openstack-selinux/COPYING
......... /usr/share/openstack-selinux
......... /usr/share/openstack-selinux/0.8.34
ufdio: 2 reads, 10017 total bytes in 0.000003 secs
......... /usr/share/openstack-selinux/0.8.34/local_settings.sh
ufdio: 2 reads, 9658 total bytes in 0.000036 secs
......... /usr/share/selinux/packages/os-barbican.pp.bz2
ufdio: 2 reads, 9787 total bytes in 0.000004 secs
......... /usr/share/selinux/packages/os-certmonger.pp.bz2
ufdio: 2 reads, 9942 total bytes in 0.000002 secs
......... /usr/share/selinux/packages/os-cinder.pp.bz2
ufdio: 2 reads, 9611 total bytes in 0.000002 secs
......... /usr/share/selinux/packages/os-collectd.pp.bz2
ufdio: 2 reads, 9579 total bytes in 0.000012 secs
......... /usr/share/selinux/packages/os-dnsmasq.pp.bz2
ufdio: 2 reads, 10418 total bytes in 0.000003 secs
......... /usr/share/selinux/packages/os-glance.pp.bz2
ufdio: 2 reads, 9618 total bytes in 0.000003 secs
......... /usr/share/selinux/packages/os-gnocchi.pp.bz2
ufdio: 2 reads, 9728 total bytes in...

[Tom Weininger (tweining)]

Thanks for providing a workaround. After making the two mods A and B from your first comment I got a new error:

2022-07-26 04:27:21.020433 | fa163e55-c1c2-e241-7efb-0000000008c5 | FATAL | Enable os_enable_vtpm SELinux boolean for vTPM | tripleo | error={"changed": false, "msg": "SELinux boolean os_enable_vtpm does not exist."}

[Tom Weininger (tweining)]

Tengu suggested on IRC to post the output of "dnf reinstall openstack-selinux" as changing "rpm -V" seems wrong. Here it is:

Failed to resolve typeattributeset statement at /var/lib/selinux/targeted/tmp/modules/400/os-certmonger/cil:5
Failed to resolve AST
OSError: [Errno 0] Error
ValueError: Boolean os_nova_use_execmem is not defined

https://paste.opendev.org/show/bEFGP2E8nLHeBbA0H6om/

[Julie Pichon (jpichon)]

The missing modules indicate that the openstack-selinux package didn't install properly, usually because of a missing symbol or because an underlying selinux library change causing an issue with parsing the policy file. Could you share the versions for all the SELinux packages as well? ($ rpm -qa | grep selinux | sort). I will try to reproduce it too.

[Julie Pichon (jpichon)]

When I convert the os-certmonger policy into cil, line 5 is the following:

(typeattributeset cil_gen_require container_runtime_t)

Do you have container-selinux installed?

[Tom Weininger (tweining)]

$ rpm -qa | grep selinux | sort
container-selinux-2.189.0-1.el9.noarch
flatpak-selinux-1.12.7-2.el9.noarch
ipa-selinux-4.10.0-2.el9.noarch
libselinux-3.4-2.el9.x86_64
libselinux-ruby-3.4-2.el9.x86_64
libselinux-utils-3.4-2.el9.x86_64
openstack-selinux-0.8.34-0.20220711202841.a241718.el9.noarch
openvswitch-selinux-extra-policy-1.0-31.el9s.noarch
python3-libselinux-3.4-2.el9.x86_64
rpm-plugin-selinux-4.16.1.3-15.el9.x86_64
selinux-policy-34.1.37-1.el9.noarch
selinux-policy-targeted-34.1.37-1.el9.noarch

[Julie Pichon (jpichon)]

That is strange, I have the same version locally but can't reproduce the error message... Does reinstalling container-selinux show an error? What is the output of "seinfo --type | grep container"? (You may have to install setools-console to get seinfo)

[John Fulton (jfulton-org)]

On the system where I reproduced this bug I ran the following:

sudo dnf reinstall openstack-selinux container-selinux
sudo rpm -V openstack-selinux

The verify didn't fail after that.

Note that I had run and observed the following prior to doing the above.

  https://paste.opendev.org/show/bW8qMWp2DlrMRCakBgqT/

[Julie Pichon (jpichon)]

From the paste, before reinstalling container-selinux:

$ seinfo --type | grep container
   container_file_t
   container_ro_file_t
   container_t

So the container_runtime_t type was missing because container-selinux didn't install properly, and in turn openstack-selinux couldn't find the symbols it needed either.

We need to find out what prevented the container-selinux rpm from installing correctly the first time, there probably was a "silent" error like in comment 5.

[John Fulton (jfulton-org)]

I built a new system and reproduced the bug:

Instantiate CentOS-Stream-GenericCloud-9-20210830.0.x86_64.qcow2
sudo dnf install python3-tripleo-repos-0.1.1-0.20220719071830.13114fc.el9.noarch
sudo -E tripleo-repos current-tripleo-dev ceph --stream
sudo yum update -y
no reboot
sudo dnf install openstack-selinux

Installing the above package produced a failure though the installatoin completed:

  https://paste.opendev.org/show/bVFkbjrkfShy1DuxL1Ff/

I then saw this error.

[stack@standalone standalone]$ sudo rpm -V openstack-selinux
Missing os-ovs!
Missing os-swift!
Missing os-nova!
Missing os-neutron!
Missing os-mysql!
Missing os-glance!
Missing os-rsync!
Missing os-rabbitmq!
Missing os-keepalived!
Missing os-keystone!
Missing os-haproxy!
Missing os-mongodb!
Missing os-ipxe!
Missing os-redis!
Missing os-cinder!
Missing os-httpd!
Missing os-gnocchi!
Missing os-collectd!
Missing os-virt!
Missing os-dnsmasq!
Missing os-octavia!
Missing os-podman!
Missing os-rsyslog!
Missing os-pbis!
Missing os-barbican!
Missing os-logrotate!
Missing os-certmonger!
Missing os-timemaster!
Found 28 missing module(s).
error: %verify(openstack-selinux-0.8.34-0.20220711202841.a241718.el9.noarch) scriptlet failed, exit status 1
[stack@standalone standalone]$

This is even before I installed python3-tripleoclient:

[stack@standalone standalone]$ rpm -q python3-tripleoclient
package python3-tripleoclient is not installed
[stack@standalone standalone]$

[John Fulton (jfulton-org)]

When I built my new system before I installed python3-tripleo-repos I installed podman
and it had the following problem during container-selinux installation:

 sudo dnf install -y podman

Transaction test succeeded.
Running transaction
  Preparing : 1/1
  Upgrading : iptables-libs-1.8.8-2.el9.x86_64 1/26
  Installing : libnftnl-1.2.2-1.el9.x86_64 2/26
  Installing : fuse3-libs-3.10.2-5.el9.x86_64 3/26
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch 4/26
  Installing : container-selinux-3:2.189.0-1.el9.noarch 4/26
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch 4/26
libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/pp failed with code: 255. (No such file or directory).
container: libsepol.policydb_read: policydb module version 21 does not match my version range 4-20
container: libsepol.sepol_module_package_read: invalid module in module package (at section 0)
container: Failed to read policy package
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
 (No such file or directory).
/usr/sbin/semodule: Failed!

[John Fulton (jfulton-org)]

I managed to get a system where `rpm -V openstack-selinux` by installing in the following order.

New cent9 stream installation instantiated the following:

  CentOS-Stream-GenericCloud-9-20210830.0.x86_64.qcow2

[stack@standalone standalone]$ rpm -qa | grep selinux | sort
libselinux-3.2-6.el9.x86_64
libselinux-utils-3.2-6.el9.x86_64
python3-libselinux-3.2-6.el9.x86_64
rpm-plugin-selinux-4.16.1.3-7.el9.x86_64
selinux-policy-34.1.14-1.el9.noarch
selinux-policy-targeted-34.1.14-1.el9.noarch
[stack@standalone standalone]$

Don't install podman but install tripleo repos and update:

sudo dnf install -y python3-tripleo-repos-0.1.1-0.20220719071830.13114fc.el9.noarch
sudo dnf update -y

After 530 packages are updated (don't reboot) and run the following:

[stack@standalone standalone]$ rpm -qa | grep selinux | sort
libselinux-3.4-3.el9.x86_64
libselinux-utils-3.4-3.el9.x86_64
python3-libselinux-3.4-3.el9.x86_64
rpm-plugin-selinux-4.16.1.3-15.el9.x86_64
selinux-policy-34.1.38-1.el9.noarch
selinux-policy-targeted-34.1.38-1.el9.noarch
[stack@standalone standalone]$

Now let's install container-selinux.

[stack@standalone standalone]$ sudo dnf install container-selinux
Repository baseos is listed more than once in the configuration
Repository appstream is listed more than once in the configuration
CentOS Stream 9 - Extras packages 8.1 kB/s | 8.5 kB 00:01
Dependencies resolved.
==============================================================================================
 Package Architecture Version Repository Size
==============================================================================================
Installing:
 container-selinux noarch 3:2.189.0-1.el9 appstream 49 k

Transaction Summary
==============================================================================================
Install 1 Package

Total download size: 49 k
Installed size: 57 k
Is this ok [y/N]: y
Downloading Packages:
container-selinux-2.189.0-1.el9.noarch.rpm 140 kB/s | 49 kB 00:00
----------------------------------------------------------------------------------------------
Total 139 kB/s | 49 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing : 1/1
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch 1/1
  Installing : container-selinux-3:2.189.0-1.el9.noarch 1/1
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch 1/1
  Verifying : container-selinux-3:2.189.0-1.el9.noarch 1/1

Installed:
  container-selinux-3:2.189.0-1.el9.noarch

Complete!
[stack@standalone standalone]$

[stack@standalone standalone]$ sudo dnf install -y openstack-selinux
Repository baseos is listed more than once in the...

[John Fulton (jfulton-org)]

My summary:
- I hit this issue because container-selinux was not installed correctly.
- It was not installed correctly when podman was installed (before any tripleo/rdo packages)
- If I `sudo -E tripleo-repos current-tripleo-dev --stream` and `sudo dnf update -y` first, then container-selinux is installed correctly.
- I then do not hit this bug

[John Fulton (jfulton-org)]

tweining: Please try the workaround in comment #10 or a new install as per comment #15 and share results here

root cause seems to be container-selinux being installed incorrectly before any tripeo packages touch the system so this might be closer to a centos or podman bug than a tripleo one.

[Julie Pichon (jpichon)]

> container: libsepol.policydb_read: policydb module version 21 does not match my version range 4-20

This is the same issue as described in [1] I believe: there was a bump in policydb version in the upstream SELinux RC that creates a build incompatibility. The system needs to be completely updated before installing additional SELinux packages.

[1] https://github.com/containers/container-selinux/issues/182#issuecomment-1168878309

[Tom Weininger (tweining)]

@jfulton Results of commands as described in #10:
https://paste.opendev.org/show/brZR8nLO61fgXC4IjQMw/

I'm just running the standalone deployment again.

Edit: Deployment succeeded!

[John Fulton (jfulton-org)]

As per #17 perhaps this should be closed as a duplicate of the following issue:

  https://github.com/containers/container-selinux/issues/182#issuecomment-1168878309

[John Fulton (jfulton-org)]

I set it to WONTFIX since it's not an issue tripleo can fix. It's container-selinux issue 182.

Just ensure your system is fully up2date before you install container-selinux and then move to to use standalone.

[Ananya Banerjee (frenzyfriday)]

This is hitting the common component line in master and wallaby c9. I am trying out the workaround in comment #10, meanwhile I am reopening it to track.

[Ananya Banerjee (frenzyfriday)]

Setting it back to Wont fix. the issue we were hitting looked similar to this but was different (and is fixed).