Comment 5 for bug 1873770

With debug enabled following Error is reported:-
2020-04-23 08:24:59 | TASK [push certificate content] ************************************************
2020-04-23 08:24:59 | Thursday 23 April 2020 08:24:59 +0000 (0:00:00.116) 0:02:20.602 ********
2020-04-23 08:25:00 | fatal: [overcloud-controller-0]: FAILED! => changed=false
2020-04-23 08:25:00 | checksum: 63c1ca987a881f8562930669e09f8053fcd47750
2020-04-23 08:25:00 | msg: Destination directory /etc/pki/tls/private does not exist
2020-04-23 08:25:00 | fatal: [overcloud-controller-1]: FAILED! => changed=false
2020-04-23 08:25:00 | checksum: 63c1ca987a881f8562930669e09f8053fcd47750
2020-04-23 08:25:00 | msg: Destination directory /etc/pki/tls/private does not exist
2020-04-23 08:25:00 | fatal: [overcloud-controller-2]: FAILED! => changed=false
2020-04-23 08:25:00 | checksum: 63c1ca987a881f8562930669e09f8053fcd47750
2020-04-23 08:25:00 | msg: Destination directory /etc/pki/tls/private does not exist

logs:- https://logserver.rdoproject.org/12/26712/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/bbd78ab/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

/etc/pki/tls/private directory is part of openssl-libs package and should exist by default. The issue(deletion of directories) is same as https://bugs.launchpad.net/tripleo/+bug/1867602.

Collected following info from latest overcloud image which has issue:-
$ curl -O https://images.rdoproject.org/centos8/master/rdo_trunk/724c195e098a6a85afd12fffc11ffbc0/overcloud-full.tar
$ tar -xvf overcloud-full.tar
$ sudo LIBGUESTFS_BACKEND=direct guestmount -i -a overcloud-full.qcow2 /mnt

$ sudo chroot /mnt rpm -Va openssl-libs
missing /etc/pki/tls/misc
missing /etc/pki/tls/private

^^ these directories are removed somehow.

Also the issue is not specific to openssl-libs, it's affecting other directories as well:-

$ sudo chroot /mnt rpm -Va |grep missing
missing /var/lib/iscsi/ifaces
missing /var/lib/iscsi/isns
missing /var/lib/iscsi/nodes
missing /var/lib/iscsi/send_targets
missing /var/lib/iscsi/slp
missing /var/lib/iscsi/static
missing /etc/qemu-ga/fsfreeze-hook.d
missing /var/log/qemu-ga
missing /usr/lib64/libxslt-plugins
missing /usr/share/kdump
missing /var/crash
missing /usr/share/doc/python3-pycurl/tests/tmp
missing /usr/share/i18n
missing /usr/share/i18n/charmaps
missing /usr/share/i18n/locales
missing /etc/NetworkManager/conf.d
missing /etc/NetworkManager/dispatcher.d/no-wait.d
missing /etc/NetworkManager/dispatcher.d/pre-down.d
missing /etc/NetworkManager/dispatcher.d/pre-up.d
missing /etc/NetworkManager/dnsmasq-shared.d
missing /etc/NetworkManager/dnsmasq.d
missing /etc/NetworkManager/system-connections
missing /usr/lib/NetworkManager
missing /usr/lib/NetworkManager/VPN
missing /usr/lib/NetworkManager/conf.d
missing /usr/lib/NetworkManager/system-connections
missing /var/lib/NetworkManager
missing /etc/modules-load.d
missing /usr/lib/systemd/system-sleep
missing /var/empty/sshd
missing /etc/sudoers.d
missing /var/db/sudo
missing /var/db/sudo/lectured
missing c /boot/grub2/grubenv
missing /var/lib/net-snmp/cert_indexes
missing /var/lib/net-snmp/mib_indexes
missing /etc/openldap/certs
missing /usr/libexec/openldap
missing c /etc/yum.repos.d/CentOS-AppStream.repo
missing c /etc/yum.repos.d/CentOS-Base.repo
missing c /etc/yum.repos.d/CentOS-CR.repo
missing c /etc/yum.repos.d/CentOS-Debuginfo.repo
missing c /etc/yum.repos.d/CentOS-Devel.repo
missing c /etc/yum.repos.d/CentOS-Extras.repo
missing c /etc/yum.repos.d/CentOS-HA.repo
missing c /etc/yum.repos.d/CentOS-Media.repo
missing c /etc/yum.repos.d/CentOS-PowerTools.repo
missing c /etc/yum.repos.d/CentOS-Sources.repo
missing c /etc/yum.repos.d/CentOS-Vault.repo
missing c /etc/yum.repos.d/CentOS-centosplus.repo
missing c /etc/yum.repos.d/CentOS-fasttrack.repo
missing /usr/share/doc/policycoreutils
missing /etc/tuned/recommend.d
missing /run/tuned
missing /var/lib/tuned
missing /var/log/tuned
missing /usr/share/doc/python3-cryptography/docs/_static
missing /usr/share/ruby/racc/rdoc
missing /etc/pki/tls/misc
missing /etc/pki/tls/private
missing /etc/binfmt.d
missing /etc/systemd/user
missing /usr/lib/binfmt.d
missing /usr/lib/systemd/system-shutdown
missing /usr/lib/systemd/system/basic.target.wants
missing /usr/lib/systemd/system/dbus.target.wants
missing /usr/lib/systemd/system/default.target.wants
missing /usr/lib/systemd/system/remote-fs.target.wants
missing /usr/lib/systemd/system/runlevel1.target.wants
missing /usr/lib/systemd/system/runlevel2.target.wants
missing /usr/lib/systemd/system/runlevel3.target.wants
missing /usr/lib/systemd/system/runlevel4.target.wants
missing /usr/lib/systemd/system/runlevel5.target.wants
missing /usr/lib/systemd/system/syslog.target.wants
missing /usr/lib/systemd/user-generators
missing /run/cloud-init
missing /var/lib/cloud
missing /var/log/audit
missing /usr/lib/gems/ruby
missing /usr/share/rubygems/rubygems/ssl_certs

On further check to get what's common in all these found that all these are empty directories:-
$ missing=$(sudo chroot /mnt rpm -Va |grep missing|sed -e "s/ c / /"|awk '{print $2}')

$ for file in $missing; do rpm=$(sudo chroot /mnt rpm -qf --queryformat=%{NAME} $file); sudo chroot /mnt rpm -ql $rpm |grep $file ;done
/var/lib/iscsi/ifaces
/var/lib/iscsi/isns
/var/lib/iscsi/nodes
/var/lib/iscsi/send_targets
/var/lib/iscsi/slp
/var/lib/iscsi/static
/etc/qemu-ga/fsfreeze-hook.d
/var/log/qemu-ga
/usr/lib64/libxslt-plugins
/usr/share/kdump
/var/crash
/usr/share/doc/python3-pycurl/tests/tmp
/usr/share/i18n
/usr/share/i18n/charmaps
/usr/share/i18n/locales
/usr/share/i18n/charmaps
/usr/share/i18n/locales
/etc/NetworkManager/conf.d
/etc/NetworkManager/dispatcher.d/no-wait.d
/etc/NetworkManager/dispatcher.d/pre-down.d
/etc/NetworkManager/dispatcher.d/pre-up.d
/etc/NetworkManager/dnsmasq-shared.d
/etc/NetworkManager/dnsmasq.d
/etc/NetworkManager/system-connections
/usr/lib/NetworkManager
/usr/lib/NetworkManager/VPN
/usr/lib/NetworkManager/conf.d
/usr/lib/NetworkManager/system-connections
/usr/lib/NetworkManager/VPN
/usr/lib/NetworkManager/conf.d
/usr/lib/NetworkManager/system-connections
/var/lib/NetworkManager
/etc/modules-load.d
/usr/lib/systemd/system-sleep
/var/empty/sshd
/etc/sudoers.d
/var/db/sudo
/var/db/sudo/lectured
/var/db/sudo/lectured
/boot/grub2/grubenv
/var/lib/net-snmp/cert_indexes
/var/lib/net-snmp/mib_indexes
/etc/openldap/certs
/usr/libexec/openldap
/usr/share/doc/policycoreutils
/etc/tuned/recommend.d
/run/tuned
/var/lib/tuned
/var/log/tuned
/usr/share/doc/python3-cryptography/docs/_static
/usr/share/ruby/racc/rdoc
/etc/pki/tls/misc
/etc/pki/tls/private
/etc/binfmt.d
/etc/systemd/user
/usr/lib/binfmt.d
/usr/lib/systemd/system-shutdown
/usr/lib/systemd/system/basic.target.wants
/usr/lib/systemd/system/dbus.target.wants
/usr/lib/systemd/system/default.target.wants
/usr/lib/systemd/system/remote-fs.target.wants
/usr/lib/systemd/system/runlevel1.target.wants
/usr/lib/systemd/system/runlevel2.target.wants
/usr/lib/systemd/system/runlevel3.target.wants
/usr/lib/systemd/system/runlevel4.target.wants
/usr/lib/systemd/system/runlevel5.target.wants
/usr/lib/systemd/system/syslog.target.wants
/usr/lib/systemd/user-generators
/run/cloud-init
/var/lib/cloud
/var/log/audit
/usr/lib/gems/ruby
/usr/share/rubygems/rubygems/ssl_certs

# trimmed some .repo and .conf files from above output

Some observations:-
- Something is removing empty directories during overcloud image build.
- Removed directories are not related to a specific package, multiple packages affected whenver image has issues.
- Not all empty directories are removed, example in this image /var/log/pcsd empty directory was not removed.

For reference, good image which do not have any missing directory:- https://images.rdoproject.org/centos8/master/rdo_trunk/b3720367a6a0349abcfb06939bed3101/overcloud-full.tar