Comment 6 for bug 1896537

Revision history for this message
Alex Schultz (alex-schultz) wrote :

So we have a change where it passed in check but not in gate (https://review.opendev.org/#/c/750071/). The difference in the packages consisted of:

pass:
 openstack-selinux-0.8.23-0.20200821105100.f05f4b2.el7.noarch
 openstack-tripleo-heat-templates-10.6.3-0.20200905130014.2026fa4.el7.noarch
 puppet-neutron-14.4.1-0.20200404011627.4aae155.el7.noarch

fail:
 openstack-selinux-0.8.24-0.20200826121419.53b8b2e.el7.noarch
 openstack-tripleo-heat-templates-10.6.3-0.20200908173018.2026fa4.el7.noarch
 puppet-neutron-14.4.1-0.20200908161637.080fd53.el7.noarch

Given that this was a change to THT, it leaves openstack-selinux or puppet-neutron as the culprit.

The only change in puppet-neutron is https://review.opendev.org/#/c/749969/ which lowered the number of workers.

Alternatively I did notice that selinux is enabled on the stein job upstream (this should be permissive)

SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31

https://b711d185688da3b864bc-5593d50c131879f6a486eeedbad80e3c.ssl.cf2.rackcdn.com/750071/1/gate/tripleo-ci-centos-7-standalone-upgrade-stein/91e3191/logs/undercloud/var/log/extra/selinux.txt

And we have the following denial:
/var/log/audit/audit.log.1:type=AVC msg=audit(1599582437.978:2102): avc: denied { net_broadcast } for pid=5066 comm="ovs-vswitchd" capability=11 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability permissive=0

https://b711d185688da3b864bc-5593d50c131879f6a486eeedbad80e3c.ssl.cf2.rackcdn.com/750071/1/gate/tripleo-ci-centos-7-standalone-upgrade-stein/91e3191/logs/undercloud/var/log/extra/denials.txt

However we had the same denial on the passing job. So if this is code related, it likely is the number of workers change in puppet-neutron.
/var/log/audit/audit.log.1:type=AVC msg=audit(1599307009.320:2793): avc: denied { net_broadcast } for pid=5013 comm="ovs-vswitchd" capability=11 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability permissive=0
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a9d/750071/1/check/tripleo-ci-centos-7-standalone-upgrade-stein/a9d2f90/logs/undercloud/var/log/extra/denials.txt

SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a9d/750071/1/check/tripleo-ci-centos-7-standalone-upgrade-stein/a9d2f90/logs/undercloud/var/log/extra/selinux.txt

The openstack-selinux changes are likely:
https://github.com/redhat-openstack/openstack-selinux/commit/f6ad869bb125b8a1834f62bfa1aeffe2a10ef504
https://github.com/redhat-openstack/openstack-selinux/pull/67

Since the denial is the same between each version, I'm thinking it's the lowering of ovn metadata workers.