Comment 4 for bug 1911128

Revision history for this message
Bernard Cafarelli (bcafarel) wrote :

OK I was about to report new bug for this job, but we apparently already have one!

As can be seen from this bug dates, the job has failed for some time, confirmed with recent runs:
https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-master-fedora&project=openstack/neutron
And recent neutron-server failure log is still similar:
https://439bb5cdebadfdb73b9f-ace28e47b1213bdab56305960a7f8056.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/4b526e7/controller/logs/screen-q-svc.txt

Digging in other logs, I think I got the root cause though: TLS cert files are not readable by ovs daemons:
https://439bb5cdebadfdb73b9f-ace28e47b1213bdab56305960a7f8056.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/4b526e7/controller/logs/openvswitch/ovs-vswitchd_log.txt
https://439bb5cdebadfdb73b9f-ace28e47b1213bdab56305960a7f8056.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/4b526e7/controller/logs/openvswitch/ovsdb-server_log.txt

Both have many permission denied errors on opening cert files:
2021-03-08T06:28:04.106Z|00005|stream_ssl|ERR|/opt/stack/data/CA/int-ca/private/devstack-cert.key: stat failed (Permission denied)
2021-03-08T06:28:04.107Z|00006|stream_ssl|ERR|SSL_use_PrivateKey_file: error:0200100D:system library:fopen:Permission denied
2021-03-08T06:28:04.107Z|00007|stream_ssl|ERR|/opt/stack/data/CA/int-ca/private/devstack-cert.key: stat failed (Permission denied)

Which can explain why neutron-server has issue connecting then to ssl:6641/6642

Compared to a Focal deployment where ovsdb-server process runs as root, Fedora has it running as its own user openvswitch. And cert files are user-only readable by stack user

I will try a job update that disables TLS to see if other issues are hidden behind (as possible from previous comments by Terry), this could also be an acceptable fix for the job - and in that case a later devstack-general fix to have tls working with Fedora jobs