Both have many permission denied errors on opening cert files:
2021-03-08T06:28:04.106Z|00005|stream_ssl|ERR|/opt/stack/data/CA/int-ca/private/devstack-cert.key: stat failed (Permission denied)
2021-03-08T06:28:04.107Z|00006|stream_ssl|ERR|SSL_use_PrivateKey_file: error:0200100D:system library:fopen:Permission denied
2021-03-08T06:28:04.107Z|00007|stream_ssl|ERR|/opt/stack/data/CA/int-ca/private/devstack-cert.key: stat failed (Permission denied)
Which can explain why neutron-server has issue connecting then to ssl:6641/6642
Compared to a Focal deployment where ovsdb-server process runs as root, Fedora has it running as its own user openvswitch. And cert files are user-only readable by stack user
I will try a job update that disables TLS to see if other issues are hidden behind (as possible from previous comments by Terry), this could also be an acceptable fix for the job - and in that case a later devstack-general fix to have tls working with Fedora jobs
OK I was about to report new bug for this job, but we apparently already have one!
As can be seen from this bug dates, the job has failed for some time, confirmed with recent runs: /zuul.openstack .org/builds? job_name= neutron- ovn-tempest- ovs-master- fedora& project= openstack/ neutron /439bb5cdebadfd b73b9f- ace28e47b1213bd ab56305960a7f80 56.ssl. cf1.rackcdn. com/periodic/ opendev. org/openstack/ neutron/ master/ neutron- ovn-tempest- ovs-master- fedora/ 4b526e7/ controller/ logs/screen- q-svc.txt
https:/
And recent neutron-server failure log is still similar:
https:/
Digging in other logs, I think I got the root cause though: TLS cert files are not readable by ovs daemons: /439bb5cdebadfd b73b9f- ace28e47b1213bd ab56305960a7f80 56.ssl. cf1.rackcdn. com/periodic/ opendev. org/openstack/ neutron/ master/ neutron- ovn-tempest- ovs-master- fedora/ 4b526e7/ controller/ logs/openvswitc h/ovs-vswitchd_ log.txt /439bb5cdebadfd b73b9f- ace28e47b1213bd ab56305960a7f80 56.ssl. cf1.rackcdn. com/periodic/ opendev. org/openstack/ neutron/ master/ neutron- ovn-tempest- ovs-master- fedora/ 4b526e7/ controller/ logs/openvswitc h/ovsdb- server_ log.txt
https:/
https:/
Both have many permission denied errors on opening cert files: 08T06:28: 04.106Z| 00005|stream_ ssl|ERR| /opt/stack/ data/CA/ int-ca/ private/ devstack- cert.key: stat failed (Permission denied) 08T06:28: 04.107Z| 00006|stream_ ssl|ERR| SSL_use_ PrivateKey_ file: error:0200100D: system library: fopen:Permissio n denied 08T06:28: 04.107Z| 00007|stream_ ssl|ERR| /opt/stack/ data/CA/ int-ca/ private/ devstack- cert.key: stat failed (Permission denied)
2021-03-
2021-03-
2021-03-
Which can explain why neutron-server has issue connecting then to ssl:6641/6642
Compared to a Focal deployment where ovsdb-server process runs as root, Fedora has it running as its own user openvswitch. And cert files are user-only readable by stack user
I will try a job update that disables TLS to see if other issues are hidden behind (as possible from previous comments by Terry), this could also be an acceptable fix for the job - and in that case a later devstack-general fix to have tls working with Fedora jobs