FIPS jobs broken

[Gregory Thiemonge]

FIPS jobs are currently broken, they are failing when devstack installs rabbitmq:

+ functions-common:sudo_with_proxies:2345 : sudo http_proxy= https_proxy= no_proxy= dnf install -y rabbitmq-server
Last metadata expiration check: 0:02:27 ago on Tue 23 May 2023 08:17:57 AM UTC.
No match for argument: rabbitmq-server
Error: Unable to find a match: rabbitmq-server
[..]
+ lib/rpc_backend:install_rpc_backend:61 : sudo systemctl --now enable rabbitmq-server
Failed to enable unit: Unit file rabbitmq-server.service does not exist.

It seems that the centos9-rabbitmq repo is not correctly setup:

+ ./stack.sh:main:396 : _install_rdo
+ ./stack.sh:_install_rdo:304 : [[ rhel9 == \r\h\e\l\8 ]]
+ ./stack.sh:_install_rdo:313 : [[ rhel9 == \r\h\e\l\9 ]]
+ ./stack.sh:_install_rdo:314 : sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos9-master/delorean-deps.repo
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 270 100 270 0 0 4090 0 --:--:-- --:--:-- --:--:-- 4153

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (35) error:1C8000E9:Provider routines::ems not enabled
+ ./stack.sh:_install_rdo:316 : sudo dnf -y update
CentOS Stream 9 - BaseOS 24 kB/s | 3.9 kB 00:00
CentOS Stream 9 - AppStream 23 kB/s | 4.4 kB 00:00
CentOS Stream 9 - CRB 27 kB/s | 4.3 kB 00:00
Dependencies resolved.
Nothing to do.
Complete!

On a standard centos 9 stream job, this repo is correctly added:

2023-05-17 05:57:44.026060 | controller | + ./stack.sh:main:396 : _install_rdo
2023-05-17 05:57:44.027801 | controller | + ./stack.sh:_install_rdo:304 : [[ rhel9 == \r\h\e\l\8 ]]
2023-05-17 05:57:44.029474 | controller | + ./stack.sh:_install_rdo:313 : [[ rhel9 == \r\h\e\l\9 ]]
2023-05-17 05:57:44.031359 | controller | + ./stack.sh:_install_rdo:314 : sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos9-master/delorean-deps.repo
2023-05-17 05:57:44.098569 | controller | % Total % Received % Xferd Average Speed Time Time Time Current
2023-05-17 05:57:44.098639 | controller | Dload Upload Total Spent Left Speed
2023-05-17 05:57:44.164652 | controller |
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 270 100 270 0 0 4090 0 --:--:-- --:--:-- --:--:-- 4090
2023-05-17 05:57:44.294620 | controller |
100 988 100 988 0 0 5040 0 --:--:-- --:--:-- --:--:-- 5040
2023-05-17 05:57:44.300746 | controller | + ./stack.sh:_install_rdo:316 : sudo dnf -y update
2023-05-17 05:57:44.939049 | controller | CentOS Stream 9 - BaseOS 51 kB/s | 3.9 kB 00:00
2023-05-17 05:57:45.064183 | controller | CentOS Stream 9 - AppStream 102 kB/s | 4.4 kB 00:00
2023-05-17 05:57:45.237280 | controller | CentOS Stream 9 - CRB 89 kB/s | 4.3 kB 00:00
2023-05-17 05:57:45.701855 | controller | dlrn-master-testing 4.1 MB/s | 1.6 MB 00:00
2023-05-17 05:57:46.680209 | controller | dlrn-master-build-deps 1.4 MB/s | 457 kB 00:00
2023-05-17 05:57:47.483923 | controller | centos9-rabbitmq 226 kB/s | 104 kB 00:00
2023-05-17 05:57:48.039733 | controller | centos9-storage 1.1 MB/s | 488 kB 00:00
2023-05-17 05:57:48.670423 | controller | centos9-opstools 114 kB/s | 42 kB 00:00
2023-05-17 05:57:49.069943 | controller | NFV SIG OpenvSwitch 414 kB/s | 119 kB 00:00

[Gregory Thiemonge (gthiemonge)]

opensearch shows that it is a global issue:

https://opensearch.logs.openstack.org/_dashboards/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Provider%20routines::ems%20not%20enabled%22'),sort:!())

many projects are impacted

[Gregory Thiemonge (gthiemonge)]

Bug for neutron: https://bugs.launchpad.net/neutron/+bug/2020661

[Alan Pevec (apevec)]

so that means curl doesn't work in FIPS mode, it's not an infra issue with trunk.rdoproject.org

[Alan Pevec (apevec)]

maybe this could be a workaround
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening#examples-of-opting-out-of-system-wide-crypto-policies_using-the-system-wide-cryptographic-policies

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Patch: https://review.opendev.org/c/openstack/devstack/+/884247

[Martin Kopec (mkopec)]

This patch that has just got merged is supposed to close this LP:
https://review.opendev.org/c/openstack/devstack/+/884277

[Gregory Thiemonge (gthiemonge)]

Thanks, this specific issue seems fixed but our FIPS jobs are still failing for another reason

https://zuul.openstack.org/builds?job_name=octavia-v2-dsvm-tls-barbican-fips&skip=0

I may open another BZ

[Gregory Thiemonge (gthiemonge)]

Fixed in another repo