[CentOS] Periodic job "neutron-ovs-tempest-fips" failing since May 18 2023

[Rodolfo Alonso]

The periodic job "neutron-ovs-tempest-fips" is failing since May 18 [1].

The issue seems to be in some missing packages:
* rabbitmq-server: https://zuul.openstack.org/build/0f33bf8c2a1c4854aaadaa48badb8c73/log/job-output.txt#5294
* openstack-network-scripts: https://zuul.openstack.org/build/07183ad3d88f461f9cebadd2bf76cdff/log/job-output.txt#6181

[1]https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-tempest-fips&branch=master&pipeline=periodic&skip=0

[yatin (yatinkarel)]

Actual issue is repos are not getting setup due to an Error, It's impacting other projects too https://bugs.launchpad.net/octavia/+bug/2020434

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

This bug was also reported in Octavia project: https://bugs.launchpad.net/octavia/+bug/2020434

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Patch proposed: https://review.opendev.org/c/openstack/devstack/+/884247

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/884248

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/884279

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

As reported in [1], FIPS 140-3 requires that only EMS KDF is in use for TLS 1.2, after May 2023. The certificate for "trunk.rdoproject.org" does not support EMS but "trunk-centos8.rdoproject.org" does. The content of both files is the same.

[1]https://bugzilla.redhat.com/show_bug.cgi?id=2157951

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

There is a bugzilla [1] created in order to provide EMS support for httpd/mod_ssl/openssl in CentOS7. After that, the certificate for "trunk.rdoproject.org" could support EMS too.

[1]https://bugzilla.redhat.com/show_bug.cgi?id=2209766

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/884248

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/884279
Reason: The devstack patch is fixing the FIPS CI issue.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

The issue has been solved in devstack [1]. Now we are using the CentOS official mirrors, that support EMS. This patch has been tested in the Neutron CI [2][3].

[1]https://review.opendev.org/c/openstack/devstack/+/884277
[2]https://review.opendev.org/c/openstack/neutron/+/884279
[3]https://zuul.opendev.org/t/openstack/build/606ed89810f34b6ba1a35c4c3d597e02/logs

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/884277
Committed: https://opendev.org/openstack/devstack/commit/b2ad00cb66bd38ec6179d3bd1bf41556b966dc8c
Submitter: "Zuul (22348)"
Branch: master

commit b2ad00cb66bd38ec6179d3bd1bf41556b966dc8c
Author: Alfredo Moralejo <email address hidden>
Date: Wed May 24 21:03:28 2023 +0200

    Use RDO official CloudSIG mirrors for C9S deployments

    Instead of using RDO Trunk repo server, CentOS official mirrors provide
    a most reliable infrastructure and supports EMS which is required when
    enabling FIPS in C9S.

    In order to install the rdo-release rpm from repo.fedoraproject.org,
    which does not support EMS, I'm using a workaround to wget, which works
    with non-EMS servers because it uses gnutls instead of openssl, and
    install it locally with rpm.

    This is also consistent to CentOS 8 implementatioin.

    Closes-Bug: #2020661
    Closes-Bug: #2020434
    Change-Id: Icd99f467d47aaafaaf3ee8f2a3c4da08842cb672

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/890120
Committed: https://opendev.org/openstack/devstack/commit/b07c9b8cea0fc893162dae8c18adaaa066e47d14
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit b07c9b8cea0fc893162dae8c18adaaa066e47d14
Author: Alfredo Moralejo <email address hidden>
Date: Wed May 24 21:03:28 2023 +0200

    Use RDO official CloudSIG mirrors for C9S deployments

    Instead of using RDO Trunk repo server, CentOS official mirrors provide
    a most reliable infrastructure and supports EMS which is required when
    enabling FIPS in C9S.

    In order to install the rdo-release rpm from repo.fedoraproject.org,
    which does not support EMS, I'm using a workaround to wget, which works
    with non-EMS servers because it uses gnutls instead of openssl, and
    install it locally with rpm.

    This is also consistent to CentOS 8 implementatioin.

    Closes-Bug: #2020661
    Closes-Bug: #2020434
    Change-Id: Icd99f467d47aaafaaf3ee8f2a3c4da08842cb672
    (cherry picked from commit b2ad00cb66bd38ec6179d3bd1bf41556b966dc8c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/890221
Committed: https://opendev.org/openstack/devstack/commit/e62f4f25f30267fb6f1b23ee598be6760a8b9fba
Submitter: "Zuul (22348)"
Branch: stable/zed

commit e62f4f25f30267fb6f1b23ee598be6760a8b9fba
Author: Alfredo Moralejo <email address hidden>
Date: Wed May 24 21:03:28 2023 +0200

    Use RDO official CloudSIG mirrors for C9S deployments

    Instead of using RDO Trunk repo server, CentOS official mirrors provide
    a most reliable infrastructure and supports EMS which is required when
    enabling FIPS in C9S.

    In order to install the rdo-release rpm from repo.fedoraproject.org,
    which does not support EMS, I'm using a workaround to wget, which works
    with non-EMS servers because it uses gnutls instead of openssl, and
    install it locally with rpm.

    This is also consistent to CentOS 8 implementatioin.

    Closes-Bug: #2020661
    Closes-Bug: #2020434
    Change-Id: Icd99f467d47aaafaaf3ee8f2a3c4da08842cb672
    (cherry picked from commit b2ad00cb66bd38ec6179d3bd1bf41556b966dc8c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/890222
Committed: https://opendev.org/openstack/devstack/commit/4062cc0f85d6e865db2cc20f635a3942d4cc4dfc
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 4062cc0f85d6e865db2cc20f635a3942d4cc4dfc
Author: Alfredo Moralejo <email address hidden>
Date: Wed May 24 21:03:28 2023 +0200

    Use RDO official CloudSIG mirrors for C9S deployments

    Instead of using RDO Trunk repo server, CentOS official mirrors provide
    a most reliable infrastructure and supports EMS which is required when
    enabling FIPS in C9S.

    In order to install the rdo-release rpm from repo.fedoraproject.org,
    which does not support EMS, I'm using a workaround to wget, which works
    with non-EMS servers because it uses gnutls instead of openssl, and
    install it locally with rpm.

    This is also consistent to CentOS 8 implementatioin.

    Closes-Bug: #2020661
    Closes-Bug: #2020434
    Change-Id: Icd99f467d47aaafaaf3ee8f2a3c4da08842cb672
    (cherry picked from commit b2ad00cb66bd38ec6179d3bd1bf41556b966dc8c)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/devstack/+/896215

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/devstack/+/896762

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/896215
Committed: https://opendev.org/openstack/devstack/commit/b050ce825328d895bc3fbb35475bbf7b6ddee81b
Submitter: "Zuul (22348)"
Branch: stable/xena

commit b050ce825328d895bc3fbb35475bbf7b6ddee81b
Author: Alfredo Moralejo <email address hidden>
Date: Wed May 24 21:03:28 2023 +0200

    Use RDO official CloudSIG mirrors for C9S deployments

    Instead of using RDO Trunk repo server, CentOS official mirrors provide
    a most reliable infrastructure and supports EMS which is required when
    enabling FIPS in C9S.

    RDO didn't publish official release rpms for Xena and Wallaby, install
    "rdo-release-yoga" release rpm for Xena. Devstack only uses RDO repository for binary dependencies such as rabbitmq, openvswitch.

    Change 0da88c4af096ab95ccf438960433bb113278181e is squashed in this
    commit.

    Closes-Bug: #2020661
    Closes-Bug: #2020434
    Change-Id: Icd99f467d47aaafaaf3ee8f2a3c4da08842cb672
    (cherry picked from commit b2ad00cb66bd38ec6179d3bd1bf41556b966dc8c)
    (cherry picked from commit 4062cc0f85d6e865db2cc20f635a3942d4cc4dfc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/896762
Committed: https://opendev.org/openstack/devstack/commit/e11905a6470df34637fa71a51e6ed446aab7df50
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit e11905a6470df34637fa71a51e6ed446aab7df50
Author: Alfredo Moralejo <email address hidden>
Date: Wed May 24 21:03:28 2023 +0200

    Use RDO official CloudSIG mirrors for C9S deployments

    Instead of using RDO Trunk repo server, CentOS official mirrors provide
    a most reliable infrastructure and supports EMS which is required when
    enabling FIPS in C9S.

    RDO didn't publish official release rpms for Xena and Wallaby, install
    "rdo-release-yoga" release rpm for Xena. Devstack only uses RDO repository for binary dependencies such as rabbitmq, openvswitch.

    Change 0da88c4af096ab95ccf438960433bb113278181e is squashed in this
    commit.

    Closes-Bug: #2020661
    Closes-Bug: #2020434
    Change-Id: Icd99f467d47aaafaaf3ee8f2a3c4da08842cb672
    (cherry picked from commit b2ad00cb66bd38ec6179d3bd1bf41556b966dc8c)
    (cherry picked from commit 4062cc0f85d6e865db2cc20f635a3942d4cc4dfc)