[FIPS] Standalone deploy failing with: "Error in GnuTLS initialization: Error while performing self checks"

[Douglas Viroel]

Standalone FIPS jobs are failing since 08-10:
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-standalone-fips

with the following error:

FATAL | Run init bundle puppet on the host for mysql | standalone | error={"changed": false, "cmd": "puppet apply --detailed-exitcodes --summarize --color=false --modulepath '/etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules' --tags 'pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation' -e '[\"Mysql_datadir\", \"Mysql_user\", \"Mysql_database\", \"Mysql_grant\", \"Mysql_plugin\"].each |String $val| { noop_resource($val) }; include ::tripleo::profile::base::pacemaker; include ::tripleo::profile::pacemaker::database::mysql_bundle'\n", "delta": "0:08:17.557003", "end": "2022-08-10 10:09:41.167441", "failed_when_result": true, "msg": "non-zero return code", "rc": 4, "start": "2022-08-10 10:01:23.610438", "stderr": "Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5\n (file: /etc/puppet/hiera.yaml)\nWarning: Undefined variable '::deploy_config_name'; \n (file & line not available)\nError in GnuTLS initialization: Error while performing self checks.\nWarning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.16/deprecated_language.html\n (file & line not available)\nError in GnuTLS initialization: Error while performing self checks.\nDeprecation Warning: This command is deprecated and will be removed. Please use 'pcs property config' instead.\nError: unable to get crm_config\nError in GnuTLS initialization: Error while performing self checks.\nCould not connect to the CIB: Update does not conform to the configured schema\nInit failed, could not perform requested operations\n\nError: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20220810-63608-vrr9bv property set stonith-enabled=false failed: Error: unable to get cib. Too many tries\n
...

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_652/852239/2/check/tripleo-ci-centos-9-standalone-fips/6528969/logs/undercloud/home/zuul/standalone_deploy.log

Comparing last success and first failed jobs we have:
Passing job[1]:
 - gnutls.x86_64 3.7.3-10.el9 @baseos
Failing job[2]:
 - gnutls.x86_64 3.7.6-4.el9 @baseos

[1] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9bc/850458/14/check/tripleo-ci-centos-9-standalone-fips/9bc1b7f/logs/undercloud/var/log/extra/package-list-installed.txt
[2] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_652/852239/2/check/tripleo-ci-centos-9-standalone-fips/6528969/logs/undercloud/var/log/extra/package-list-installed.txt

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/852914

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/852914
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/2d2fbe3ba6ca1edb6b2b31d587ef4cf6054fb0fb
Submitter: "Zuul (22348)"
Branch: master

commit 2d2fbe3ba6ca1edb6b2b31d587ef4cf6054fb0fb
Author: Douglas Viroel <email address hidden>
Date: Thu Aug 11 15:42:07 2022 -0300

    [FIPS] Exclude gnutls 3.7.6 which is failing on fips

    Excludes gnutls 3.7.6 package from baseos repo, until
    fips related issue is solved.

    Related-Bug: #1984237
    Change-Id: Ie5eb9d5fde53916f11d371f1014b1aff966cc32c

[Damien Ciabrini (dciabrin)]

As to why GnuTLS in involved:

The failure happening during ansible task "Run init bundle puppet on the host for mysql" is due to several factors.

1. This tasks essentially runs puppet on the host like the following:

   # puppet apply --debug --verbose --detailed-exitcodes --summarize --tags 'pacemaker::property' -e 'noop_resource("Mysql_datadir"); include ::tripleo::profile::base::pacemaker'

The resource we're noop'ing tells puppet to load puppet-mysql, which - as a side effects - configures LD_LIBRARY_PATH during the puppet run.

2. When puppet-pacemaker runs a subshell to inspect the state of the pacemaker cluster, the environment of the subshell is basically:

   # LD_LIBRARY_PATH=:/usr/lib:/usr/lib64:/opt/rh/rh-mysql56/root/usr/lib:/opt/rh/rh-mysql56/root/usr/lib64:/opt/rh/rh-mysql57/root/usr/lib:/opt/rh/rh-mysql57/root/usr/lib64:/opt/rh/rh-mysql80/root/usr/lib:/opt/rh/rh-mysql80/root/usr/lib64:/opt/rh/rh-mariadb100/root/usr/lib:/opt/rh/rh-mariadb100/root/usr/lib64:/opt/rh/rh-mariadb101/root/usr/lib:/opt/rh/rh-mariadb101/root/usr/lib64:/opt/rh/rh-mariadb102/root/usr/lib:/opt/rh/rh-mariadb102/root/usr/lib64:/opt/rh/rh-mariadb103/root/usr/lib:/opt/rh/rh-mariadb103/root/usr/lib64:/opt/rh/mysql55/root/usr/lib:/opt/rh/mysql55/root/usr/lib64:/opt/rh/mariadb55/root/usr/lib:/opt/rh/mariadb55/root/usr/lib64:/usr/mysql/5.5/lib:/usr/mysql/5.5/lib64:/usr/mysql/5.6/lib:/usr/mysql/5.6/lib64:/usr/mysql/5.7/lib:/usr/mysql/5.7/lib64 crm_node -l
   Error in GnuTLS initialization: Error while performing self checks.
   1 standalone member

3. While the call to crm_node succeeds, there's additional output on stdout, and a later subshell ran by puppet-pacemaker cannot parse the output it expects:

   # LD_LIBRARY_PATH=/usr/lib64:<...puppet-mysql-libs...> cibadmin -Q --scope crm_config
   Error in GnuTLS initialization: Error while performing self checks.
   <crm_config>
     <cluster_property_set id="cib-bootstrap-options">
       <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/>
       <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.1.4-2.el9-dc6eb4362e"/>
       <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
       <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="tripleo_cluster"/>
       <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/>
     </cluster_property_set>
   </crm_config>

4. This bad parsing ultimately makes the puppet run fail.

[Marios Andreou (marios-b)]

chkumar posted https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582/ with this as related-bug

Revert "[FIPS] Exclude gnutls 3.7.6 which is failing on fips"

[chandan kumar (chkumar246)]

gnutls 3.7.6 package is no longer exists and is breaking the content provider
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider
with following error
```
No package gnutls-3.7.3-10.el9 available.", "2022-10-21 03:06:25 | Packages for argument containers-common-1-40.el9 available, but not installed.", "2022-10-21 03:06:25 | Error: No packages marked for downgrade."]}
```

As a proposed fix, revert the pinning of gnutls
https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582 will unblock the gate.

[Marios Andreou (marios-b)]

the gnutls issue also hitting the integration lines

master https://review.rdoproject.org/zuul/build/e2c88a92218c4f1f98b4e03010d13b3f

wallaby https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-9-push-wallaby/eed19c3/job-output.txt

waiting for https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582 to unblock that too

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/ac6d97ab256ce37157b1cd649be962398936966a
Submitter: "Zuul (22348)"
Branch: master

commit ac6d97ab256ce37157b1cd649be962398936966a
Author: chandan kumar <email address hidden>
Date: Fri Sep 9 09:37:55 2022 +0000

    Revert "[FIPS] Exclude gnutls 3.7.6 which is failing on fips"

    This reverts commit 2d2fbe3ba6ca1edb6b2b31d587ef4cf6054fb0fb.

    Reason for revert: this version is no longer available
    Fixed version gnutls-3.7.6-11.el9 is now published:
    https://bugzilla.redhat.com/show_bug.cgi?id=2119822#c7

    Note: It is not a clean revert
    The content provider jobs are failing with following error[1]:
    ```
    Packages for argument containers-common-1-40.el9 available, but not installed.
    ```
    It modifies the downgrade command to downgrade a containers-common
    package only if it is installed.

    [1]. https://zuul.opendev.org/t/openstack/build/75f028ea950349c9a2653f6d2aa2a157/log/logs/undercloud/home/zuul/repo_setup.log#220

    Related-Bug: #1984237

    Change-Id: I888b446518f30b3f63f9f566c4341a0ad289b51d

[Dariusz Smigiel (smigiel-dariusz)]

The fix has been merged.

[Douglas Viroel (dviroel)]

FIPS jobs are still failing even with a newer version of gnuTLS[1][2]:

gnutls.x86_64 3.7.6-12.el9 @baseos
nettle.x86_64 3.8-3.el9 @baseos

[1] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_921/periodic/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-scenario010-standalone-fips/921ef14/logs/undercloud/home/zuul/standalone_deploy.log
[2] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_921/periodic/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-scenario010-standalone-fips/921ef14/logs/undercloud/var/log/extra/package-list-installed.txt

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-common/+/878524

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/878654

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/884544
Committed: https://opendev.org/openstack/tripleo-common/commit/a27e9268aaba5e9301f1653c0cb45eca0d8598ef
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit a27e9268aaba5e9301f1653c0cb45eca0d8598ef
Author: Douglas Viroel <email address hidden>
Date: Tue May 30 14:33:07 2023 +0000

    Always install nettle and gnutls in CentOS-Stream-9

    This patch update previous workaround to always update
    both "nettle" and "gnutls" packages in base container.
    This solution guarantee that we don't have any of these
    packages from ubi repositories, avoiding GNUTLS FIPS
    validation issues. The fix should always install the latest
    package version from CentOS repos and should not fail even
    if some container build needs to downgrade gnutls version.

    This reverts commit 5fc8ba5c8ad49b08c5147fc3ba59563c9f7d27ed:
      "[FIPS] Install nettle-3.8-3.el9 in tcib base container"
    Reason for revert: new ubi release

    Closes-Bug: #1984237

    Depends-On: https://review.opendev.org/c/openstack/tripleo-quickstart/+/884774

    Change-Id: Icb3519879bfb44536e5a8812e39a50ea61d61595

[Douglas Viroel (dviroel)]

https://review.opendev.org/c/openstack/tripleo-common/+/884544 fixes the issue.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/878654
Reason: TrieplO project is retiring now, for details, please see https://review.opendev.org/c/openstack/governance/+/905145 or reach out to OpenStack TC.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-common/+/878524
Reason: TrieplO project is retiring now, for details, please see https://review.opendev.org/c/openstack/governance/+/905145 or reach out to OpenStack TC.