tripleo-ci-centos-8-standalone-on-multinode-ipa/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master failing while configuring FreeIPA server with RuntimeError: CA configuration failed.

[Sandeep Yadav]

tripleo-ci-centos-8-standalone-on-multinode-ipa is failing in Check/Gate/Periodic while configuring FreeIPA server with RuntimeError: CA configuration failed.

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master/f967069/job-output.txt

~~~
2020-11-01 01:24:56.641717 | primary | TASK [ipa-multinode : configure FreeIPA] ***************************************
2020-11-01 01:24:56.642024 | primary | Sunday 01 November 2020 01:24:56 +0000 (0:00:01.056) 0:01:59.048 *******
2020-11-01 01:27:34.964988 | primary | fatal: [subnode-1]: FAILED! => {
2020-11-01 01:27:34.965301 | primary | "changed": true,
2020-11-01 01:27:34.965323 | primary | "cmd": "ipa-server-install --realm OOO.TEST --ds-password fce95318204114530f31f885c9df588f --admin-password fce95318204114530f31f885c9df588f --hostname ipa.ooo.test --setup-dns --forwarder \"1.1.1.1\" --unattended --ip-address 192.168.100.136\n",
2020-11-01 01:27:34.965340 | primary | "delta": "0:02:37.730238",
2020-11-01 01:27:34.965379 | primary | "end": "2020-11-01 01:27:34.908269",
2020-11-01 01:27:34.965400 | primary | "rc": 1,
2020-11-01 01:27:34.965410 | primary | "start": "2020-11-01 01:24:57.178031"
2020-11-01 01:27:34.965415 | primary | }
2020-11-01 01:27:34.965439 | primary |
2020-11-01 01:27:34.965448 | primary | STDOUT:
2020-11-01 01:27:34.965453 | primary |
2020-11-01 01:27:34.965459 | primary |
.
.
2020-11-01 01:27:34.966288 | primary | Failed to configure CA instance: CalledProcessError(Command ['/usr/sbin/pkispawn', '-s', 'CA', '-f', '/tmp/tmpid00wzyj'] returned non-zero exit status 1: 'Notice: Trust flag u is set automatically if the private key is present.\nERROR: Exception: Server unreachable due to SSL error: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:897)\n File "/usr/lib/python3.6/site-packages/pki/server/pkispawn.py", line 562, in main\n scriptlet.spawn(deployer)\n File "/usr/lib/python3.6/site-packages/pki/server/deployment/scriptlets/configuration.py", line 836, in spawn\n request_timeout=status_request_timeout,\n File "/usr/lib/python3.6/site-packages/pki/server/deployment/pkihelper.py", line 911, in wait_for_startup\n raise Exception(\'Server unreachable due to SSL error: %s\' % reason) from exc\n\n')
2020-11-01 01:27:34.966318 | primary | See the installation logs and the following files/directories for more information:
~~~

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master/21e52d6/logs/192.168.103.47/var/log/ipaserver-install.log.txt.gz
~~~
Installing CA into /var/lib/pki/pki-tomcat.

Installation failed: Server unreachable due to SSL error: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:897)

2020-11-01T17:32:52Z DEBUG stderr=Notice: Trust flag u is set automatically if the private key is present.
ERROR: Exception: Server unreachable due to SSL error: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:897)
  File "/usr/lib/python3.6/site-packages/pki/server/pkispawn.py", line 562, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3.6/site-packages/pki/server/deployment/scriptlets/configuration.py", line 836, in spawn
    request_timeout=status_request_timeout,
  File "/usr/lib/python3.6/site-packages/pki/server/deployment/pkihelper.py", line 911, in wait_for_startup
    raise Exception('Server unreachable due to SSL error: %s' % reason) from exc

2020-11-01T17:32:52Z CRITICAL Failed to configure CA instance: CalledProcessError(Command ['/usr/sbin/pkispawn', '-s', 'CA', '-f', '/tmp/tmp8vs9p8dx'] returned non-zero exit status 1: 'Notice: Trust flag u is set automatically if the private key is present.\nERROR: Exception: Server unreachable due to SSL error: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:897)\n File "/usr/lib/python3.6/site-packages/pki/server/pkispawn.py", line 562, in main\n scriptlet.spawn(deployer)\n File "/usr/lib/python3.6/site-packages/pki/server/deployment/scriptlets/configuration.py", line 836, in spawn\n request_timeout=status_request_timeout,\n File "/usr/lib/python3.6/site-packages/pki/server/deployment/pkihelper.py", line 911, in wait_for_startup\n raise Exception(\'Server unreachable due to SSL error: %s\' % reason) from exc\n\n')
~~~

Another example:

https://5f188933102896d51c08-8bc1c0202523f17b73621207314548bd.ssl.cf2.rackcdn.com/759912/15/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/5220512/job-output.txt

https://5f188933102896d51c08-8bc1c0202523f17b73621207314548bd.ssl.cf2.rackcdn.com/759912/15/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/5220512/logs/10.4.70.43/var/log/ipaserver-install.log

[Damien Ciabrini (dciabrin)]

This seems to be related to a recent RHEL change spotted by bandini and lmiccini [1] that probably deprecates old TLS versions in java. Quoting the bz:

""
java-1.8.0-openjdk-1:1.8.0.272.b10-1.el8_2.x86_64 -> Breaks FreeIPA install
java-1.8.0-openjdk-devel-1:1.8.0.265.b01-0.el8_2.x86_64 -> Works correctly with FreeIPA install

if rpm -q --queryformat '%{version}' java-1.8.0-openjdk |grep "1.8.0.272"; then dnf downgrade -y java-1.8.0-openjdk java-1.8.0-openjdk-headless; fi

The installation of freeipa proceeded normally (java-1.8.0-openjdk-1.8.0.265.b01-0.el8_2.x86_64 is what we downgraded to)
""

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1892216

[Sandeep Yadav (sandeepyadav93)]

Hi,

Readded promotion-blocker flag as earlier it failed to trigger the escalation card

[Marios Andreou (marios-b)]

workaround posted there https://review.opendev.org/#/c/760994/

as discussed in https://bugzilla.redhat.com/show_bug.cgi?id=1892216 (see comment #1 above) the workaround can be removed once that bug is resolved.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/760994
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=8ad3c828572d627aa16f4c1c5bb94ccf07caac88
Submitter: Zuul
Branch: master

commit 8ad3c828572d627aa16f4c1c5bb94ccf07caac88
Author: Ade Lee <email address hidden>
Date: Mon Nov 2 14:29:49 2020 -0500

    Workaround for jdk zstream breaking freeipa install

    See https://bugzilla.redhat.com/show_bug.cgi?id=1892216.
    We need to downgrade openjdk in order to get a successful install
    until freeipa is fixed or some other rhel workaround is implemented.

    Depends-On: https://review.opendev.org/#/c/761402/
    Closes-Bug: #1902478
    Change-Id: If4598ac495f00fbfd58d1ad9910900d65209bab1

[Sandeep Yadav (sandeepyadav93)]

Hello,

periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master passed the earlier issue after https://review.opendev.org/#/c/760994/ merged but featureset039 is still hitting the same issue

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/2dcea9f/job-output.txt

~~~
2020-11-09 02:08:32.850116 | primary | TASK [freeipa-setup : Deploy FreeIPA] ******************************************
2020-11-09 02:08:32.850464 | primary | Monday 09 November 2020 02:08:32 +0000 (0:00:02.977) 0:15:42.921 *******
2020-11-09 02:12:10.066509 | primary | fatal: [supplemental]: FAILED! => {
2020-11-09 02:12:10.067114 | primary | "changed": true,
2020-11-09 02:12:10.067152 | primary | "cmd": "~centos/deploy_freeipa.sh &> ~centos/deploy_freeipa.log",
2020-11-09 02:12:10.067226 | primary | "delta": "0:03:36.620599",
2020-11-09 02:12:10.067263 | primary | "end": "2020-11-09 02:12:10.013007",
2020-11-09 02:12:10.067298 | primary | "rc": 1,
2020-11-09 02:12:10.067317 | primary | "start": "2020-11-09 02:08:33.392408"
2020-11-09 02:12:10.067329 | primary | }
~~~

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/2dcea9f/logs/supplemental/home/centos/deploy_freeipa.log.txt.gz

~~~
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
Failed to configure CA instance: CalledProcessError(Command ['/usr/sbin/pkispawn', '-s', 'CA', '-f', '/tmp/tmp67xjehu3'] returned non-zero exit status 1: 'Notice: Trust flag u is set automatically if the private key is present.\nERROR: Exception: Server unreachable due to SSL error: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:897)\n File "/usr/lib/python3.6/site-packages/pki/server/pkispawn.py", line 562, in main\n scriptlet.spawn(deployer)\n File "/usr/lib/python3.6/site-packages/pki/server/deployment/scriptlets/configuration.py", line 836, in spawn\n request_timeout=status_request_timeout,\n File "/usr/lib/python3.6/site-packages/pki/server/deployment/pkihelper.py", line 911, in wait_for_startup\n raise Exception(\'Server unreachable due to SSL error: %s\' % reason) from exc\n\n')
See the installation logs and the following files/directories for more information:
  /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
CA configuration failed.
The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
~~~

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/2dcea9f/logs/supplemental/var/log/extra/package-list-installed.txt.gz
~~~
java-1.8.0-openjdk.x86_64 1:1.8.0.272.b10-1.el8_2 @quickstart-centos-appstreams
java-1.8.0-openjdk-devel.x86_64 1:1.8.0.272.b10-1.el8_2 @quickstart-centos-appstreams
java-1.8.0-openjdk-he...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/761863

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/761863
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=9fef1c0e42b5ee755196e08705f7368c1f2b5e1b
Submitter: Zuul
Branch: master

commit 9fef1c0e42b5ee755196e08705f7368c1f2b5e1b
Author: Sandeep Yadav <email address hidden>
Date: Mon Nov 9 13:01:33 2020 +0530

    Workaround for jdk zstream breaking freeipa instal

    See https://bugzilla.redhat.com/show_bug.cgi?id=1892216.
    We need to downgrade openjdk in order to get a successful install
    until freeipa is fixed or some other rhel workaround is implemented.

    With this patch we are adding workaround in freeipa-setup role, we
    have added similiar workaround in ipa-multinode role here[1]

    [1] https://review.opendev.org/#/c/760994/
    Closes-Bug: #1902478

    Change-Id: I18e3dae31898b4baea92e0bb39d48553c8319921