jss-5.2.0-0.2.beta1 breaks freeipa setup (centos9)

[Luca Miccini]

Freeipa setup fails with:

    INFO: Creating new security domain
    INFO: Using CA at https://freeipa-0.bgp.ftw:443
    INFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
    INFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg
    INFO: Removing existing database
    DEBUG: Command: /usr/sbin/runuser -u pkiuser -- /usr/lib/jvm/jre-11-openjdk/bin/java -classpath /usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/tomcat-servlet-api.jar:/usr/share/pki/ca/webapps/ca/WEB-INF/lib/*:/var/lib/pki/pki-tomcat/common/lib/*:/usr/share/pki/lib/* -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/etc/pki/pki-tomcat/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dcom.redhat.fips=false org.dogtagpki.server.cli.PKIServerCLI ca-db-remove --force --debug
    InvocationTargetException: null
    CalledProcessError: Command '['/usr/sbin/runuser', '-u', 'pkiuser', '--', '/usr/lib/jvm/jre-11-openjdk/bin/java', '-classpath', '/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/tomcat-servlet-api.jar:/usr/share/pki/ca/webapps/ca/WEB-INF/lib/*:/var/lib/pki/pki-tomcat/common/lib/*:/usr/share/pki/lib/*', '-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory', '-Dcatalina.base=/var/lib/pki/pki-tomcat', '-Dcatalina.home=/usr/share/tomcat', '-Djava.endorsed.dirs=', '-Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp', '-Djava.util.logging.config.file=/etc/pki/pki-tomcat/logging.properties', '-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager', '-Dcom.redhat.fips=false', 'org.dogtagpki.server.cli.PKIServerCLI', 'ca-db-remove', '--force', '--debug']' returned non-zero exit status 255.
      File "/usr/lib/python3.9/site-packages/pki/server/pkispawn.py", line 585, in main
        scriptlet.spawn(deployer)
      File "/usr/lib/python3.9/site-packages/pki/server/deployment/scriptlets/configuration.py", line 594, in spawn
        subsystem.remove_database(force=True)
      File "/usr/lib/python3.9/site-packages/pki/server/subsystem.py", line 1028, in remove_database
        self.run(cmd, as_current_user=as_current_user)
      File "/usr/lib/python3.9/site-packages/pki/server/subsystem.py", line 1650, in run
        return subprocess.run(
      File "/usr/lib64/python3.9/subprocess.py", line 528, in run
        raise CalledProcessError(retcode, process.args,

    2022-04-20T10:30:27Z CRITICAL Failed to configure CA instance
    2022-04-20T10:30:27Z CRITICAL See the installation logs and the following files/directories for more information:
    2022-04-20T10:30:27Z CRITICAL /var/log/pki/pki-tomcat
    2022-04-20T10:30:27Z DEBUG Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 635, in start_creation
        run_step(full_msg, method)
      File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 621, in run_step
        method()
      File "/usr/lib/python3.9/site-packages/ipaserver/install/cainstance.py", line 626, in __spawn_instance
        DogtagInstance.spawn_instance(
      File "/usr/lib/python3.9/site-packages/ipaserver/install/dogtaginstance.py", line 227, in spawn_instance
        self.handle_setup_error(e)
      File "/usr/lib/python3.9/site-packages/ipaserver/install/dogtaginstance.py", line 594, in handle_setup_error
        raise RuntimeError(
    RuntimeError: CA configuration failed.

    2022-04-20T10:30:27Z DEBUG [error] RuntimeError: CA configuration failed.

this with jss-5.2.0-0.2.beta1.el9.x86_6 .

Only workaround I've found is downgrading to jss-5.0.3-1.el9.x86_64 for now.

[chandan kumar (chkumar246)]

Thank you for opening the bug. We are seeing similar issue in FS039

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/a9822ed/logs/supplemental/var/log/ipaserver-install.log.txt.gz

```
DEBUG: Command: /usr/sbin/runuser -u pkiuser -- /usr/lib/jvm/jre-11-openjdk/bin/java -classpath /usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/tomcat-servlet-api.jar:/usr/share/pki/ca/webapps/ca/WEB-INF/lib/*:/var/lib/pki/pki-tomcat/common/lib/*:/usr/share/pki/lib/* -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/etc/pki/pki-tomcat/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dcom.redhat.fips=false org.dogtagpki.server.cli.PKIServerCLI ca-db-remove --force --debug
InvocationTargetException: null
CalledProcessError: Command '['/usr/sbin/runuser', '-u', 'pkiuser', '--', '/usr/lib/jvm/jre-11-openjdk/bin/java', '-classpath', '/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/tomcat-servlet-api.jar:/usr/share/pki/ca/webapps/ca/WEB-INF/lib/*:/var/lib/pki/pki-tomcat/common/lib/*:/usr/share/pki/lib/*', '-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory', '-Dcatalina.base=/var/lib/pki/pki-tomcat', '-Dcatalina.home=/usr/share/tomcat', '-Djava.endorsed.dirs=', '-Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp', '-Djava.util.logging.config.file=/etc/pki/pki-tomcat/logging.properties', '-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager', '-Dcom.redhat.fips=false', 'org.dogtagpki.server.cli.PKIServerCLI', 'ca-db-remove', '--force', '--debug']' returned non-zero exit status 255.
  File "/usr/lib/python3.9/site-packages/pki/server/pkispawn.py", line 585, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3.9/site-packages/pki/server/deployment/scriptlets/configuration.py", line 594, in spawn
    subsystem.remove_database(force=True)
  File "/usr/lib/python3.9/site-packages/pki/server/subsystem.py", line 1028, in remove_database
    self.run(cmd, as_current_user=as_current_user)
  File "/usr/lib/python3.9/site-packages/pki/server/subsystem.py", line 1650, in run
    return subprocess.run(
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,

2022-04-20T09:55:16Z CRITICAL Failed to configure CA instance
2022-04-20T09:55:16Z CRITICAL See the installation logs and the following files/directories for more information:
2022-04-20T09:55:16Z CRITICAL /var/log/pki/pki-tomcat
2022-04-20T09:55:16Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 635, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 621, in run_step
    method()
  File "/usr/lib/python3.9/site-packages/ipaserver/install/cainstance.py", line 626, in __spawn_instance
    DogtagInstance...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/c51bdd60189af2ea2630a62c64f4269fc83dec76
Submitter: "Zuul (22348)"
Branch: master

commit c51bdd60189af2ea2630a62c64f4269fc83dec76
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Fri Apr 22 13:06:18 2022 +0530

    Install jss-5.0.3 in freeipa installation

    Currently ss-5.2.0-0.2 is causing issues while deploying
    freeipa.

    Installing jss-5.0.3 fixes the issue.

    Related-Bug: #1969613

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: I81195d6b93162619e367a570ab148ff324a98a5d

[Damien Ciabrini (dciabrin)]

We might need to tweak the fix for wallaby, I'm hitting this error on one of my reviews [1] as of yesterday:

2022-04-26 09:50:22.299317 | primary | TASK [ipa-multinode : Temporarily downgrade jss] *******************************
2022-04-26 09:50:22.299493 | primary | Tuesday 26 April 2022 09:50:22 +0000 (0:01:14.602) 0:03:36.925 *********
2022-04-26 09:50:24.677861 | primary | fatal: [subnode-1]: FAILED! => {"changed": true, "cmd": "dnf downgrade jss-5.0.3-1.el9 -y;\n", "delta": "0:00:00.748072", "end": "2022-04-26 09:50:24.430022", "msg": "non-zero return code", "rc": 1, "start": "2022-04-26 09:50:23.681950", "stderr": "Error: \n Problem: problem with installed package pki-java-11.2.0-0.2.beta1.el9.noarch\n - package pki-java-11.2.0-0.2.beta1.el9.noarch requires jss >= 5.2.0, but none of the providers can be installed\n - cannot install both jss-5.0.3-1.el9.x86_64 and jss-5.2.0-0.2.beta1.el9.x86_64\n - cannot install both jss-5.2.0-0.2.beta1.el9.x86_64 and jss-5.0.3-1.el9.x86_64\n - conflicting requests", "stderr_lines": ["Error: ", " Problem: problem with installed package pki-java-11.2.0-0.2.beta1.el9.noarch", " - package pki-java-11.2.0-0.2.beta1.el9.noarch requires jss >= 5.2.0, but none of the providers can be installed", " - cannot install both jss-5.0.3-1.el9.x86_64 and jss-5.2.0-0.2.beta1.el9.x86_64", " - cannot install both jss-5.2.0-0.2.beta1.el9.x86_64 and jss-5.0.3-1.el9.x86_64", " - conflicting requests"], "stdout": "Last metadata expiration check: 0:01:20 ago on Tue 26 Apr 2022 09:49:04 AM UTC.\n(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)", "stdout_lines": ["Last metadata expiration check: 0:01:20 ago on Tue 26 Apr 2022 09:49:04 AM UTC.", "(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)"]}
2022-04-26 09:50:24.679488 | primary |
2022-04-26 09:50:24.679523 | primary | PLAY RECAP *********************************************************************
2022-04-26 09:50:24.679564 | primary | subnode-1 : ok=25 changed=5 unreachable=0 failed=1 skipped=19 rescued=0 ignored=0
2022-04-26 09:50:24.679702 | primary | undercloud : ok=10 changed=4 unreachable=0 failed=0 skipped=18 rescued=0 ignored=0
[1] https://review.opendev.org/c/openstack/tripleo-heat-templates/+/831570

[Alan Pevec (apevec)]

Downgrade was reverted https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839333/ there was followup https://bugs.launchpad.net/tripleo/+bug/1970406

But unclear what actually fixed it, bug title is not valid since jss-5.2.0-0.2.beta1 works now with FreeIPA ?

[Luca Miccini (lmiccini2)]

afaik this issue has been fixed in centos by rebasing java/pki libs, so we can probably close this bug.

[Bhagyashri Shewale (bhagyashri-shewale)]

We had old version of java and jss was new , So to fix that issue we downgraded the jss version here https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996 and after downgrade , java got updated and we again started facing the issue.So reverted the jss downgrade change here https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839333 as we have new version of java.
Note: Patch merged closing the bug