tomcat more or less broken -- java compat issues

[Juan Tobon]

[Impact]

The issue occurs while installing IPA server. More specifically whist configuring pki-tomcatd. The following error is produced.

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
      [1/28]: configuring certificate server instance
    ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance: CalledProcessError(Command ['/usr/sbin/pkispawn', '-s', 'CA', '-f', '/tmp/tmpEHq9Ex'] returned non-zero exit status 1: u"pkispawn : ERROR ....... subprocess.CalledProcessError: Command '['sysctl', 'crypto.fips_enabled', '-bn']' returned non-zero exit status 255!\npkispawn : ERROR ........... server did not start after 60s\npkispawn : ERROR ....... server failed to restart\n")
    ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the following files/directories for more information:
    ipaserver.install.dogtaginstance: CRITICAL /var/log/pki/pki-tomcat
      [error] RuntimeError: CA configuration failed.
    ipapython.admintool: ERROR CA configuration failed.
    ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

The cause for this is that tomcat8 is built with JDK9 and is not compatible with instances that have to use JRE8 for other reasons.

[Test Case]

Install freeipa-server, run ipa-server-install.

[Regression Potential]

The fix is a fairly big patch for tomcat8 to modify the code so that it runs with JRE8. It passes the upstream test suite though, when run with JRE8 though tomcat itself was built with the default JDK.

[Other info]

Patch will be sent upstream too.

[Juan Tobon (juantobon78)]

I would also like to ask why freeipa version in this Ubuntu release when from the intended 4.6 to what appears to be 4.7?

[Timo Aaltonen (tjaalton)]

curl/ssl not working is probably because the setup didn't get far enough, check /var/log/pki/pki-tomcat/* for errors

Are you able to reproduce the setup error each time? The setup is racy on slower machines where the tomcat startup takes "long", some later steps can fail because of that but I haven't seen it this early.

The upstream issues seem fixed already, and we have those versions. The error was different there anyway.

[Timo Aaltonen (tjaalton)]

I was able to reproduce this, and the cause is tomcat8 built against newer JDK now with 8.5.30-1

[Timo Aaltonen (tjaalton)]

Bumping priority, this breaks more than just freeipa/dogtag.

I've uploaded a new version to bionic a week ago which adds support for JRE8, but the patch is big and not yet upstream.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Timo Aaltonen (tjaalton)]

I've uploaded a new tomcat8 (8.5.30-1ubuntu1.2) to ppa:freeipa/ppa

https://launchpad.net/~freeipa/+archive/ubuntu/ppa

-1ubuntu1.1 has an incomplete patch and doesn't work properly

[Kees Bakker (keestux)]

To confirm, with the PPA the installation continues, and "Configuring certificate server" succeeds.

However, now "Configuring the web interface" fails with

  [12/21]: setting up ssl
  [error] RuntimeError: Certificate issuance failed (CA_REJECTED)
ipapython.admintool: ERROR Certificate issuance failed (CA_REJECTED)
ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

and in the log there is this:

2018-05-04T07:48:09Z DEBUG [12/21]: setting up ssl
2018-05-04T07:48:13Z DEBUG certmonger request is in state dbus.String(u'GENERATING_KEY_PAIR', variant_level=1)
2018-05-04T07:48:18Z DEBUG certmonger request is in state dbus.String(u'CA_REJECTED', variant_level=1)
2018-05-04T07:48:22Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 555, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 541, in run_step
    method()
  File "/usr/lib/python2.7/dist-packages/ipaserver/install/httpinstance.py", line 376, in __setup_ssl
    passwd_fname=key_passwd_file
  File "/usr/lib/python2.7/dist-packages/ipalib/install/certmonger.py", line 320, in request_and_wait_for_cert
    raise RuntimeError("Certificate issuance failed ({})".format(state))
RuntimeError: Certificate issuance failed (CA_REJECTED)

[Timo Aaltonen (tjaalton)]

file a separate bug, I'm not able to reproduce that

[Jared Szechy (szechyjs)]

dogtag-pki server now runs on bionic using 8.5.30-1ubuntu1.2 from the ppa.

[gianluca (amato)]

ipa-server-install still fails for me during step "[24/28]: migrating certificate profiles to LDAP". It gives me the following error:

NetworkError: cannot connect to 'https://ipa.labeconomnia.unich.it:8443/ca/rest/account/login': [Errno 111] Connection refused

The problem is that, when this error happens, there is no process listening on port 8843 (checked with netstat -tnlp). During previous steps, a java process (Tomcat?) is listening on port 8843, but it periodically goes down and up. Some of these restarts seems triggered by ipa-server-install, but other seems gratuitous.

[Timo Aaltonen (tjaalton)]

the restarts are caused by certmonger requests.. I've added a (very gross) 'sleep 80' to that stage which at least made it pass reliably on my qemu host, but looks like that's not enough. I'll ask upstream why it creates so many requests these days..

[gianluca (amato)]

Right... it was a race condition. Also, increasing the number of CPU and amount of memory in my virtual machine solved the problem.

[gianluca (amato)]

Now I have another problem. ipa-server-install stops at step "[19/21]: starting httpd" of HTTP configuration. From my investigation, it seems that the problem is that the SSL private key in /var/lib/ipa/private/httpd.key has a passphrase, saved in /var/lib/ipa/<host>-443-RSA. The passphrase is correct (I checked with openssl), but Apache does not find it. These are the messages I get in /var/log/apache2/error.log:

[Sat May 05 19:02:57.836869 2018] [mpm_event:notice] [pid 967:tid 140026405403584] AH00491: caught SIGTERM, shutting down
[Sat May 05 19:03:10.609244 2018] [ssl:emerg] [pid 6154:tid 140498019421120] AH02580: Init: Pass phrase incorrect for key ipa.labeconomia.unich.it:443:0
[Sat May 05 19:03:10.609443 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag
[Sat May 05 19:03:10.609465 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error
[Sat May 05 19:03:10.609481 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag
[Sat May 05 19:03:10.609498 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error (Type=RSA)
[Sat May 05 19:03:10.609514 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:04093004:rsa routines:old_rsa_priv_decode:RSA lib
[Sat May 05 19:03:10.609530 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag
[Sat May 05 19:03:10.609546 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error (Type=PKCS8_PRIV_KEY_INFO)
[Sat May 05 19:03:10.609564 2018] [ssl:emerg] [pid 6154:tid 140498019421120] AH02311: Fatal error initialising mod_ssl, exiting. See /var/log/apache2/error.log for more information
[Sat May 05 19:03:10.609576 2018] [ssl:emerg] [pid 6154:tid 140498019421120] AH02564: Failed to configure encrypted (?) private key ipa.labeconomia.unich.it:443:0, check /var/lib/ipa/private/httpd.key

[Timo Aaltonen (tjaalton)]

file a new bug..

[Launchpad Janitor (janitor)]

This bug was fixed in the package tomcat8 - 8.5.30-1ubuntu2

---------------
tomcat8 (8.5.30-1ubuntu2) cosmic; urgency=medium

  * support-jre8.diff: Fix running tomcat with JRE8. (LP: #1765616)

 -- Timo Aaltonen <email address hidden> Tue, 24 Apr 2018 23:47:45 +0300

[Thomas (lostexception)]

Sorry if I'm getting this completely wrong, but the fix seems to be for cosmic only. Does this mean tomcat8 will remain broken on bionic (which is an LTS)?

[Timo Aaltonen (tjaalton)]

no, a task for bionic is open and a version still waiting in proposed, it just needs to be fixed in the devel series first

[Timo Aaltonen (tjaalton)]

..waiting on the queue, not in proposed yet

[Thomas (lostexception)]

Timo, thanks a lot for clarification. Maybe you should change the subject of this bug to "Tomcat mostly broken on bionic" to get some more attention ;)

[Łukasz Zemczak (sil2100)]

Hello Juan, or anyone else affected,

Accepted tomcat8 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tomcat8/8.5.30-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Timo Aaltonen (tjaalton)]

note that the freeipa ppa had a newer package version, so enabling -proposed isn't enough if you installed it from the ppa.. need to manually install the version from proposed like this:

apt install libtomcat8-java=8.5.30-1ubuntu1.1

etc

[Kees Bakker (keestux)]

Didn't you mean to say?

apt install libtomcat8-java=8.5.30-1ubuntu2

[Timo Aaltonen (tjaalton)]

no, the ppa has -1ubuntu2, bionic-proposed has 1.1

[Jared Szechy (szechyjs)]

I just installed 8.5.30-1ubuntu1.1 and it resolved the issue.

[Sebastian (slovdahl)]

I can also confirm that 8.5.30-1ubuntu1.1 now works with openjdk-8 in bionic.

[Thomas (lostexception)]

I can also confirm tomcat8 now working on bionic using openjdk-8-jre-headless (disclaimer: for my use case).

[Timo Aaltonen (tjaalton)]

thanks a lot for testing and feedback!

[Luca Santarelli (luca-santarelli)]

Issue seems to be back with 8.5.30-1ubuntu1.4

It doesn't seem to affect startup, but happens in some situations, e.g.: direct output of files through response stream.
Issue is in file tomcat-coyote.jar, as replacing Ubuntu's file with upstream's tomcat-coyote.jar makes the issue disappear.

Upstream's and Ubuntu's files have indeed different md5sums.

gem-lx1-sv@gem-lx1-sv:/usr/share/java$ md5sum tomcat8-coyote-8.5.30-apache.jar tomcat8-coyote-8.5.30.jar
993e7d3920e00f39b7287fa5f5177a33 tomcat8-coyote-8.5.30-apache.jar
91de49bd30f68be4cbf64e217e98fbc8 tomcat8-coyote-8.5.30.jar

gem-lx1-sv@gem-lx1-sv:/usr/share/java$ ls -lha tomcat8-coyote*
-rw-r--r-- 1 root root 782K Nov 8 15:46 tomcat8-coyote-8.5.30-apache.jar
-rw-r--r-- 1 root root 782K Aug 13 22:23 tomcat8-coyote-8.5.30.jar
lrwxrwxrwx 1 root root 25 Aug 13 22:23 tomcat8-coyote.jar -> tomcat8-coyote-8.5.30-apache.jar

Stack trace for the bug:
[08-Nov-2018 13:25:26.651 SEVERE [http-nio-8080-exec-1] org.apache.coyote.http11.Http11Processor.service Error processing request
 java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
        at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:111)
        at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:226)
        at org.apache.coyote.Response.doWrite(Response.java:541)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)
        at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:815)
        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:310)
        at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:263)
        at org.apache.catalina.connector.Response.finishResponse(Response.java:484)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1463)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

I kindly request to either not recompile upstream's binary jars, or set a java8 JDK as default compiler for 18.04 or until java11 is stable on Ubuntu.

[Luca Santarelli (luca-santarelli)]

Sorry, forgot to add:
gem-lx1-sv@gem-lx1-sv:/usr/share/java$ dpkg -l libtomcat8-java
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==================================================-==============================-==============================-=========================================================================================================
ii libtomcat8-java 8.5.30-1ubuntu1.4 all Apache Tomcat 8 - Servlet and JSP engine -- core libraries

[Brian Murray (brian-murray)]

Thank you for taking the time to verify this stable release fix. We have noticed that you have used the verification-done tag for marking the bug as verified and would like to point out that due to a recent change in SRU bug verification policy fixes now have to be marked with per-release tags (i.e. verification-done-$RELEASE). Please remove the verification-done tag and add one for the release you have tested the package in. Thank you!

https://wiki.ubuntu.com/StableReleaseUpdates#Verification

[Łukasz Zemczak (sil2100)]

Hello Juan, or anyone else affected,

Accepted tomcat8 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tomcat8/8.5.39-1ubuntu1~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package tomcat8 - 8.5.39-1ubuntu1~18.04.1

---------------
tomcat8 (8.5.39-1ubuntu1~18.04.1) bionic; urgency=medium

  [ Matthias Klose ]
  * Backport for OpenJDK 11. LP: #1817567.
    /usr/share/doc/tomcat8/NEWS.gz. LP: #1819721.

  [ Tiago Stürmer Daitx ]
  * debian/tomcat8.service: removed, use the init.d script instead.
    LP: #1819721.
  * debian/tomcat8.init, debian/logging.properties: revert back to the
    conffiles from the previous version; this allows unattended-upgrades
    to update tomcat8 even when local changes are present.
  * debian/series: no longer apply 0023-disable-shutdown-by-socket.patch
    so server.xml conffile is unmodified from previous version.

tomcat8 (8.5.39-1ubuntu1) disco; urgency=medium

  * Merge with Debian; remaining changes:
    - d/control: Break/replace tomcat8.0 binaries.

tomcat8 (8.5.39-1) experimental; urgency=medium

  * Team upload.
  * New upstream release
    - Refreshed the patches
  * Track and download the new releases from GitHub

tomcat8 (8.5.38-2ubuntu1) disco; urgency=medium

  * Merge with Debian; remaining changes:
    - d/control: Break/replace tomcat8.0 binaries.

tomcat8 (8.5.38-2) unstable; urgency=high

  * Team upload.
  * Apply upstream patch to unbreak the startup script (Closes: #922863)

tomcat8 (8.5.38-1ubuntu1) disco; urgency=medium

  * Merge with Debian unstable (LP: #1815601). Remaining changes:
    - d/control: Break/replace tomcat8.0 binaries. (LP: 1717998)
  Dropped Changes:
    - support-jre8.diff.

tomcat8 (8.5.38-1) unstable; urgency=medium

  * Team upload.
  * New upstream release
    - Refreshed the patches

tomcat8 (8.5.37-2) unstable; urgency=medium

  * Team upload.
  * No longer build the JavaEE API packages
  * Standards-Version updated to 4.3.0

tomcat8 (8.5.37-1) unstable; urgency=medium

  * Team upload.
  * New upstream release
    - Refreshed the patches

tomcat8 (8.5.35-3) unstable; urgency=medium

  * Team upload.
  * Split libservlet3.1-java into separate JavaEE API packages
    (libjsp-api-java, libel-api-java and libwebsocket-api-java)
  * Updated the version required for libtcnative-1 (>= 1.2.18)
  * Install the Russian translation added in Tomcat 8.5.33

tomcat8 (8.5.35-2) unstable; urgency=medium

  * Team upload.
  * Fixed the build failure with Easymock 4 (Closes: #913402)

tomcat8 (8.5.35-1) unstable; urgency=medium

  * Team upload.

  [ Thomas Opfer ]
  * Removed old version requirement for package ant-optional that is not
    required any more.

  [ Emmanuel Bourg ]
  * New upstream release
    - Refreshed the patches

tomcat8 (8.5.34-1ubuntu1) cosmic; urgency=medium

  * Merge from Debian unstable. Remaining changes:
    - control: Break/replace tomcat8.0 binaries. (LP: #1717998)
    - support-jre8.diff.

tomcat8 (8.5.34-1) unstable; urgency=medium

  * Team upload.
  * New upstream release
    - Refreshed the patches

tomcat8 (8.5.33-1) unstable; urgency=medium

  * Team upload.
  * New upstream version 8.5.33.
    - Tomcat compiles to Java 7 bytecode and passes release=7 to javac now.
      This ensures backwards compatibility with older JREs. (Closes: #906447)
  * Declare compliance with Debian Policy 4.2....