tomcat more or less broken -- java compat issues

Bug #1765616 reported by Juan Tobon on 2018-04-20
90
This bug affects 16 people
Affects Status Importance Assigned to Milestone
tomcat8 (Debian)
Fix Released
Unknown
tomcat8 (Ubuntu)
Critical
Timo Aaltonen
Bionic
Critical
Unassigned

Bug Description

[Impact]

The issue occurs while installing IPA server. More specifically whist configuring pki-tomcatd. The following error is produced.

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
      [1/28]: configuring certificate server instance
    ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance: CalledProcessError(Command ['/usr/sbin/pkispawn', '-s', 'CA', '-f', '/tmp/tmpEHq9Ex'] returned non-zero exit status 1: u"pkispawn : ERROR ....... subprocess.CalledProcessError: Command '['sysctl', 'crypto.fips_enabled', '-bn']' returned non-zero exit status 255!\npkispawn : ERROR ........... server did not start after 60s\npkispawn : ERROR ....... server failed to restart\n")
    ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the following files/directories for more information:
    ipaserver.install.dogtaginstance: CRITICAL /var/log/pki/pki-tomcat
      [error] RuntimeError: CA configuration failed.
    ipapython.admintool: ERROR CA configuration failed.
    ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

The cause for this is that tomcat8 is built with JDK9 and is not compatible with instances that have to use JRE8 for other reasons.

[Test Case]

Install freeipa-server, run ipa-server-install.

[Regression Potential]

The fix is a fairly big patch for tomcat8 to modify the code so that it runs with JRE8. It passes the upstream test suite though, when run with JRE8 though tomcat itself was built with the default JDK.

[Other info]

Patch will be sent upstream too.

Juan Tobon (juantobon78) wrote :

I would also like to ask why freeipa version in this Ubuntu release when from the intended 4.6 to what appears to be 4.7?

Timo Aaltonen (tjaalton) wrote :

curl/ssl not working is probably because the setup didn't get far enough, check /var/log/pki/pki-tomcat/* for errors

Are you able to reproduce the setup error each time? The setup is racy on slower machines where the tomcat startup takes "long", some later steps can fail because of that but I haven't seen it this early.

The upstream issues seem fixed already, and we have those versions. The error was different there anyway.

Changed in freeipa (Ubuntu):
status: New → Incomplete
Timo Aaltonen (tjaalton) wrote :

I was able to reproduce this, and the cause is tomcat8 built against newer JDK now with 8.5.30-1

Timo Aaltonen (tjaalton) on 2018-04-27
Changed in freeipa (Ubuntu):
status: Incomplete → Invalid
Timo Aaltonen (tjaalton) wrote :

Bumping priority, this breaks more than just freeipa/dogtag.

I've uploaded a new version to bionic a week ago which adds support for JRE8, but the patch is big and not yet upstream.

Changed in tomcat8 (Ubuntu):
assignee: nobody → Timo Aaltonen (tjaalton)
importance: Undecided → Critical
status: New → In Progress
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in freeipa (Ubuntu Bionic):
status: New → Confirmed
Changed in tomcat8 (Ubuntu Bionic):
status: New → Confirmed
Timo Aaltonen (tjaalton) on 2018-05-03
Changed in freeipa (Ubuntu Bionic):
status: Confirmed → Invalid
Timo Aaltonen (tjaalton) on 2018-05-03
description: updated
Timo Aaltonen (tjaalton) wrote :

I've uploaded a new tomcat8 (8.5.30-1ubuntu1.2) to ppa:freeipa/ppa

https://launchpad.net/~freeipa/+archive/ubuntu/ppa

-1ubuntu1.1 has an incomplete patch and doesn't work properly

Changed in tomcat8 (Ubuntu Bionic):
importance: Undecided → Critical
Changed in tomcat8 (Debian):
status: Unknown → New
keestux (kees-bakker-xs4all) wrote :

To confirm, with the PPA the installation continues, and "Configuring certificate server" succeeds.

However, now "Configuring the web interface" fails with

  [12/21]: setting up ssl
  [error] RuntimeError: Certificate issuance failed (CA_REJECTED)
ipapython.admintool: ERROR Certificate issuance failed (CA_REJECTED)
ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

and in the log there is this:

2018-05-04T07:48:09Z DEBUG [12/21]: setting up ssl
2018-05-04T07:48:13Z DEBUG certmonger request is in state dbus.String(u'GENERATING_KEY_PAIR', variant_level=1)
2018-05-04T07:48:18Z DEBUG certmonger request is in state dbus.String(u'CA_REJECTED', variant_level=1)
2018-05-04T07:48:22Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 555, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 541, in run_step
    method()
  File "/usr/lib/python2.7/dist-packages/ipaserver/install/httpinstance.py", line 376, in __setup_ssl
    passwd_fname=key_passwd_file
  File "/usr/lib/python2.7/dist-packages/ipalib/install/certmonger.py", line 320, in request_and_wait_for_cert
    raise RuntimeError("Certificate issuance failed ({})".format(state))
RuntimeError: Certificate issuance failed (CA_REJECTED)

Timo Aaltonen (tjaalton) wrote :

file a separate bug, I'm not able to reproduce that

Jared Szechy (szechyjs) wrote :

dogtag-pki server now runs on bionic using 8.5.30-1ubuntu1.2 from the ppa.

gianluca (amato) wrote :

ipa-server-install still fails for me during step "[24/28]: migrating certificate profiles to LDAP". It gives me the following error:

NetworkError: cannot connect to 'https://ipa.labeconomnia.unich.it:8443/ca/rest/account/login': [Errno 111] Connection refused

The problem is that, when this error happens, there is no process listening on port 8843 (checked with netstat -tnlp). During previous steps, a java process (Tomcat?) is listening on port 8843, but it periodically goes down and up. Some of these restarts seems triggered by ipa-server-install, but other seems gratuitous.

Timo Aaltonen (tjaalton) wrote :

the restarts are caused by certmonger requests.. I've added a (very gross) 'sleep 80' to that stage which at least made it pass reliably on my qemu host, but looks like that's not enough. I'll ask upstream why it creates so many requests these days..

gianluca (amato) wrote :

Right... it was a race condition. Also, increasing the number of CPU and amount of memory in my virtual machine solved the problem.

gianluca (amato) wrote :

Now I have another problem. ipa-server-install stops at step "[19/21]: starting httpd" of HTTP configuration. From my investigation, it seems that the problem is that the SSL private key in /var/lib/ipa/private/httpd.key has a passphrase, saved in /var/lib/ipa/<host>-443-RSA. The passphrase is correct (I checked with openssl), but Apache does not find it. These are the messages I get in /var/log/apache2/error.log:

[Sat May 05 19:02:57.836869 2018] [mpm_event:notice] [pid 967:tid 140026405403584] AH00491: caught SIGTERM, shutting down
[Sat May 05 19:03:10.609244 2018] [ssl:emerg] [pid 6154:tid 140498019421120] AH02580: Init: Pass phrase incorrect for key ipa.labeconomia.unich.it:443:0
[Sat May 05 19:03:10.609443 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag
[Sat May 05 19:03:10.609465 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error
[Sat May 05 19:03:10.609481 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag
[Sat May 05 19:03:10.609498 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error (Type=RSA)
[Sat May 05 19:03:10.609514 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:04093004:rsa routines:old_rsa_priv_decode:RSA lib
[Sat May 05 19:03:10.609530 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag
[Sat May 05 19:03:10.609546 2018] [ssl:emerg] [pid 6154:tid 140498019421120] SSL Library Error: error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error (Type=PKCS8_PRIV_KEY_INFO)
[Sat May 05 19:03:10.609564 2018] [ssl:emerg] [pid 6154:tid 140498019421120] AH02311: Fatal error initialising mod_ssl, exiting. See /var/log/apache2/error.log for more information
[Sat May 05 19:03:10.609576 2018] [ssl:emerg] [pid 6154:tid 140498019421120] AH02564: Failed to configure encrypted (?) private key ipa.labeconomia.unich.it:443:0, check /var/lib/ipa/private/httpd.key

Timo Aaltonen (tjaalton) wrote :

file a new bug..

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package tomcat8 - 8.5.30-1ubuntu2

---------------
tomcat8 (8.5.30-1ubuntu2) cosmic; urgency=medium

  * support-jre8.diff: Fix running tomcat with JRE8. (LP: #1765616)

 -- Timo Aaltonen <email address hidden> Tue, 24 Apr 2018 23:47:45 +0300

Changed in tomcat8 (Ubuntu):
status: In Progress → Fix Released
Thomas (lostexception) wrote :

Sorry if I'm getting this completely wrong, but the fix seems to be for cosmic only. Does this mean tomcat8 will remain broken on bionic (which is an LTS)?

Timo Aaltonen (tjaalton) wrote :

no, a task for bionic is open and a version still waiting in proposed, it just needs to be fixed in the devel series first

Timo Aaltonen (tjaalton) wrote :

..waiting on the queue, not in proposed yet

Thomas (lostexception) wrote :

Timo, thanks a lot for clarification. Maybe you should change the subject of this bug to "Tomcat mostly broken on bionic" to get some more attention ;)

Timo Aaltonen (tjaalton) on 2018-05-08
summary: - freeipa server install fails - RuntimeError: CA configuration failed.
+ tomcat more or less broken -- java compat issues
no longer affects: freeipa (Ubuntu Bionic)
no longer affects: freeipa (Ubuntu)
tags: added: bionic cosmic

Hello Juan, or anyone else affected,

Accepted tomcat8 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tomcat8/8.5.30-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in tomcat8 (Ubuntu Bionic):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-bionic
Timo Aaltonen (tjaalton) wrote :

note that the freeipa ppa had a newer package version, so enabling -proposed isn't enough if you installed it from the ppa.. need to manually install the version from proposed like this:

apt install libtomcat8-java=8.5.30-1ubuntu1.1

etc

keestux (kees-bakker-xs4all) wrote :

Didn't you mean to say?

apt install libtomcat8-java=8.5.30-1ubuntu2

Timo Aaltonen (tjaalton) wrote :

no, the ppa has -1ubuntu2, bionic-proposed has 1.1

Jared Szechy (szechyjs) wrote :

I just installed 8.5.30-1ubuntu1.1 and it resolved the issue.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Sebastian (slovdahl) wrote :

I can also confirm that 8.5.30-1ubuntu1.1 now works with openjdk-8 in bionic.

Thomas (lostexception) wrote :

I can also confirm tomcat8 now working on bionic using openjdk-8-jre-headless (disclaimer: for my use case).

Timo Aaltonen (tjaalton) wrote :

thanks a lot for testing and feedback!

Rüdiger Kuhlmann (rk2b) on 2018-06-05
Changed in tomcat8 (Ubuntu Bionic):
status: Fix Committed → Confirmed
status: Confirmed → Fix Committed
Timo Aaltonen (tjaalton) on 2018-06-13
tags: added: verification-done
removed: verification-needed
Changed in tomcat8 (Debian):
status: New → Fix Released
Changed in tomcat8 (Ubuntu Bionic):
status: Fix Committed → Fix Released

Issue seems to be back with 8.5.30-1ubuntu1.4

It doesn't seem to affect startup, but happens in some situations, e.g.: direct output of files through response stream.
Issue is in file tomcat-coyote.jar, as replacing Ubuntu's file with upstream's tomcat-coyote.jar makes the issue disappear.

Upstream's and Ubuntu's files have indeed different md5sums.

gem-lx1-sv@gem-lx1-sv:/usr/share/java$ md5sum tomcat8-coyote-8.5.30-apache.jar tomcat8-coyote-8.5.30.jar
993e7d3920e00f39b7287fa5f5177a33 tomcat8-coyote-8.5.30-apache.jar
91de49bd30f68be4cbf64e217e98fbc8 tomcat8-coyote-8.5.30.jar

gem-lx1-sv@gem-lx1-sv:/usr/share/java$ ls -lha tomcat8-coyote*
-rw-r--r-- 1 root root 782K Nov 8 15:46 tomcat8-coyote-8.5.30-apache.jar
-rw-r--r-- 1 root root 782K Aug 13 22:23 tomcat8-coyote-8.5.30.jar
lrwxrwxrwx 1 root root 25 Aug 13 22:23 tomcat8-coyote.jar -> tomcat8-coyote-8.5.30-apache.jar

Stack trace for the bug:
[08-Nov-2018 13:25:26.651 SEVERE [http-nio-8080-exec-1] org.apache.coyote.http11.Http11Processor.service Error processing request
 java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
        at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:111)
        at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:226)
        at org.apache.coyote.Response.doWrite(Response.java:541)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)
        at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:815)
        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:310)
        at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:263)
        at org.apache.catalina.connector.Response.finishResponse(Response.java:484)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1463)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

I kindly request to either not recompile upstream's binary jars, or set a java8 JDK as default compiler for 18.04 or until java11 is stable on Ubuntu.

Sorry, forgot to add:
gem-lx1-sv@gem-lx1-sv:/usr/share/java$ dpkg -l libtomcat8-java
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==================================================-==============================-==============================-=========================================================================================================
ii libtomcat8-java 8.5.30-1ubuntu1.4 all Apache Tomcat 8 - Servlet and JSP engine -- core libraries

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.