Broken symlinks for JSP support in libjetty-extra-java version 6.1.26-1ubuntu1.1

[Mauro]

Version 6.1.26-1ubuntu1.1 of libjetty-extra-java has changed the dependency for Jasper (to enable JSP support) from libtomcat6-java package to libtomcat7-java.

The Jasper JAR files provided by libtomcat7-java have different names:
tomcat-jasper.jar instead of jasper.jar
tomcat-jasper-el.jar instead of jasper-el.jar

However the symlinks provided by the libjetty-extra-java in /usr/share/jetty/lib/jsp-2.1 folder have not been updated, as they still point to /usr/share/java/jasper.jar and /usr/share/java/jasper-el.jar: hence, the symlinks are broken and Jetty starts without JSP support (even if the libjetty-extra-java package is installed).

Changing the symlinks to point to the updated file names bring to NoClassDefFoundExceptions, probably because of different dependencies of Tomcat 7 Jasper libraries:

[main] WARN org.mortbay.log - failed ContextHandlerCollection@37403a09: java.lang.NoClassDefFoundError: org/apache/tomcat/PeriodicEventListener
[main] INFO org.mortbay.log - Opened /var/log/jetty/2015_10_21.request.log
[main] WARN org.mortbay.log - failed HandlerCollection@4c3db835: java.lang.NoClassDefFoundError: org/apache/tomcat/PeriodicEventListener
[main] ERROR org.mortbay.log - Error starting handlers
java.lang.NoClassDefFoundError: org/apache/tomcat/PeriodicEventListener
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
        at org.mortbay.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:401)
        at org.mortbay.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:363)
        at org.mortbay.util.Loader.loadClass(Loader.java:91)
        at org.mortbay.util.Loader.loadClass(Loader.java:71)
        at org.mortbay.jetty.webapp.WebXmlConfiguration.initServlet(WebXmlConfiguration.java:510)
        at org.mortbay.jetty.webapp.WebXmlConfiguration.initWebXmlElement(WebXmlConfiguration.java:335)
        at org.mortbay.jetty.webapp.WebXmlConfiguration.initialize(WebXmlConfiguration.java:289)
        at org.mortbay.jetty.webapp.WebXmlConfiguration.configure(WebXmlConfiguration.java:222)
        at org.mortbay.jetty.webapp.WebXmlConfiguration.configureDefaults(WebXmlConfiguration.java:162)
        at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1262)
        at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
        at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
        at org.mortbay.jetty.Server.doStart(Server.java:224)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:985)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.mortbay.start.Main.invokeMain(Main.java:194)
        at org.mortbay.start.Main.start(Main.java:534)
        at org.mortbay.jetty.start.daemon.Bootstrap.start(Bootstrap.java:30)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: java.lang.ClassNotFoundException: org.apache.tomcat.PeriodicEventListener
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)

Please note that WebXmlConfiguration.java:510 is exactly the line in which the class for JSP support is loaded.
I'll try to see if I can make this work with Tomcat 7 Jasper, but I was wondering why the dependency has been upgraded from libtomcat6-java to libtomcat7-java.

[Mauro (mauromol)]

To fix the aforementioned NoClassDefFoundError exception, an additional symlink must be added in /usr/share/jetty/lib/jsp-2.1 pointing to /usr/share/java/tomcat-api.jar.

Now Jetty starts cleanly for me and with JSP support. I haven't tested a JSP invocation yet, though.

[Mauro (mauromol)]

I got to a dead end. To try to make EL actually work, I had to change the el-api.jar symlink to point to /usr/share/java/tomcat-el-api-2.2.jar and add a new symlink to /usr/share/java/tomcat-util.jar. The next error, however, is a NoSuchMethodError caused by a conflicting Servlet API JAR. I don't think I can upgrade this without causing big troubles to Jetty 6...

So, I think I must find a way to rollback to libtomcat6-java...

[Mauro (mauromol)]

I reverted to Jetty package version 6.1.26-1ubuntu1 to solve this problem.

However, that version has also another problem, as outlined by Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=740596
In practice, a new symlink to /usr/share/java/tomcat-coyote.jar is needed in /usr/share/jetty/lib/jsp-2.1 to make EL work correctly.
This may be considered yet another bug.

Anyway, the problem described here with version 6.1.26-1ubuntu1.1 of Jetty package remains: I don't think Jetty 6 can easily be made to work with Tomcat 7 Jasper/EL implementation.

[Hans Joachim Desserud (hjd)]

Hello and thanks for reporting this issue.

>I was wondering why the dependency has been upgraded from libtomcat6-java to libtomcat7-java.

I see that you have found bug 1286779 which contains more details about the update in 14.04. The short version is that conflicts between tomcat6 and tomcat7 made it harder to install packages together. There seemed to be a simple patch which changed the dependency so that was picked from the later version and applied.

Unfortunately, I now see that it has caused a regression and I am of course sorry about that.

I looked further into this, since newer versions of libjetty-extra-java contain the same patch and found that the symlinks are also broken on Ubuntu 15.10 and also Debian Sid. So this is actually affecting all newer versions of the package. :(

I've forwarded the bug to Debian so hopefully the package maintainers will take a look at this. One thing is of course the 14.04 update (which of course is a shame), but the issue will need to be resolved in general for newer versions. So I believe that once we have a solution for this we can look at whether that should be applied to 14.04 as well, or whether it should be rolled back to tomcat6 (which of course would re-introduce the package conflicts.)

[Mauro (mauromol)]

IMHO trying to use Tomcat 7 Jasper/EL implementations with Jetty 6 is not worth the effort, if it will ever work.

I mean, Jetty 6 implements Servlet 2.5/JSP 2.1/EL 2.1, which are the same specification versions implemented by Tomcat 6.
Tomcat 7, instead, implements Servlet 3.0/JSP 2.2/EL 2.2 specification versions, although it supports older versions, of course.
If the Tomcat 7 EL implementation requires an upgrade of the EL and Servlet API jars to properly work (as I experienced when I tried to just fix the symlinks), I'm really sceptical that this can be done without causing problems like multiple conflicting versions of the same JARs in classpath or other side effects on Jetty 6 standard operations. Maybe the new APIs and corresponding Tomcat implementations are perfectly backward-compatible, but setting this up properly could be not so easy (you're actually changing the core API JARs of the servlet container).

Going back to the original problem of the conflicting dependencies, I was wondering if it really is a "problem". I mean, just like you say that Tomcat 7 in Ubuntu can't be installed together with Tomcat 6, why not say that even Jetty 6 can't be installed together with Tomcat 7? We're talking about a servlet container which is EOL since Nov 2010 (see: https://wiki.eclipse.org/Jetty/Starting/Jetty_Version_Comparison_Table), so you may either:
1) remove it from recent Ubuntu/Debian official repositories
2) or just keep the old dependency and force the user to alternatively have just Jetty 6 or any other library that depends on a newer Tomcat version

After all, I think that if one needs to use Jetty 6 on a recent Ubuntu (like me), it's because he needs to set up a dedicated system to run a very old application that is not worth to update to run on a newer version of Jetty (or any other recent version of another servlet container, like Tomcat). In such a scenario, having the ability to run much newer packaged software on the same system is most probably useless.

The third solution might be to change the dependency against libtomcat6-java to optional: after all, Jetty can run even without JSP and EL support. The user then may choose to either a) leave these features disabled, or b) to install libtomcat6-java to enable them or c) to manually provide the required Jasper/EL JARs by himself for the same purpose, if he needs to keep the conflicting libtomcat7-java package installed instead.

[Hans Joachim Desserud (hjd)]

Added regression tags:
regression-release: because later Ubuntu releases (ie 15.04, 15.10...) are affected
regression-update: because the SRU introduced it in 14.04

[Hans Joachim Desserud (hjd)]

Hello again. I discussed this on IRC with one of the Debian Java packagers here the other day. We didn't reach any conclusions but a couple of suggestions was mentioned.

1. I realize you are setting up an older application, but did you try with jetty8? Apparently it might work out of the box without any porting.

2. There's a couple more tomcat jars missing, could you try add symlinks for the following:
tomcat-util
tomcat-api
tomcat-juli
in addition to jasper and jasper-el? (Looks like you already found tomcat-util, based on comment #2).
See also the list of symlinks used by jetty8 which also use tomcat7 https://sources.debian.net/src/jetty8/8.1.18-1/debian/libjetty8-extra-java.links/.

[Dobedani (dobedani)]

I did indeed experience problems and tried to correct the symlinks that were broken, as was visible from the File manager. I also added a few extra symlinks to Tomcat jars in the /usr/share/java folder. It then seemed that I was getting a step further, but unfortunately I then got this error: javax.el.ExpressionFactory.newInstance()Ljavax/el/ExpressionFactory
I concluded that the symlinks el-api.jar and jsp-api.jarhad to be adapted to point to version 2.2 jars, so I upgraded them. Only to get more errors: javax.servlet.ServletContext.getClassLoader()Ljava/lang/ClassLoader; and
No org.apache.tomcat.InstanceManager set in ServletContext. I still hope that this can be fixed. Jetty is important as web server for Solr.

[SaintlyVi (wiebke-toussaint)]

I spent the past two days trying to install solr from solr-jetty on Ubuntu 14.04 as part of a ckan installation. I've run into the same problem - solr not running due to no support for JSP. Thanks to @mauromol I found the broken symlinks and pointed them to the tomcat-jasper files. Now landed up with the same 'ClassNotFoundException'.

Please let me know if you find a work around or fix to the issue.

[Eric Ding (ericding-alum)]

After spending a couple hours trying to fix or work around this issue, I eventually just decided that the only workaround for now is a downgrade. So I installed jetty and libjetty-extra-java 6.1.26-ubuntu1, but then bumped into a problem with the most recent version of libtomcat6-java (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=740596), so I manually downloaded and installed the earlier packaged version of libtomcat6-java (6.0.37-1). I'm happy to report that Solr now works on my machine.

Better would be if jetty6 were fixed to only depend on libtomcat6-java again instead of libtomcat7-java, I suspect, along with a backport of the fixes that were done to resolve the Debian bug mentioned above; but for my purposes, I'm content to just stick with earlier packages for what is a localhost-only setup.

[Rarylson Freitas (rarylson)]

Hi,

I'm having the same problem when trying to run Solr from the `solr-jetty` package in Trusty.

I discovered that the problem was solved in Debian, and I'm hope the solution will be backported to Ubuntu 14.04 soon.

However, I'm here to share three **workarounds**.

All of them consist on use both `jetty` and `libtomcat7-java`, but adding the `jsp-api-2.1-6.0.2.jar` file (or some other similar) to the Jetty classpath.

I don't know if they have some problem. Use them at your own risk!

Workaround 1 - Install the fix package proposed by vshn

I found this workaround here: https://github.com/ckan/ckan/pull/2966

In short:

```
cd /tmp
wget https://launchpad.net/~vshn/+archive/ubuntu/solr/+files/solr-jetty-jsp-fix_1.0.2_all.deb
dpkg -i solr-jetty-jsp-fix_1.0.2_all.deb
service jetty restart
```

This will install a proper jsp-lib jar that works (the file will be named `jsp-2.1-6.0.2.jar`, but is the same `jsp-api-2.1-6.0.2.jar` file from other solutions).

Workaround 2 - Manually install the JSP jar

Download the jar to your server.

You can use one of these URLs:

- http://www.java2s.com/Code/Jar/j/Downloadjspapi21602jar.htm
- https://mvnrepository.com/artifact/org.mortbay.jetty/jsp-api-2.1/6.0.2

For example:

```
wget http://central.maven.org/maven2/org/mortbay/jetty/jsp-api-2.1/6.0.2/jsp-api-2.1-6.0.2.jar
```

Now, move it to a proper location inside the Jetty config dir. I did it this way:

```
mkdir /etc/jetty/extra-jars
mv jsp-api-2.1-6.0.2.jar /etc/jetty/extra-jars
```

And add a line like this one in the Jetty `start.config` file:

```
echo "/etc/jetty/extra-jars/jsp-api-2.1-6.0.2.jar" >> /etc/jetty/start.config
```

And:

```
service jetty restart
```

Workaround 3 - Use the JSP jar file from the `libservlet2.5-java` package

You can install `libservlet2.5-java` (files from Tomcat 6) even if `libtomcat7-java` is installed. There is no conflict here.

This package will install also the JSP API 2.1 - 6.0.39-1. Now, let's add this file to the Jetty classpath:

```
echo "/usr/share/maven-repo/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar" >> /etc/jetty/start.config
```

And:

```
service jetty restart
```

I did not test this last solution. If someone test it, please share the experience with us :).

[Rarylson Freitas (rarylson)]

I've tested all the previous three workarounds with Jetty+Solr, and Solr is working normally.

Maybe the last solution is the better one.

[Rarylson Freitas (rarylson)]

Sorry, the last workaround does NOT work. Use the first or the second one.

[Rarylson Freitas (rarylson)]

I updated the workarounds in this post: http://stackoverflow.com/a/40893892/2530295.