tomcat8 has webapp class loader memory use bug

[Robin Miller]

This bug is causing excessive memory usage with one of our webapps, and has been fixed in 8.0.33. See https://tomcat.apache.org/tomcat-8.0-doc/changelog.html under 'Tomcat 8.0.33':
"Refactor the web application class loader to reduce the impact of JAR scanning on the memory footprint of the web application. (markt)"

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: tomcat8 8.0.32-1ubuntu1
ProcVersionSignature: Ubuntu 4.4.0-24.43-generic 4.4.10
Uname: Linux 4.4.0-24-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Fri Jun 17 19:51:36 2016
Ec2AMI: ami-f652979b
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1a
Ec2InstanceType: t2.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: tomcat8
UpgradeStatus: No upgrade log present (probably fresh install)

[Nish Aravamudan (nacc)]

Hello and thank you for filing this bug report! I have submitted a test backport of the required fix at:

https://launchpad.net/~nacc/+archive/ubuntu/lp1593854

It might take some time for it to build, but once it does, please test it.

I have requested yakkety get this fix via a sync from Debian with bug #1593857. Once that gets in, we can work on the SRU process for this bug, which will need to follow: https://wiki.ubuntu.com/StableReleaseUpdates.

In particular, if you can modify the description of this bug to follow the template from that page, that would be a huge help from an SRU perspective.

[Robin Miller (robincello)]

Thank you Nish. Unfortunately it looks like the patch applied in the PPA is not quite the one(s) we need. The patch we need is:

https://github.com/apache/tomcat80/commit/3c0177232399bd2fe5d675c6000dbb6503e177c2#diff-3f0901a8e99cbddfc81363aeb77a0109

...as well as...

https://github.com/apache/tomcat80/commit/a2fc002c0db8431b867c00dfbddc2100c70f71c6#diff-3f0901a8e99cbddfc81363aeb77a0109

Would it be possible to rebuild the test backport with those patches instead (or in addition)?

[Nish Aravamudan (nacc)]

On 23.06.2016 [15:57:30 -0000], Robin Miller wrote:
> Thank you Nish. Unfortunately it looks like the patch applied in the PPA
> is not quite the one(s) we need. The patch we need is:
>
> https://github.com/apache/tomcat80/commit/3c0177232399bd2fe5d675c6000dbb6503e177c2
> #diff-3f0901a8e99cbddfc81363aeb77a0109
>
> ...as well as...
>
> https://github.com/apache/tomcat80/commit/a2fc002c0db8431b867c00dfbddc2100c70f71c6
> #diff-3f0901a8e99cbddfc81363aeb77a0109
>
> Would it be possible to rebuild the test backport with those patches
> instead (or in addition)?

I have added them to the existing and uploaded 8.0.32-1ubuntu1.1~ppa2 to
the same PPA. Please test once it's available. Can you clarify with your
IT department which patches are actually needed for your issue? I can
provide test builds that only have these two patches applied, if that
would help.

-Nish

[Robin Miller (robincello)]

Hi Nish,

Just waiting for the package to publish so I can test, but looking at the changelog, I am wondering if this contains both the commits in comment #3 above, or just the first one?

I apologize for the confusion here BTW. I think it started due to me not specifying the commit I needed clearly in the initial report. The commit you included in 8.0.32-1ubuntu1.1~ppa, https://github.com/apache/tomcat80/commit/103cf11c1cdec6edc1bd497e015435c7d309d4a2, was not the correct one, but had a similar sounding changelog entry to the requested one ('Refactor the JAR and JAR-in-WAR resource handling to reduce the memory footprint of the web application. (markt)', vs 'Refactor the web application class loader to reduce the impact of JAR scanning on the memory footprint of the web application. (markt)').

After double checking with one of our developers who located the source of our memory use problem, it is the two commits noted in comment #3 that we need (the first of which was the one originally requested).

Hope that makes sense.

Thanks,
Robin

[Nish Aravamudan (nacc)]

On 23.06.2016 [22:44:17 -0000], Robin Miller wrote:
> Hi Nish,
>
> Just waiting for the package to publish so I can test, but looking at
> the changelog, I am wondering if this contains both the commits in
> comment #3 above, or just the first one?

The second was a patch to the first, so I folded them together. You can
see this in the source package:

Description: Backport of 'Refactor class loading so JAR scanning does
not trigger the caching of the byte[] for every scanned class until the
class is loaded.'
Author: Nishanth Aravamudan <email address hidden>
Bug-Ubuntu: https://bugs.launchpad.net/bugs/1593854
Origin: upstream,
https://github.com/apache/tomcat80/commit/3c0177232399bd2fe5d675c6000dbb6503e177c2
Origin: upstream,
https://github.com/apache/tomcat80/commit/a2fc002c0db8431b867c00dfbddc2100c70f71c6

> I apologize for the confusion here BTW. I think it started due to me not
> specifying the commit I needed clearly in the initial report. The commit
> you included in 8.0.32-1ubuntu1.1~ppa,
> https://github.com/apache/tomcat80/commit/103cf11c1cdec6edc1bd497e015435c7d309d4a2,
> was not the correct one, but had a similar sounding changelog entry to
> the requested one ('Refactor the JAR and JAR-in-WAR resource handling to
> reduce the memory footprint of the web application. (markt)', vs
> 'Refactor the web application class loader to reduce the impact of JAR
> scanning on the memory footprint of the web application. (markt)').
>
> After double checking with one of our developers who located the source
> of our memory use problem, it is the two commits noted in comment #3
> that we need (the first of which was the one originally requested).

Ok, so you just need the two commits from c#3? And don't need the
initial commit (~ppa1) at all? I'll submit a test build (~ppa3).

[Robin Miller (robincello)]

Hi Nish,

Yes, what we'd like is the two commits from c#3 only.

I just finished trying out ~ppa2 and unfortunately it appears to have something broken... we get the attached 'SEVERE' errors in catalina.out when trying to start our webapp (which worked fine with the first ~ppa, as well as with the current xenial package 8.0.32-1ubuntu1 and with the currrent yakkety package 8.0.36-1)...

-Robin

[Nish Aravamudan (nacc)]

On 24.06.2016 [17:52:14 -0000], Robin Miller wrote:
> Hi Nish,

> Yes, what we'd like is the two commits from c#3 only.

Ok, that should be what it's in ~ppa3 which I just uploaded to the
same PPA.

> I just finished trying out ~ppa2 and unfortunately it appears to have
> something broken... we get the attached 'SEVERE' errors in catalina.out
> when trying to start our webapp (which worked fine with the first ~ppa,
> as well as with the current xenial package 8.0.32-1ubuntu1 and with the
> currrent yakkety package 8.0.36-1)...

Possibly something further is needed to fix the underlying issue, then.
I will see if I can spend some time on this, but given that I don't
have a tomcat8 setup, it's relatively difficult.

[Nish Aravamudan (nacc)]

16.10 and 17.04 both have > 8.0.33 now, only Xenial needs the appropriate SRU.

[Nish Aravamudan (nacc)]

Hi Robin,

Can you test: https://launchpad.net/~nacc/+archive/ubuntu/tomcat8v2 specifically 8.0.32-1ubuntu1.3~ppa2 which should have your specified fixes plus some other backports from 8.0.33?

Thanks,
Nish

[Robin Miller (robincello)]

Hi Nish,

Thanks for getting back to this. I've been swamped with other work this week but will try to get that tested early next week, and will let you know.

Best,
Robin

> On Dec 14, 2016, at 4:34 PM, Nish Aravamudan <email address hidden> wrote:
>
> Hi Robin,
>
> Can you test: https://launchpad.net/~nacc/+archive/ubuntu/tomcat8v2
> specifically 8.0.32-1ubuntu1.3~ppa2 which should have your specified
> fixes plus some other backports from 8.0.33?
>
> Thanks,
> Nish
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1593854
>
> Title:
> tomcat8 has webapp class loader memory use bug
>
> Status in tomcat8 package in Ubuntu:
> Fix Released
> Status in tomcat8 source package in Xenial:
> In Progress
> Status in tomcat8 source package in Yakkety:
> Fix Released
>
> Bug description:
> This bug is causing excessive memory usage with one of our webapps, and has been fixed in 8.0.33. See https://tomcat.apache.org/tomcat-8.0-doc/changelog.html under 'Tomcat 8.0.33':
> "Refactor the web application class loader to reduce the impact of JAR scanning on the memory footprint of the web application. (markt)"
>
> ProblemType: Bug
> DistroRelease: Ubuntu 16.04
> Package: tomcat8 8.0.32-1ubuntu1
> ProcVersionSignature: Ubuntu 4.4.0-24.43-generic 4.4.10
> Uname: Linux 4.4.0-24-generic x86_64
> ApportVersion: 2.20.1-0ubuntu2.1
> Architecture: amd64
> Date: Fri Jun 17 19:51:36 2016
> Ec2AMI: ami-f652979b
> Ec2AMIManifest: (unknown)
> Ec2AvailabilityZone: us-east-1a
> Ec2InstanceType: t2.small
> Ec2Kernel: unavailable
> Ec2Ramdisk: unavailable
> PackageArchitecture: all
> ProcEnviron:
> TERM=xterm-256color
> PATH=(custom, no user)
> LANG=en_US.UTF-8
> SHELL=/bin/bash
> SourcePackage: tomcat8
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/tomcat8/+bug/1593854/+subscriptions