openjdk6 regression causes finalizers never to be called

[Alex Bligh]

Upgrading openjdk6 from 6b27-1.12.6-1ubuntu0.12.04.4 to 6b30-1.13.1-1ubuntu2~0.12.04.1 (Precise update in Jan / Feb 2014) causes finalizers not to run reliably. Whilst it is good practice to avoid use of finalizers, JDBC uses them to close connections, sockets use them to close sockets etc., and after this upgrade both of them leak apparently randomly under moderate load.

This is difficult to replicate but we believe it is due to: https://java.net/jira/browse/OPENJDK6-29 which was introduced in b28. The critical sentences here are: "This can cause mayhem in the JVM because the flag fields at the end of the struct end up taking values determined by the contents of the stack frame where info is allocated. In particular, it can cause the post-jdk6-b28 GC to enqueue discovered References using the wrong link field which means that no reference processing (including finalization) occurs."

This is fixed in b31, and the fix is a one line obviously correct change:
  http://hg.openjdk.java.net/jdk6/jdk6/jdk/rev/04e4c3ec6516

which correctly memset's a structure to zero, as opposed to only setting the first 8 bytes (64 bit) or 4 bytes (32 bit) to zero.

As this is a clear regression caused by an update with a substantial impact (unresolvable memory leaks of objects with finalizers, other resource leaks including FD leaks), I think this is an SRU candidate for Precise (which is where we are seeing it).

[Alex Bligh (ubuntu-alex-org)]

https://launchpad.net/ubuntu/+source/openjdk-6/+publishinghistory

confirms that this would have been introduced into Precise on 2014-02-27

[Alex Bligh (ubuntu-alex-org)]

Mailing list discussion:
  http://comments.gmane.org/gmane.comp.java.openjdk.jdk6.devel/1114

cleaned up in openjdk7 amongst a much larger commit

[Alex Bligh (ubuntu-alex-org)]

I'm currently running with the following patch (not entirely sure how the packaging works and whether I've done this the right wy). Passes all tests.

[Alex Bligh (ubuntu-alex-org)]

The above patch has a second instance of the same issue fixed. Mailing list thread from upstream here:
http://mail.openjdk.java.net/pipermail/jdk6-dev/2014-March/003253.html

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Patch to ubuntu packaging" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jamie Strandboge (jdstrand)]

I have uploaded packages for Ubuntu 12.04 and 14.04 to https://launchpad.net/~ubuntu-security-proposed/+archive/ppa and they should be done building in a couple hours.

Can someone who is affected by this bug comment on either to confirm that they fix this bug? If the issue is fixed, I will prepare packages for the other releases and a USN.

Thanks!

[Alex Bligh (ubuntu-alex-org)]

Jamie: we've verified that the patch I applied *appears* to fix the bug we are seeing. I say *appears* as the bug manifesting itself seems to be a function of uninitialized values on the stack; on some runs it explodes with leaking stuff straight away, other times it runs fine for many many hours, and only later explodes. I haven't verified your .deb files but could you tell me what patch(es) they are using? I can then get QA here to play with them.

[Jamie Strandboge (jdstrand)]

Alex, I used the patch from http://hg.openjdk.java.net/jdk6/jdk6/jdk/rev/04e4c3ec6516. You can see it here:
https://launchpadlibrarian.net/170707257/openjdk-6_6b30-1.13.1-1ubuntu2~0.12.04.1_6b30-1.13.1-1ubuntu2~0.12.04.2.diff.gz

[Jamie Strandboge (jdstrand)]

Hrm, seems I missed the patch to hotspot: https://bugs.openjdk.java.net/browse/JDK-8035893.

[Alex Bligh (ubuntu-alex-org)]

Yes , the version I tested (with my patch, as attached to this bug) has both of those fixed.

[Jamie Strandboge (jdstrand)]

Updated packages have been uploaded to the security-proposed PPA (6b30-1.13.1-1ubuntu2~0.12.04.3 for precise and 6b30-1.13.1-1ubuntu4 for trusty). These incorporate both fixes. Also, I verified that these patches are applied in upstream IcedTea as well so this should not regress in the next update.

[Jamie Strandboge (jdstrand)]

Can someone who is affected by this bug comment on either the new 14.04 or the 12.04 packages to confirm that they fix this bug? If the issue is fixed, I will prepare packages for the other releases and issue a USN.

Thanks!

[Alex Bligh (ubuntu-alex-org)]

Will do once they have finished building. Is there a link to the diff in the mean time so I can check it against the one we've been using?

[Jamie Strandboge (jdstrand)]

Here is the debdiff used for precise.

[Alex Bligh (ubuntu-alex-org)]

Jamie: thanks. They look good to me (from inspection of the diffs). Will get these checked tomorrow once they have built. Normally our tests take a couple of days to run.

[Michael Kosteva (mkosteva)]

A half-day's worth of before/after testing in our QA environment, and we have not seen a recurrence of the bugged behaviour with the builds from PPA (12.04 amd64), while consistently being able to reproduce it on pre-patched debs on the same instances.

thanks

[Alex Bligh (ubuntu-alex-org)]

48 hours of testing show no problems with these packages, whereas the previously released packages show leaks. We will continue to run these tests over the weekend, but it's looking good for us so far.

[Jamie Strandboge (jdstrand)]

Thanks for the feedback! I uploaded updates for 10.04, 12.10 and 13.10 to the security-proposed ppa. Unless a problem comes up over the weekend, I plan to push these out on Monday.

[Alex Bligh (ubuntu-alex-org)]

I can confirm the Precise packages passed our torture tests over the weekend. These are good to go as far as I am concerned.

[Alex Bligh (ubuntu-alex-org)]

Jamie: any news on this one?

[Stefan Huehner (stefan-huehner)]

Jamie:
I can also confirm that those packages do fix the problem for us.

I've added your proposed precise package 6b30-1.13.1-1ubuntu2~0.12.04.3 0 to two of our servers which were crashing with the original 1.13.1 packages frequently (twice a day).

Both servers have been running now around 25hour without any problems.

[pionic (o-marius)]

Any update on this ticket? Thanks

[Jamie Strandboge (jdstrand)]

It will be pushed today.

[Alex Bligh (ubuntu-alex-org)]

Any news on the fix being released for Lucid?

[Launchpad Janitor (janitor)]

This bug was fixed in the package openjdk-6 - 6b30-1.13.1-1ubuntu2~0.10.04.2

---------------
openjdk-6 (6b30-1.13.1-1ubuntu2~0.10.04.2) lucid-security; urgency=medium

  * Update to properly zero version info (LP: #1295987)
    - 6989972.diff: JDK fails to zero jdk_version_info. Apply in
      DISTRIBUTION_PATCHES
    - debian/openjdk/8035893.diff: JVM_GetVersionInfo fails to zero structure.
      Apply in DISTRIBUTION_PATCHES
 -- Jamie Strandboge <email address hidden> Fri, 28 Mar 2014 12:58:47 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package openjdk-6 - 6b30-1.13.1-1ubuntu2~0.12.10.2

---------------
openjdk-6 (6b30-1.13.1-1ubuntu2~0.12.10.2) quantal-security; urgency=medium

  * Update to properly zero version info (LP: #1295987)
    - 6989972.diff: JDK fails to zero jdk_version_info. Apply in
      DISTRIBUTION_PATCHES
    - debian/openjdk/8035893.diff: JVM_GetVersionInfo fails to zero structure.
      Apply in DISTRIBUTION_PATCHES
 -- Jamie Strandboge <email address hidden> Fri, 28 Mar 2014 13:00:04 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package openjdk-6 - 6b30-1.13.1-1ubuntu2~0.13.10.2

---------------
openjdk-6 (6b30-1.13.1-1ubuntu2~0.13.10.2) saucy-security; urgency=medium

  * Update to properly zero version info (LP: #1295987)
    - 6989972.diff: JDK fails to zero jdk_version_info. Apply in
      DISTRIBUTION_PATCHES
    - debian/openjdk/8035893.diff: JVM_GetVersionInfo fails to zero structure.
      Apply in DISTRIBUTION_PATCHES
 -- Jamie Strandboge <email address hidden> Fri, 28 Mar 2014 13:01:05 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package openjdk-6 - 6b30-1.13.1-1ubuntu2~0.12.04.3

---------------
openjdk-6 (6b30-1.13.1-1ubuntu2~0.12.04.3) precise-security; urgency=medium

  * Update to properly zero version info (LP: #1295987)
    - 6989972.diff: JDK fails to zero jdk_version_info. Apply in
      DISTRIBUTION_PATCHES
    - debian/openjdk/8035893.diff: JVM_GetVersionInfo fails to zero structure.
      Apply in DISTRIBUTION_PATCHES
 -- Jamie Strandboge <email address hidden> Wed, 26 Mar 2014 08:43:13 -0500