Fresh install with JDK 9 can't use the generated PKCS12 cacerts keystore file

[Antti S. Lankila]

[Impact]
Any user doing a new install can be affected as soon as they install any openjdk-11 package.

[Cause]
The ca-certificate-java version 20170930 (or earlier) used OpenJDK's default keystore to create /etc/ssl/certs/java/cacerts - if the file already existed its contents were just updated without changing the keystore type.

From openjdk-9 upwards the default keystore type changed from 'jks' to 'pkcs12' [1] by means of JEP 229 [2]. A JKS keystore can be read without supplying a password (or by supplying an empty one) while a PKCS12 keystore requires a password to be set.

Thus a /etc/ssl/certs/java/cacerts created in the pkcs12 format will fail to be loaded as, by default, the truststore password is empty - in order to avoid that the user must set -Djavax.net.ssl.trustStorePassword=<passwd> or define it in /etc/java-XX-openjdk/management/management.properties. A JKS keystore will work normally, as the certificates in it can be ready when the truststore password is empty.

Ubuntu does *not* set the javax.net.ssl.trustStorePassword by default
thus any user that got a cacerts generated in JKCS12 won't be able
to use any secure connections from java.

[Test Case - Fix not applied]
Start on a new bionic install/chroot without openjdk

1. Install openjdk-11
$ sudo apt-get install openjdk-11-jdk

2. Test the keystore with an empty password (optional) and make sure it is a PKCS12
$ keytool -list -cacerts
Enter keystore password: <leave empty>
***************** WARNING WARNING WARNING *****************
* The integrity of the information stored in your keystore *
* has NOT been verified! In order to verify its integrity, *
* you must provide your keystore password. *
***************** WARNING WARNING WARNING *****************
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 0 entries

3. Test with the "changeit" password
$ keytool -list -cacerts
Enter keystore password: changeit
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 133 entries
<snipped various certs>

4. Create the java test file
$ cat <<EOF >HttpsTester.java
import java.net.URL;
import javax.net.ssl.HttpsURLConnection;
public class HttpsTester {
public static void main(String[] args) throws java.io.IOException {
HttpsURLConnection connection = (HttpsURLConnection) new URL("https://www.ubuntu.com").openConnection();
System.out.println("Response code: " + connection.getResponseCode());
System.out.println("It worked!");
}
}
EOF

5. Compile it
$ javac HttpsTester.java

6. Call it
$ /usr/lib/jvm/java-11-openjdk-amd64/bin/java HttpsTester

7. Call it again, this time set the store password
$ /usr/lib/jvm/java-11-openjdk-amd64/bin/java \
  -Djavax.net.ssl.trustStorePassword=changeit HttpsTester
Response code: 200
It worked!

[Test Case - Fix applied]
Start on a new bionic install/chroot without openjdk

1. Install openjdk-11
$ sudo apt-get install openjdk-11-jdk

2. Test the keystore with an empty password (optional) and make sure it is a JKS
$ keytool -list -cacerts
Enter keystore password:
***************** WARNING WARNING WARNING *****************
* The integrity of the information stored in your keystore *
* has NOT been verified! In order to verify its integrity, *
* you must provide your keystore password. *
***************** WARNING WARNING WARNING *****************
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 133 entries
<snipped various certs>

3. Test with the "changeit" password
keytool -list -cacerts
Enter keystore password: changeit
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 133 entries
<snipped various certs>

4. Create the java test file
$ cat <<EOF >HttpsTester.java
import java.net.URL;
import javax.net.ssl.HttpsURLConnection;
public class HttpsTester {
public static void main(String[] args) throws java.io.IOException {
HttpsURLConnection connection = (HttpsURLConnection) new URL("https://www.ubuntu.com").openConnection();
System.out.println("Response code: " + connection.getResponseCode());
System.out.println("It worked!");
}
}
EOF

5. Compile it
$ javac HttpsTester.java

6. Call it
$ /usr/lib/jvm/java-11-openjdk-amd64/bin/java HttpsTester
Response code: 200
It worked!

7. Call it again, this time set the store password
$ /usr/lib/jvm/java-11-openjdk-amd64/bin/java \
  -Djavax.net.ssl.trustStorePassword=changeit HttpsTester
Response code: 200
It worked!

[Regression Potential]
* Forcing ca-certificates-java to create the keystore in the old default JKS format does not cause any regressions, only forces the system to behave as it did before openjdk-9 changed the default keystore from JKS to PKCS12.

[References]
[1] The default keystore is defined by the keystore.type in the
/etc/java-XX-openjdk/security/java.security file.
http://hg.openjdk.java.net/jdk-updates/jdk9u/jdk/annotate/46bd35a597eb/src/java.base/share/conf/security/java.security#l186

[2] JEP 229: Create PKCS12 Keystores by Default
http://openjdk.java.net/jeps/229

[Original bug description]
I ran into a problem after doing approximately the following on an install of Ubuntu 17.10:

sudo apt-get install openjdk-9-jdk maven ca-certificates-java

Running "mvn package" on my own project threw this error without downloading anything:

java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

It seems that all TLS connections fail due to missing trust anchors in Java 9!

After some investigation, I discovered that the JDK's lib/security/cacerts is a symlink to /etc/ssl/certs/java/cacerts, which is provided by ca-certificates-java package. This file appeared to be a PKCS12 file with password "changeit" protecting it. I was able to list its contents using both keytool -list -cacerts and openssl pkcs12 -in cacerts with that password, confirming that the file actually did hold the certificates. Regardless, Java 9 was not able to use the contents of this file for whatever reason.

To workaround the issue, I downgraded to openjdk-8-jdk, did rm /etc/ssl/certs/java/cacerts, then did update-ca-certificates -f, then upgraded back to openjdk-9-jdk. The old Java 8 -generated JKS file with empty string as password was usable in the Java 9, permitting mvn and other things to make TLS connections again.

The problem can be reintroduced by having java 9 installed and doing rm /etc/ssl/certs/java/cacerts and then update-ca-certificates -f.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: ca-certificates-java 20170930
ProcVersionSignature: Ubuntu 4.13.0-21.24-generic 4.13.13
Uname: Linux 4.13.0-21-generic x86_64
ApportVersion: 2.20.8-0ubuntu5
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Thu Dec 21 17:36:05 2017
EcryptfsInUse: Yes
InstallationDate: Installed on 2017-12-21 (0 days ago)
InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64 (20171018)
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=fi_FI.UTF-8
 SHELL=/bin/bash
SourcePackage: ca-certificates-java
UpgradeStatus: Upgraded to bionic on 2017-12-21 (0 days ago)
modified.conffile..etc.default.cacerts: [inaccessible: [Errno 13] Lupa evätty: '/etc/default/cacerts']

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Artur Godlinski (artgod)]

The OpenJDK was shipped with an empty cacerts file, which was fixed with version 9.0.4 (See: https://bugs.java.com/view_bug.do?bug_id=JDK-8189131)

Workaround:
Extract the 'cacerts' file from the latest OpenJDK and copy it into /etc/ssl/certs/java

[Antti S. Lankila (alankila)]

While it may be so that OpenJDK ships with empty certificates file, this is not sufficient to explain the issue, or consistent with the bug report I made. Quoting from the original bug report: "I discovered that the JDK's lib/security/cacerts is a symlink to /etc/ssl/certs/java/cacerts, which is provided by ca-certificates-java package".

This symlink exists, and it is the one used by JDK. The issue was that JDK9 is unable to read the contents of PKCS12-formatted keystore file, but is able to read its old JKS keystore file. In both cases, the files do contain certificates.

[kb3gtn (kb3gtn)]

I have noticed a similar issue with openjdk-9 on Ubuntu 18.04 alpha and getting errors in java applications with exception message:

java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

This is due to the default cacerts for java installed in /etc/ssl/certs/java/cacerts being in the jks format where the default keystore.type for OpenJDK-9 is pkcs12.

A simple work around is to edit /etc/java-9-openjdk/security/java.security and change "keystore.type=pkcs12" to "keystore.type=jks"

[Antti S. Lankila (alankila)]

To comment #5: If what you describe is true, then that is a different bug, somehow. The default cacerts file should be in pkcs12 format, which can't be used by java for some reason. The JKS keystore file can be read, regardless of the keystore type setting in the security file.

However, I am happy about the suggestion to change the keystore.type=jks parameter in the java.security file. Once this change has been made, openjdk-9 can also generate JKS keystores from "udpate-ca-certificates -f" and that is altogether simpler way to recover from this bug than installing JDK 8, let it generate keystore, and then update to JDK 9 that preserves the cacerts in JKS format regardless of the settings of JDK 9.

So here are the workaround steps that can be done instead, to fix TLS for Java 9 when the keystore type happens to be PKCS12.

1. edit /etc/java-9-openjdk/security/java.security file. Find the line that says keystore.type = pkcs12 and change that to jks

2. rm /etc/ssl/certs/java/cacerts file

3. run "update-ca-certificates -f"

[mjw99 (mark-williamson)]

I am seeing this with a 18.04 nightly; this is pretty much a showstopper for any one developing with Maven.

[pkmo-linux (pkmo-linux-deactivatedaccount)]

This issue has been resolved upstream in 20180413 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=894979

Can we please see it here sometime soon?

[Don-vip (vincent-privat)]

See https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1769013 for merge of ca-certificates-java 20180413

[Launchpad Janitor (janitor)]

This bug was fixed in the package ca-certificates-java - 20180413ubuntu1

---------------
ca-certificates-java (20180413ubuntu1) cosmic; urgency=medium

  * Merge from debian unstable. Remaining changes: (LP: #1769013,
    LP: #1739631)
    + debian/control: Bump javahelper build dependency.
    + debian/rules:
      - Explicitly depend on openjdk-11-jre-headless, needed to configure.
      - Replace javac arguments '-source 1.7 -target 1.7' with '--release 7'
        as, per JEP-247, it also takes care of setting the right -bootclasspath
        argument.
  * debian/jks-keystore.hook.in: don't create a jvm-*.cfg file, a default file
    with the right configuration is already supplied by the openjdk packages.

ca-certificates-java (20180413) unstable; urgency=medium

  * Team upload.
  * Always generate a JKS keystore instead of using the default format
    (Closes: #894979)
  * Look for Java 10 and Java 11 when detecting the JRE
  * Removed Damien Raude-Morvan from the uploaders (Closes: #889412)
  * Standards-Version updated to 4.1.4
  * Switch to debhelper level 11

 -- Tiago Stürmer Daitx <email address hidden> Fri, 04 May 2018 01:31:24 +0000

[Łukasz Zemczak (sil2100)]

Hello Antti, or anyone else affected,

Accepted ca-certificates-java into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ca-certificates-java/20180516ubuntu1~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mikael Gueck (gumi)]

Tested the proposed fix version 20180516ubuntu1~18.04.1 in a Docker container, and it DID fix the issue, both as an upgrade to a previously installed package version 20170930ubuntu1, and as a first install.

Verification steps: Ran the TestHttps program from https://git.mikael.io/mikaelhg/broken-docker-jdk9-cacerts. It successfully completed without throwing an exception, after the upgrade to 20180516ubuntu1~18.04.1.

Verified package version:

root@89353b964227:/app# apt-cache show ca-certificates-java
Package: ca-certificates-java
Architecture: all
Version: 20180516ubuntu1~18.04.1
Multi-Arch: foreign
Priority: optional
Section: misc
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Debian Java Maintainers <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 42
Depends: ca-certificates (>= 20121114), openjdk-11-jre-headless | java8-runtime-headless, libnss3 (>= 3.12.9+ckbi-1.82-0ubuntu3~)
Filename: pool/main/c/ca-certificates-java/ca-certificates-java_20180516ubuntu1~18.04.1_all.deb
Size: 12156
MD5sum: fed1dbe07d960d581a8870b6e103eb69
SHA1: c0305a200fb55296a077014af3fd3ad7a4de756d
SHA256: 2c312d1c8a14781fc9a074569c9d591e17e00419ab9597a148223d0ac4065bb2
Description: Common CA certificates (JKS keystore)
Description-md5: 304cd3554728e5d076f8ecbb3b5057d8
Task: kubuntu-desktop, kubuntu-full
Supported: 5y

[Launchpad Janitor (janitor)]

This bug was fixed in the package ca-certificates-java - 20180516ubuntu1~18.04.1

---------------
ca-certificates-java (20180516ubuntu1~18.04.1) bionic; urgency=medium

  * Backport from Cosmic. (LP: #1770553)

ca-certificates-java (20180516ubuntu1) cosmic; urgency=low

  * Merge from Debian unstable (LP: #1771815). Remaining changes:
    - debian/control: Bump javahelper build dependency.
    - debian/rules:
      + Explicitly depend on openjdk-11-jre-headless, needed to configure.
      + Replace javac arguments '-source 1.7 -target 1.7' with '--release 7'
        as, per JEP-247, it also takes care of setting the right -bootclasspath
        argument.

ca-certificates-java (20180516) unstable; urgency=medium

  * Team upload.

  [ Tiago Stürmer Daitx ]
  * debian/jks-keystore.hook.in: don't create a jvm-*.cfg file, a default file
    with the right configuration is already supplied by the openjdk packages.
  * debian/jks-keystore.hook.in, debian/postinst.in: Only export JAVA_HOME
    and update PATH if a known jvm was found.
  * debian/postinst.in: Detect PKCS12 cacert keystore generated by
    previous ca-certificates-java and convert them to JKS. (Closes: #898678)
    (LP: #1771363)

  [ Matthias Klose ]
  * debian/rules: Explicitly depend on openjdk-11-jre-headless, needed to
    configure.

  [ Emmanuel Bourg ]
  * Use salsa.debian.org Vcs-* URLs

ca-certificates-java (20180413ubuntu1) cosmic; urgency=medium

  * Merge from debian unstable. Remaining changes: (LP: #1769013,
    LP: #1739631)
    + debian/control: Bump javahelper build dependency.
    + debian/rules:
      - Explicitly depend on openjdk-11-jre-headless, needed to configure.
      - Replace javac arguments '-source 1.7 -target 1.7' with '--release 7'
        as, per JEP-247, it also takes care of setting the right -bootclasspath
        argument.
  * debian/jks-keystore.hook.in: don't create a jvm-*.cfg file, a default file
    with the right configuration is already supplied by the openjdk packages.

ca-certificates-java (20180413) unstable; urgency=medium

  * Team upload.
  * Always generate a JKS keystore instead of using the default format
    (Closes: #894979)
  * Look for Java 10 and Java 11 when detecting the JRE
  * Removed Damien Raude-Morvan from the uploaders (Closes: #889412)
  * Standards-Version updated to 4.1.4
  * Switch to debhelper level 11

 -- Tiago Stürmer Daitx <email address hidden> Thu, 17 May 2018 14:10:59 +0000

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ca-certificates-java has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.