Apps expecting an OpenSSL 1.1 -formatted openssl.cnf fail

Bug #1979639 reported by David Zuelke
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
nodejs (Ubuntu)
Confirmed
Medium
Unassigned
Jammy
Confirmed
Medium
Unassigned
Kinetic
Won't Fix
Medium
Unassigned
openssl (Ubuntu)
Fix Released
High
Unassigned
Jammy
Won't Fix
High
Unassigned
Kinetic
Fix Released
High
Unassigned

Bug Description

[Impact]
While the default configuration works fine for every package that uses the system libssl3, libssl1.1.1 (which implicitly loads the configuration on OPENSSL_crypto_init()) fails to parse it.

Our nodejs package vendors openssl 1.1.1, which means it will trigger this bug. In addition, upstream NodeJS explicitly points their statically-linked OpenSSL to this file as well, and ships 1.1.1 in their current LTS (branch 16.x).
Finally, we can also expect breakage for third-party packages that still depends on libssl1.1.

If the provider section isn't present in the configuration, libssl3 will load the default provider, which means that commenting out the section won't impact the behavior of standard libssl3 users.

[Test Plan]

On a system with a pristine openssl configuration:

$ sudo apt install nodejs
$ nodejs - <<EOF
var crypto = require('crypto')
const { privateKey, publicKey } = crypto.generateKeyPairSync('rsa', { modulusLength: 2048 });
var sign = crypto.createSign('RSA-SHA256');
sign.update(Buffer.from("hello"));
sign.sign(privateKey.export({type: 'pkcs1', format: 'pem'}));
EOF

Without the fix, the nodejs execution will have a non-zero return code and an uncaught exception with the following line:
Error: error:25066067:DSO support routines:dlfcn_load:could not load the shared library

With the fix, there shouldn't be any output, and an exit code of 0.

[Where problems could occur]

There could easily be user errors when trying to merge the new configuration, for instance if they enabled the legacy provider, as they might comment out the default provider loading section (which is necessary if any other provider is explicitly loaded).

[Other Info]

Dear SRU team, please do not move this from -proposed to -updates before
the apt phasing fix has reached it first:
https://bugs.launchpad.net/charm-mysql-innodb-cluster/+bug/1979244

[Original report]
~ $ lsb-release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04 LTS
Release: 22.04
Codename: jammy

https://launchpad.net/debian/+source/openssl/3.0.3-7 includes a single change, https://sources.debian.org/src/openssl/3.0.3-8/debian/patches/Remove-the-provider-section.patch/

That patch solves a problem with programs that use OpenSSL v1 (statically or dynamically linked); these still read /etc/ssl/openssl.cnf, but the v3-specific sections in the sid/jammy default config may cause a failure.

One example: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1011051

Another example: a (non-Ubuntu) Node.js v16 (OpenSSL compiled statically) hits an error in its crypto lib:

~ $ node
Welcome to Node.js v16.15.0.
Type ".help" for more information.
> const { privateKey, publicKey } = crypto.generateKeyPairSync('rsa', { modulusLength: 2048 });

> var sign = crypto.createSign('RSA-SHA256')

> sign.update(Buffer.from("hello"))

> sign.sign(privateKey.export({type: 'pkcs1', format: 'pem'}))
Uncaught:
Error: error:25066067:DSO support routines:dlfcn_load:could not load the shared library
    at Sign.sign (node:internal/crypto/sig:131:29) {
  opensslErrorStack: [
    'error:0E076071:configuration file routines:module_run:unknown module name',
    'error:0E07506E:configuration file routines:module_load_dso:error loading dso',
    'error:25070067:DSO support routines:DSO_load:could not load the shared library'
  ],
  library: 'DSO support routines',
  function: 'dlfcn_load',
  reason: 'could not load the shared library',
  code: 'ERR_OSSL_DSO_COULD_NOT_LOAD_THE_SHARED_LIBRARY'
}

Removing the relevant provider section lines (the Debian patch doesn't apply cleanly, hence the use of sed) fixes it:

~ $ sed -i '/_sect\b/s/^/# /' /etc/ssl/openssl.cnf
~ $ node
Welcome to Node.js v16.15.0.
Type ".help" for more information.
> const { privateKey, publicKey } = crypto.generateKeyPairSync('rsa', { modulusLength: 2048 });

> var sign = crypto.createSign('RSA-SHA256')

> sign.update(Buffer.from("hello"))

> sign.sign(privateKey.export({type: 'pkcs1', format: 'pem'}))
<Buffer c5 e7 ba 01 5a 33 3f 26 43 bb 4e 47 99 49 e4 c7 60 41 be c6 91 63 c6 5d 0a af 78 5c 15 4a 9f 1a e7 24 99 ce 6a f0 05 b5 48 96 4e 59 b8 d5 69 df 3c bc ... 206 more bytes>

I realize there is no libssl1.1 on jammy, but a statically linked OpenSSL is not uncommon (Node.js being a very prominent example).

Would it be possible to get this Debian sid change ported to jammy?

Tags: fr-2496

CVE References

Revision history for this message
David Zuelke (dzuelke) wrote :

For reference, here is a related Node.js discussion (converted from an issue): https://github.com/nodejs/node/discussions/43184

Revision history for this message
Simon Chopin (schopin) wrote : Re: [Bug 1979639] [NEW] openssl 3.0.3-7 needs port from sid to jammy

Hi,

Thanks for this report, I was actually pondering whether or not to drop
this change when merging 3.0.4-1.

IMO this doesn't merit a SRU of its own as there's an easy workaround,
but it definitely should be bundled into the next one. Meanwhile it'll get
included in my upcoming upload to kinetic (today or tomorrow)

Simon Chopin (schopin)
tags: added: rls-jj-incoming
Revision history for this message
Simon Chopin (schopin) wrote : Re: openssl 3.0.3-7 needs port from sid to jammy

Oh, I just realized it actually affects one package in the archive: nodejs. Indeed, due to upstream needing ABI compatibility with libssl1.1 it was decided to use the embedded copy of openssl, see
https://lists.ubuntu.com/archives/ubuntu-devel/2021-October/041643.html and https://launchpad.net/ubuntu/+source/nodejs/12.22.7~dfsg-2ubuntu2

And indeed, my local, packaged copy of nodejs has the exact same behavior as the one described in OP.

Changed in nodejs (Ubuntu Jammy):
status: New → Confirmed
Changed in nodejs (Ubuntu Kinetic):
status: New → Confirmed
tags: added: fr-2496
Simon Chopin (schopin)
tags: removed: rls-jj-incoming
Simon Chopin (schopin)
Changed in nodejs (Ubuntu Jammy):
importance: Undecided → Medium
Changed in nodejs (Ubuntu Kinetic):
importance: Undecided → Medium
Revision history for this message
Simon Chopin (schopin) wrote :

There are a couple of issues here.

First, we should indeed patch the /etc openssl config to comment out the problematic section, especially since they are the same as the default.

Furthermore, we should also patch nodejs in Jammy *not to* look into this configuration file, as it'd be surprising for the user to have their nodejs programs fail with obscure errors when touching an unrelated config file, especially in ways that are presumably perfectly fine.

Changed in openssl (Ubuntu Jammy):
status: New → Confirmed
Changed in openssl (Ubuntu Kinetic):
status: New → Fix Committed
Changed in openssl (Ubuntu Jammy):
importance: Undecided → High
Changed in openssl (Ubuntu Kinetic):
importance: Undecided → High
Revision history for this message
David Zuelke (dzuelke) wrote :

I am not 100% sure about patching nodejs; using a config file is documented behavior:

- https://nodejs.org/api/cli.html#--openssl-configfile
- https://nodejs.org/api/cli.html#openssl_conffile

So if you do patch it this way, then at least the logic should be to e.g. call OPENSSL_no_config iff the env var OPENSSL_CONF is empty and no --openssl-config is given (in order to allow users to still explicitly pass a config).

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssl - 3.0.4-1ubuntu1

---------------
openssl (3.0.4-1ubuntu1) kinetic; urgency=medium

  * Merge from Debian unstable (LP: #1979639). Remaining changes:
    - Replace duplicate files in the doc directory with symlinks.
    - d/libssl3.postinst: Revert Debian deletion
      + Skip services restart & reboot notification if needrestart is in-use.
      + Bump version check to to 1.1.1.
      + Use a different priority for libssl1.1/restart-services depending
        on whether a desktop, or server dist-upgrade is being performed.
      + Import libraries/restart-without-asking template as used by above.
    - Add support for building with noudeb build profile.
    - Revert "Enable system default config to enforce TLS1.2 as a
      minimum" & "Increase default security level from 1 to 2".
    - Set OPENSSL_TLS_SECURITY_LEVEL=2 as compiled-in minimum security
      level. Change meaning of SECURITY_LEVEL=2 to prohibit TLS versions
      below 1.2 and update documentation. Previous default of 1, can be set
      by calling SSL_CTX_set_security_level(), SSL_set_security_level() or
      using ':@SECLEVEL=1' CipherString value in openssl.cfg.
    - Use perl:native in the autopkgtest for installability on i386.
    - d/p/skip_tls1.1_seclevel3_tests.patch: new Ubuntu-specific patch for the
      testsuite
    - d/p/Set-systemwide-default-settings-for-libssl-users: partially apply it
      on Ubuntu to make it easier for user to change security level
  * Dropped changes, merged upstream:
    - Add some more string comparison fixes
    - d/p/lp1947588.patch: Cherry-picked as our patches make it very easy to
      trigger the underlying bug
    - d/p/lp1978093/*: renew some expiring test certificates
  * d/p/fix-avx512-overflow.patch: Cherry-picked from upstream to fix a 3.0.4
    regression on AVX-512 capable CPUs.

openssl (3.0.4-1) unstable; urgency=medium

  * Import 3.0.3
    - CVE-2022-2068 (The c_rehash script allows command injection)

openssl (3.0.3-8) unstable; urgency=medium

  * Update to openssl-3.0 head.
  * Avoid reusing the init_lock for a different purpose (Closes: #1011339).

openssl (3.0.3-7) unstable; urgency=medium

  * Remove the provider section from the provided openssl.cnf
   (Closes: #1011051).

openssl (3.0.3-6) unstable; urgency=medium

  * Update to openssl-3.0 head which fixes the expired certs in the testsuite.

 -- Simon Chopin <email address hidden> Thu, 23 Jun 2022 12:43:23 +0200

Changed in openssl (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Revision history for this message
David Zuelke (dzuelke) wrote :

Thank you for that release, Simon.

Do you have a rough ETA on when this is going to hit jammy-updates?

Revision history for this message
Simon Chopin (schopin) wrote : Re: [Bug 1979639] Re: openssl 3.0.3-7 needs port from sid to jammy

I'll work on the Jammy SRU as soon as
https://bugs.launchpad.net/charm-mysql-innodb-cluster/+bug/1979244
is solved on 22.04, as I'd like to avoid recreating the issue. Julian
has uploaded a fix to Kinetic last week, so I'm guessing the SRU
shouldn't take too long.

Simon Chopin (schopin)
description: updated
Revision history for this message
Robie Basak (racb) wrote : Re: openssl 3.0.3-7 needs port from sid to jammy

> openssl 3.0.3-7 needs port from sid to jammy

This is presuming the solution, rather than describing the problem, so I'll amend this.

summary: - openssl 3.0.3-7 needs port from sid to jammy
+ Apps expecting an OpenSSL 1.1 -formatted openssl.cnf fail
Revision history for this message
Robie Basak (racb) wrote :

Thank you for working on this!

The proposed SRU carries quite a bit of regression risk I think, since there are many packages both in and out of the archive that may parse this file under various different circumstances. A regression that leads to a TLS failure, possibly across multiple packages at once, would be severe.

I wonder if it's acceptable to just fix nodejs in the archive to be able to handle this situation, on the basis that software (whether in the archive or not) needs to be able to be compatible with libssl3 if it expects to both examine the *system* openssl configuration and be functional on Ubuntu 22.04? This alternate approach would carry far less risk, and would avoid the conffile prompt (in cases where users have changed it). Why wouldn't this be acceptable?

If not, how long must we keep compatibility against libssl1.1 in our shipped openssl.cnf? What's the deprecation plan?

> either separately packaged (e.g. as an upgrade leftover)

These packages would be broken and not expected to work. Presumably they depended on libssl1.1 anyway. What would be the consequence of openssl declaring a Breaks against libssl1.1?

If we do end up proceeding with the approach you've put in the upload queue, then I think your Test Plan additionally needs:

* to verify that the default provider is indeed still being used by libssl3 (ie. the wrong configuration doesn't get used by accident)

* a basic smoke test of libssl3 to ensure that we aren't about to break the normal use of libssl3 for the entire archive

Changed in openssl (Ubuntu Jammy):
status: Confirmed → Incomplete
Revision history for this message
David Zuelke (dzuelke) wrote :

So I think a big class of programs to consider that are affected by this potentially are those that use a statically linked OpenSSL v1, not just Node.js or any leftover programs that dynamically link against libssl1.1.

And that's... a lot of software off the internet ;)

If it indeed is the case that OpenSSL falls back to the exact default behavior the current config specifies in its sections if those sections aren't there, then IMO the backport still makes a lot of sense.

Anyway... it appears as though Node.js have fixed this as a side-effect of their recent update: https://nodejs.org/en/blog/vulnerability/july-2022-security-releases/#dll-hijacking-on-windows-high-cve-2022-32223

I was able to verify that the latest 14.20 and 16.16 releases work fine.

Revision history for this message
Robie Basak (racb) wrote : Re: [Bug 1979639] Re: Apps expecting an OpenSSL 1.1 -formatted openssl.cnf fail

On Thu, Jul 14, 2022 at 03:56:04PM -0000, David Zuelke wrote:
> So I think a big class of programs to consider that are affected by this
> potentially are those that use a statically linked OpenSSL v1, not just
> Node.js or any leftover programs that dynamically link against
> libssl1.1.
>
> And that's... a lot of software off the internet ;)

This is true. Maybe we need to be pragmatic about this type of case. I'd
like to see some data on what's actually affected though.

Regardless, such programs are buggy. They're not properly built if they
have a dependency on the system /etc by assuming libssl1.1 is available
on the system like this. This is a perfect demonstration of how merely
asking the linker to link statically does not result in a properly
independent binary.

In our ecosystem it's pretty unusual for there to be an expectation that
binaries will continue working following distribution release upgrades
like this, and if they don't, usually the third party distributor is
expected to produce a fixed binary. And if we do choose to maintain
compatibility, then this stifles improvements. What's the time limit for
backwards incompatibility like this? Usually a major distribution
release upgrade is the appropriate time to make these kinds of breaking
changes.

I have no issue with pragmatically arranging the configuration to be
backwards compatible for a while. But the time to do this is before
release. Doing it after is much riskier, and so we should be more
cautious. The people affected who want the change are not the people who
would be affected and be forced to deal with any fallout.

So how big is this impact really? That's why I'd like to see the
evidence.

Revision history for this message
Simon Chopin (schopin) wrote :

Many thanks for raising this :)

I was prepared to argue for this change as (if working as intended) it has basically no impact on our own applications and could help quite a few unsuspecting users, but since you asked for evidence I did a little digging, and was somewhat surprised by what I found. While it is true that there are quite a bit of software out there that embeds older versions of OpenSSL, NodeJS are the outliers in pointing to /etc/ssl as their OPENSSLDIR. In fact, even our own package points to /usr/lib/ssl, with symlinks to /etc for the config file and the certificates.

I erroneously thought that the OpenSSL upstream default was also /etc/ssl, but as it turns out they're using the more sensible /usr/local/ssl as a default.

This limits the impact to

1/ third-party packages depending on our old libssl1.1,
2/ our nodejs package
3/ upstream NodeJS builds

I'm not particularly keen on doing this SRU just for the sake of 1/, and 2/ can be fixed in that package, but do we want to break things for 3/ ?

I don't know much about the JS ecosystem. My first Google result for `installing nodejs on Ubuntu` was pretty quick to point me towards `nvm` to "easily install different versions of Node", and that tool seems to download upstream binaries directly.

I'm OK either way, and I've already started working on extending the test case as per your requests.

Simon Chopin (schopin)
description: updated
Revision history for this message
Robie Basak (racb) wrote :

> ...it has basically no impact on our own applications...

Just because you and I can't find a case that would break doesn't mean there isn't one. So there's risk. And I think the consequences, because of the nature of this package, could be quite large.

> I'm not particularly keen on doing this SRU just for the sake of 1/, and 2/ can be fixed in that package, but do we want to break things for 3/ ?

We'll have to break it at some point, right? Otherwise the configuration file format in /etc could never move on. A new release would be the appropriate time, as has happened in Jammy. So now seems a good a time as any, given that it's already happened, and it requires effort and risk to reverse it.

Upstream NodeJS should be able to release a fixed version themselves. In fact I believe it's the norm for external software sources to have to adjust themselves for new distribution releases anyway. Is there any reason this isn't acceptable to them?

Revision history for this message
David Zuelke (dzuelke) wrote :

It's fixed upstream in Node.js already; versions 14.20 and 16.16 (18.x was never affected).

I'm now inclined to agree this shouldn't be ported from Debian (and reverted in Kinetic).

Revision history for this message
David Zuelke (dzuelke) wrote :

(older Node.js versions didn't get updates, but they're all EOL).

Revision history for this message
Robie Basak (racb) wrote :

Thanks everyone for the discussion. As there have been no further comments, I think it's appropriate to make a decision. I'll reject the openssl upload from Jammy, and mark that task Won't Fix. We won't update the default configuration in Jammy. Third parties who ship software intended for Jammy need to adapt to be able to read OpenSSL 3's configuration format if they choose to read the system configuration. If this is a problem with the nodejs package in Ubuntu, then that package should be patched, and so those bug tasks remain open.

We can look again at this decision if new information is presented.

Changed in openssl (Ubuntu Jammy):
status: Incomplete → Won't Fix
Revision history for this message
Robie Basak (racb) wrote : Proposed package upload rejected

An upload of openssl to jammy-proposed has been rejected from the upload queue for the following reason: "See discussion in LP: #1979639".

Revision history for this message
David Zuelke (dzuelke) wrote :

Does this mean the fix will remain in Kinetic, though?

Should it not be reverted there, then?

Revision history for this message
Robie Basak (racb) wrote :

That's a good question. I think it's up to Simon and the Foundations team what they want to do there. There is far less of a risk to users either way in Kinetic, because it hasn't been released yet. So I have no preference for Kinetic, though it does make sense to revert to me.

Revision history for this message
Simon Chopin (schopin) wrote : Re: [Bug 1979639] Re: Apps expecting an OpenSSL 1.1 -formatted openssl.cnf fail

Hi!

I apologize, I thought I'd sent here the gameplan for the issue, but it
must have only been discussed out of band within the team. As noted by
Robie, the consensus is not to change the default configuration, but
instead patching node.js on Jammy (patches welcome for that one though).

I've already reverted the change in kinetic in my last upload (3.0.5-2ubuntu1)

Thanks for the follow-up, Robie!

Revision history for this message
Utkarsh Gupta (utkarsh) wrote :

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.

Changed in nodejs (Ubuntu Kinetic):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.