OpenSSL 3.0.2 crash in Ubuntu 22.04.2 LTS

[Michael Baentsch]

Full bug report at https://github.com/openssl/openssl/issues/20981

No upstream impact: OpenSSL 3.0.9-dev does not contain the problem any more.

[Adrien Nader (adrien)]

Hi,

Thank you for taking the time to report this issue and providing a reproducer. Unfortunately I have not succeeded in reproducing the issue. In a fresh jammy container, using "OPENSSL_BRANCH=openssl-3.0.3 scripts/fullbuild.sh", I then ran "ln -s oqsprovider.so _build/lib/oqsprovider2.so" which gave me the layout below:

    root@jammy:~/oqs-provider# ls -l _build/lib/
    total 6472
    lrwxrwxrwx 1 root root 14 May 17 15:39 oqsprovider2.so -> oqsprovider.so
    lrwxrwxrwx 1 root root 16 May 17 15:32 oqsprovider.so -> oqsprovider.so.1
    -rwxr-xr-x 1 root root 6625696 May 17 15:32 oqsprovider.so.0.5.0-dev
    lrwxrwxrwx 1 root root 24 May 17 15:32 oqsprovider.so.1 -> oqsprovider.so.0.5.0-dev

and then

    root@jammy:~/oqs-provider# OPENSSL_MODULES=_build/lib/ OPENSSL_CONF=scripts/o-ca.cnf ./.local/bin/openssl version
    OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

which didn't crash.

[Michael Baentsch (baentsch)]

Thanks for trying to reproduce this. A crash can only be triggered with two providers active. Your command "OPENSSL_MODULES=_build/lib/ OPENSSL_CONF=scripts/o-ca.cnf ./.local/bin/openssl version" is not quite conclusive: The config file "o-ca.cnf" doesn't look right. Please verify that your setup confirms two instances of the provider being active. You can do this with the option "list -providers": This is what must be displayed (also in the test run) to trigger/showcase the problem:

OPENSSL_MODULES=_build/lib/ OPENSSL_CONF=scripts/openssl-ca.cnf LD_LIBRARY_PATH=.local/lib64 ./.local/bin/openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.2.0
    status: active
  oqsprovider
    name: OpenSSL OQS Provider
    version: 0.5.0-dev
    status: active
  oqsprovider2
    name: OpenSSL OQS Provider
    version: 0.5.0-dev
    status: active

[Adrien Nader (adrien)]

OK, thanks. I had to edit the config file which I had used almost blindly.

I'm not getting a crash however:

ubuntu@jammy:~/openssl-provider/oqs-provider$ OPENSSL_MODULES=$(pwd)/_build/lib/ OPENSSL_CONF=$(pwd)/scripts/o-ca.cnf ./.local/bin/openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.0.2
    status: active
  oqsprovider
    name: OpenSSL OQS Provider
    version: 0.5.0-dev
    status: active
  oqsprovider2
    name: OpenSSL OQS Provider
    version: 0.5.0-dev
    status: active
ubuntu@jammy:~/openssl-provider/oqs-provider$ OPENSSL_MODULES=$(pwd)/_build/lib/ OPENSSL_CONF=$(pwd)/scripts/o-ca.cnf ./.local/bin/openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

Taking some steps back, you mentioned that people are unlikely to encounter the issue and openssl is a component we have been more wary to update in past releases due to breakage across minor versions. As such, without more users being impacted, it is unlikely we push an update to 22.04 for this. The bug report is good to have though since it makes it easier for others to chime in.

PS: I don't know if it's time to re-visit that practice of not updating through minor openssl versions; it's risky to try.

[Michael Baentsch (baentsch)]

Fine for me -- I'm working around this (not using Ubuntu for CI any more and/or using/building less buggy OpenSSL releases for CI).

> if it's time to re-visit that practice of not updating through minor openssl versions; it's risky to try.

What risks do you see? I find it much more risky _not_ to do it: You'll retain buggy versions that are possibly also security risks: I'm counting 24 CVEs that Ubuntu with this policy willingly does not fix -- as an Ubuntu user I wouldn't be happy.... (see https://www.openssl.org/news/vulnerabilities-3.0.html)

[Matt Caswell (matt-openssl)]

> if it's time to re-visit that practice of not updating through minor openssl versions; it's risky to try.

As an upstream OpenSSL maintainer we try very hard to ensure that stable releases remain stable across patch releases. We only allow bug fixes (no new features etc) into our patch releases and always seek to ensure backwards compatibility. Of course every bug fix is ultimately a change of behaviour and sometimes end users rely on that buggy behaviour. Sometimes we get it wrong and inadvertently break something. I hope those are few and far between though.

Ultimately if you fix bugs there will always be a residual risk that you break something somewhere. Hopefully the risk is small enough that it is acceptable.

[Seth Arnold (seth-arnold)]

Michael, Ubuntu backports specific fixes as they are identified; you can check the status of our OpenSSL packages on our website:

All OpenSSL issues:

https://ubuntu.com/security/cves?q=&package=openssl&priority=&version=&status=

OpenSSL issues, restricted to just Jammy:

https://ubuntu.com/security/cves?q=&package=openssl&priority=&version=jammy&status=

OpenSSL issues without a release, restricted to just Jammy:

https://ubuntu.com/security/cves?q=&package=openssl&priority=&version=jammy&status=needed
https://ubuntu.com/security/cves?q=&package=openssl&priority=&version=jammy&status=deferred
https://ubuntu.com/security/cves?q=&package=openssl&priority=&version=jammy&status=ignored
https://ubuntu.com/security/cves?q=&package=openssl&priority=&version=jammy&status=needs-triage

(Yeah, it's unfortunate that currently takes multiple pages to view. Hopefully this will be addressed soon.)

Thanks

[Michael Baentsch (baentsch)]

Thanks for this explanation, Seth. Very good to know--from a security perspective. A bit less satisfying from a functionality perspective, particularly given the assurance by Matt from the OpenSSL team above.

I do understand though that you're aiming for "bug completeness" to support those that rely on those bugs. The others then need to work around them (using other distros or building packages from source).

Please feel free to close this report then.

[Seth Arnold (seth-arnold)]

It's less about bug completeness and more about the risks of breaking users. The general rule for the whole distribution is backporting specific fixes for specific bugs; however, there's a handful of packages where that's not feasible, desired, etc.

Firefox and Chromium are the most obvious cases of wanting the same, recent, version, on all supported releases. The upstream developers for these projects have way more resources and way more comprehensive test suites than we could ever hope to achieve ourselves, and they've got loads of experience making frequent releases.

MySQL, MariaDB, PostgreSQL are common for "the most recent version of the release that was used at the time of release" (or something like that; it's an ugly mouthful). Moving from MySQL 5.5 to 8 would be a huge jump, but 5.5.32 to 5.5.36 to 5.5.40 etc shouldn't be a big deal. (Alas, it is. LP:2019203.) These are also far more complex than we can realistically engineer ourselves.

We've done full-version jumps with Samba before; some of their security fixes involve hundreds of patches with huge refactoring. There's no good choices with Samba. The risks of backporting are huge, the opportunity costs are even larger, and if we backport that much, we'll wind up with software that nobody is familiar with. So, we will sometimes ship entirely new versions, and just deal with all the fallout from regressions.

OpenSSL is a challenging case. It'd be ideal to run the same version as upstream, so when there's issues, there's a much larger community working on them. Perhaps the OpenSSL upstream developers have an extensive enough test suite today to reduce the risks of using entirely new versions. I know that historically we've found some issues with security patches via our testing that the OpenSSL upstream testing missed. I also know that our testing is focused on what ships in our distribution, it doesn't test the wide world of propriety software or not-yet-packaged software, so we know we have blind spots.

If we explore shipping upstream OpenSSL packages, I'd really like to see it trialed in our 'interim' releases: eg, ship an OpenSSL update halfway through the support cycle for 23.04, 23.10, 24.10, 25.04, 25.10, and if those all go well, consider it for 26.04 and future releases. There's way fewer users of our interim releases (which is both a benefit and a curse, here) and the consequences of a problem are thus constrained.

Users expect (if not appreciate) breaking changes at release points. They don't expect (and detest) such changes in LTS releases.

Given the foundational importance of OpenSSL, I think it makes sense to go very slowly in testing such a hypothetical change.

Thanks

[Michael Baentsch (baentsch)]

Thanks very much for this complete answer and apologies for using the term "bug completeness": I meant "feature stability" (with bugs being on the arguably negative side of that term -- and this discussion being one regarding a bug).

Clearly there's more users (valuing stability) than developers (valuing "progress" -- with known risks).

I personally think the OpenSSL team goes to great lengths to not break things with new sub-sub-version releases but I can appreciate that not all projects do the same and you don't want to annoy your users.

Indeed your distribution is one of the first to have integrated OpenSSL3 -- which gave me the ability to integrate quantum safe crypto and thus make available that capability (in 3.0.2 only for KEMs, in future releases also for signatures/certificates) to Ubuntu users: So, a big Thanks to you for having taken that step -- with regard to quantum security possibly still many, many years before it's a true necessity.

[Seth Arnold (seth-arnold)]

That's awesome! I figure post-quantum cryptography will feel like something for years in the future until the week when we all realize we should have moved years earlier. Capture-and-store has to be going on right now, on the assumptions that someone will bring a reliable quantum machine to market. Thanks for working on it :)

[Launchpad Janitor (janitor)]

[Expired for openssl (Ubuntu) because there has been no activity for 60 days.]