backport fix for "OpenSSL 3 cannot decrypt data encrypted with OpenSSL 1.1 with blowfish in OFB or CFB modes" to Jammy

[Nathan Stratton Treadway]

=== SRU information ===
[Meta]
(This bug was once the fourth in a series of bugs grouped for a single SRU, with
the "central" bug with the global information and debdiff being http://pad.lv/2033422 . However, this particular bug is no longer included in that SRU effort.)

[Impact]
Decryption for Blowfish with OFB and CFB modes fails due to using a key shorter than expected by default.
Encryption will also use a key shorter than expected.
Exchange of encrypted data from/to Jammy using BF OFB/CFB will therefore lead to decryption issues.

[Test plan]
On Focal, run the following and copy the output to your clipboard

    for cipher in bf-cbc bf-cfb bf-ecb bf-ofb; do echo "Test with ${cipher}" | openssl enc -${cipher} -k test -pbkdf2 -out "pouet.${cipher}"; done
    tar c pouet.bf-* | xz | base64 -w 60

You can also run this on Lunar or Mantic if you add "-provider legacy -provider default" to the "openssl enc" invocation.

On Jammy, run the following and paste your clipboard

    base64 -d | xz -d | tar x
    for cipher in bf-cbc bf-cfb bf-ecb bf-ofb; do openssl enc -d -provider legacy -provider default -${cipher} -k test -pbkdf2 -d -in "pouet.${cipher}"; done

Only "Test with bf-cbc" and "Test with bf-ecb" will be properly decrypted: the other two will result in garbage on screen.

Here is the result of the enc + tar + xz + base64 on Focal (works with Lunar/Mantic too but you need to added ):

    /Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4Cf/ARBdADgbyxDlZ/1Xd7bAmZw7
    8pbqQTu5j8StVybo1p1B2ydBc5VcodF6fu0hEp801tvirgSFNMSAHk5HMN/w
    hCgU1BIr/nK51g3A3Lkdv7QNbaUw2ux1AmO/MpCLKLffCB9ElFZH4tuOS5AR
    m9CJMzi6LQOw9wytGKm2IK3Ph7WpU6JQ/3HJilffQwHbFLnukiWGpLNO5v0O
    D/4AJikrU9iemfChT0jXDbIRZ8a8VpVhJqu0u6eYOheVTqmSRiHHpIC/p1VA
    ecFb0mACF/TQhjxcMUWGSGO/mtof+VaLiyg0KB87GKlChfwXTEvgbNuP9hmu
    GL64VhX568Oy9EakSxlcXiIRk14kJKv0MdHQqY1R22wAACzqSr/nzpwqAAGs
    AoBQAACjzq5WscRn+wIAAAAABFla

Here is the same but from Jammy if you want to test encryption on Jammy and decryption on Lunar/Mantic:

    /Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4Cf/ARFdADgbyxDlZ/1Xd7bAmZw7
    8pbqQTu5j8StVybo1p1B2ydBc1zK4HR2g3CiLJet+R++nZy/gph6RscQ6hI3
    HySjdDOFRfjIVttiNK3DvRsZb37r8SXkj/JCYWicZGjWPZxVE3OAZhEed5qe
    jrFv871QAbm4jVGD4oIc4cOb5V/xDN7KWgwEzpWQy6+tcfPm3KLPQvULx56N
    2qQf60hP//p5EXS3RpCitUsrGUoYzTynjOUIRy2yCmgZDh62RmchUshyWePa
    k0nEYlDbl5/dSHXbWEWESqW+QDj136MZRwQRY+QC4MvLXg2Bo8H+Dl/xvNDF
    /5J4layZdFlh76lWOtFRVoIbX6JtpAP34g4zx1422GSNAAAAAABRzyqPdCqX
    1AABrQKAUAAABh3ynbHEZ/sCAAAAAARZWg==

The contents are expected to be different due to the use of randomness. Don't try to compare the base64 outputs: I'm only using them to ease testing across containers.

[Where problems could occur]
This patch makes openssl match the documented default (see "man openssl-enc" and search for "Blowfish" for instance) and fixes decryption from an up-to-date Jammy to pretty much everything else, but it also create an issue for data encrypted on Jammy without this patch and Jammy with this patch.

There are two possible cases: encrypted data being streamed across this boundary or data at rest being transferred or read later.

Streaming is probably not an issue in practice because it's rather the current situation that has been an issue and it's easy to remedy by updating everything (which is relatively few machines since that's only Jammy and not any other OS or distribution).

Data at rest is more annoying since updating Jammy will make it impossible to read the data again without updates to other pieces of software. That sounds like a really bad thing and it kind of is but at the same, the benefits are much larger than the issues. Indeed, there is already an incompatibility at the moment between Jammy and everything else and the more time passes by, the more such problematic files can be created. Luckily very few people are using blowfish nowadays and it's not even enabled by default anymore in openssl. Moreover the software update to work around the issue should be a single API call which is documented in the upstream bug report ( https://github.com/openssl/openssl/issues/18359 ). Finally, I have warned the two projects that I am aware are impacted; this is made easier by the fact that they encountered the initial incompatibility.

[Patches]
The patches come directly from upstream and apply cleanly.

https://github.com/openssl/openssl/issues/18359

* https://git.launchpad.net/~adrien-n/ubuntu/+source/openssl/tree/debian/patches/jammy-sru-0001-Fix-regression-in-default-key-length-for-Blowfish-CF.patch?h=jammy-sru&id=04ef023920ab08fba214817523fba897527dfff0
* https://git.launchpad.net/~adrien-n/ubuntu/+source/openssl/tree/debian/patches/jammy-sru-0002-Test-the-default-key-length-of-the-Blowfish-ciphers.patch?h=jammy-sru&id=04ef023920ab08fba214817523fba897527dfff0

=== Original description ===

OpenSSL upstream implemented a fix for their issue #18359 "OpenSSL 3 cannot decrypt data encrypted with OpenSSL 1.1 with blowfish in OFB or CFB modes"
  https://github.com/openssl/openssl/issues/18359
as of libssl3 3.0.4 (and thus it is included in recent libssl3 versions in Kinetic).

Could this fix be backported to libssl3 in Jammy?

[Fredrik Wendt (fredrik-wendt)]

If I read upstream correctly, a fix for this regression was put in master over a year ago (May 2022). Any update to when this will be rolled out on Jammy/current LTS?

[Adrien Nader (adrien)]

I plan to prepare a batch with several fixes late August/September.

[Adrien Nader (adrien)]

I've created a PPA for Jammy that incorporates the fix mentionned. The details are available at https://launchpad.net/~adrien-n/+archive/ubuntu/openssl-jammy-sru . Could you test it and confirm your issue is solved?

[Nathan Stratton Treadway (nathanst)]

Yes, libssl3 3.0.2-0ubuntu1.11~ppa2 appears to fix the Blowfish incompatibility (at least for the original case of connecting to Tinc running on old distributions of Ubuntu).

Test steps:
disabled the OPENSSL_MODULES workaround I had in place on a my Jammy node, and confirmed that Tinc was unable to connect to the remote Tinc node, and generated "Bogus data received from" syslog error messages (as expected due to this bug).

enabled the adrien-n/openssl-jammy-sru PPA, and installed the ...ubuntu1.11~ppa2 versions of openssl and libssl3.

restarted the Tinc service... and saw that the Jammy node was once again able to connect to the remote node (without the OPENSSL_MODULES workaround).

Let me know if I can provide any other information.

(Do you have an ETA for publishing this package to Jammy?)

Thanks.

[Adrien Nader (adrien)]

Thanks a lot for taking the time to test and provide feedback.

I'll continue with the SRU process which should take a few more weeks (I'd say between two and four but that's a very rough guess).

[Sergio Durigan Junior (sergiodj)]

Hello,

ubuntu-sponsors is subscribed to this bug but I couldn't find anything actionable. I'm unsubscribing ubuntu-sponsors; feel free to subscribe it again if there's anything that needs sponsoring. Thanks.

[Sergio Durigan Junior (sergiodj)]

Ah, I noticed that this is part of a big SRU that's being completed on bug #2033422. Just leaving a comment here for the record.

[Simon Chopin (schopin)]

I'm very much against uploading this fix as an SRU. While it is a shame that we have interop problems with the rest of the world, fixing this in Jammy right now means we *will* break production for anyone that stores Blowfish-encrypted data.

Doing `apt upgrade` on a production server is supposed to be a fairly routine affair, whereas running into the interop issue means someone somewhere did a dist-upgrade, changed some configuration, wrote new code, or installed a new system, all of which are actions where the user *should* expect possible breakage.

[Nathan Stratton Treadway (nathanst)]

On Thu, Oct 26, 2023 at 09:54:51AM -0000, Simon Chopin wrote:
> I'm very much against uploading this fix as an SRU. While it is a shame
> that we have interop problems with the rest of the world, fixing this in
> Jammy right now means we *will* break production for anyone that stores
> Blowfish-encrypted data.

Simon, just to clarify: the original issue I ran into was not
"interop problems with the rest of the world" exactly, but rather
interop between Tinc nodes running on different versions of
Ubuntu (i.e. a Tinc node running Jammy can't connect to existing
nodes running on older releases). (Though obviously interop will
fail with the rest of the world, too.)

I am not particularly involved in the OpenSSL community, but have
been trying to keep an eye on this specific issue since I ran
into it, and as far as I have noticed/found on the internet there
haven't been any mentions about this OpenSSL bug other than in
the two LP bugs related to this Tinc issue and the related
Github-for-tinc bug ( https://github.com/gsliepen/tinc/issues/414
).

So while I certainly agree in general with need for "apt upgrade"
to be a very safe operation, in this particular case I am not
sure how to weigh the chance that someone might possibly have a
custom app using Blowfish to encrypt local data (while at the
same time not already having in place mechanisms to deal with
this compatibility issue for pre-and-post Jammy Ubuntu) with the
publicly-raised-by-at-least-a-few-people problem of Tinc
incompatibility between nodes that this causes....

                                                Nathan

[Nathan Stratton Treadway (nathanst)]

On Tue, Oct 31, 2023 at 01:13:11PM -0000, Adrien Nader wrote:
> ** Description changed:
>
> === SRU information ===
> [Meta]
> - This bug is part of a series of four bugs for a single SRU.
> + This bug is part of a series of three bugs for a single SRU.
> The "central" bug with the global information and debdiff is http://pad.lv/2033422
>
> [Impact]
> Decryption for Blowfish with OFB and CFB modes fails due to using a key shorter than expected by default.

Adrien, am I correct that this particular bug (1990216) is actually no longer
included at all in the LP:#2033422 SRU ?

                                                        Nathan

[Adrien Nader (adrien)]

Hi Nathan,

Sorry, I didn't have enough time to comment here before a few days of vacation.

This one is indeed not in the SRU at the moment. The description edit itself did not make much sense.

I first discussed this topic with Simon but then also with Steve Langasek, with others attending the same meeting. The general agreement is that bugs when setting up something for the first time are far less severe than bugs that appear afterwards. One major issue here is everything that exists but that we don't know of, including custom software or scripts.

As far as I'm concerned, I evaluate this roughly like you did but I cannot do something that the SRU team is against (I also trust them and their experience, even when my feeling is different).

Lastly, 22.04 was released more than 18 months ago and 24.04 is around the corner; 18 months is a long delay to introduce breaking changes and by now people probably expect very few changes to 22.04.

As far as I understand, tinc could fairly easily work around this issue by explicitely setting the key size before doing operations. This is the safest approach. It might even be faster than waiting for the SRU and corresponding phased updates.

[Nathan Stratton Treadway (nathanst)]

On Wed, Nov 01, 2023 at 02:39:27PM -0000, Adrien Nader wrote:
> This one is indeed not in the SRU at the moment. The description edit
> itself did not make much sense.

(Okay, that's what I thought. For what it's worth, I noticed afterwards
that the description for LP: #2033422 still has the "four bugs" version
of that sentence.)

> As far as I understand, tinc could fairly easily work around this issue
> by explicitely setting the key size before doing operations. This is the
> safest approach. It might even be faster than waiting for the SRU and
> corresponding phased updates.

Actually this isn't really easy on the Tinc side (at least for
Ubuntu end users).

In particular, the Tinc package actually hasn't changed at all in
several Ubuntu releases, so the big surprise here was that Tinc
1.0.36 on Jammy fails to connect in a network where Tinc 1.0.36
on Focal worked fine (even after making the configuration
adjustments that were expected due to the openssl 1.1 -> 3
transition).

Tinc does not provide any user-level control on these parameters,
and I don't think it even sets the parameters explicitly
internally in the code; rather, it relies on the default settings
for the algorithm, which is what was broken in that brief range
of OpenSSL v3 upstream releases. (Presumably if it already set
the parameters explicitly then it wouldn't have been affected by
this bug.)

In any case, solving this from the Tinc side would involve
(someone) making a Ubuntu-specific customization to the Tinc
source code -- which seems a lot more fraught than simply using a
fixed libssl3 library...

One thought that occurred to me is the idea of a semi-official
PPA build with this Blowfish patch included in it.
"Semi-official" in that you (or someone involved in the OpenSSL
Ubuntu packaging work) would publish it, building on top of the
official Ubuntu package, and thus it would track any other fixes
that were published to the main OpenSSL Jammy package -- but
since it's in a PPA it would be opt-in for those who need it.

This would be certainly be preferable to me over my current
situation using an "orphan" version of the legacy.so library
file, but I don't know how much effort it would be to provide
that on your end...

                                                Nathan

[Adrien Nader (adrien)]

Apologies for not answering earlier; I wasn't available when I first saw your message.

FWIW, there's just been another report of the same issue with a different scenario but that's half-way between the "streaming" case and the "data at rest" one.

The reason this fix is difficult to integrate in a stable release is because while we know we would introduce breakage, we do not and cannot know how much. Imagine even 100 Jammy machines which can talk together today; that's quite expected because they're on the same release. This change would break that unless every machine is upgraded at once which is not the case by default due to phasing of updates (and we want to phase openssl updates slowly because it's a very central component).

While I really sympathetize with your issue, this is the stance of the Ubuntu project. Both including the fix and not including the fix are problematic unfortunately.

One additional reason to not include the fix is that we're now close to Ubuntu 24.04. I realize you've reported this over a year ago but analysis, preparing a new verion, and the SRU process all take time. This is especially true for openssl which is central and which updates have caused issues several times.

Ultimately I'd like to be able to keep Ubuntu releases updated with the latest openssl versions (no jump from 3.x to 3.x+1 though, merely 3.x.y to 3.x.y+1). This is not done at the moment because incompatibilities and issues in openssl are often found late but are also very painful. In order to do these, I will have to apply the same analysis and criteria to every change in the openssl releases to ensure package upgrades are safe and uneventful. Had this been in place for 22.04, this change would likely have been integrated when it was released in June because that was very close to the initial 22.04 release. If we want this to happen, we need to draw the line somewhere and stick to it. Unfortunately this change is on the other side of the line

That's not to say that we cannot do anything for tinc or for others affected packages but rather that it won't be done in openssl.

I see two ways to improve this, tinc side.

1) Switch to another cipher. Blowfish uses a 64-bit block size which is small and limits how much data can be safely encrypted with the same key ( https://en.wikipedia.org/wiki/Blowfish_(cipher)#Weakness_and_successors ). I guess this requires cooperation from the server which you might not control but it is the best long-term solution (and would also help performance because computing a MAC on top of BF is surprisingly expensive, and re-keying every n GB stalls data transfers and incurs a spike in latency).

2) Modify tinc because there's apparently a portable work-around as I've mentioned in https://github.com/gsliepen/tinc/issues/414#issuecomment-1741038601 . I think there's more context on the github openssl bug tracker. My time is scarce until the end of the year but I will gladly help with the packaging if needed. I don't know if it could be uploaded because there would be the same compatibility concern, this time with servers on 22.04. A PPA might be more appropriate and not too inconvenient since tinc development has stall...

[Nathan Stratton Treadway (nathanst)]

On Fri, Nov 24, 2023 at 12:04:59PM -0000, Adrien Nader wrote:
> FWIW, there's just been another report of the same issue with a
> different scenario but that's half-way between the "streaming" case and
> the "data at rest" one.

Is this report you mention an LP bug? I look through the bug list for
the OpenSSL package but didn't immediately see a bug that seemed related
to this.

> The reason this fix is difficult to integrate in a stable release is
> because while we know we would introduce breakage, we do not and cannot

Yes, understood.

> I see two ways to improve this, tinc side.
>
> 1) Switch to another cipher. Blowfish uses a 64-bit block size which is
> small and limits how much data can be safely encrypted with the same key
> (
> https://en.wikipedia.org/wiki/Blowfish_(cipher)#Weakness_and_successors
> ). I guess this requires cooperation from the server which you might not
> control but it is the best long-term solution (and would also help
[...]
> 2) Modify tinc because there's apparently a portable work-around as I've

Blowfish is only needed for backward compatibility with now-obsolete
versions of Tinc, so I don't think making changes on the Tinc side is
feasible/helpful. (More-recent versions of Tinc have switched to using
a newer algorythm.)

What would be helpful from a Tinc-network-administrator standpoint is
some easier way to install a fixed version of the libssl3 legacy.so
file, though I am not sure what the best way to get there would be.

(I would imagine having such an option could also be helpful to any
users who have custom applications affected by this Blowfish bug as they
prepare to upgrade from Jammy to later releases of Ubuntu.)

                                                        Nathan

[Jeremy Sowden (a3a3el)]

On 2023-11-24, at 17:25:11 -0000, Nathan Stratton Treadway wrote:
> On Fri, Nov 24, 2023 at 12:04:59PM -0000, Adrien Nader wrote:
> > FWIW, there's just been another report of the same issue with a
> > different scenario but that's half-way between the "streaming" case and
> > the "data at rest" one.
>
> Is this report you mention an LP bug? I look through the bug list for
> the OpenSSL package but didn't immediately see a bug that seemed related
> to this.

I created this report a couple of days ago:

  https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2044391

J.

> > The reason this fix is difficult to integrate in a stable release is
> > because while we know we would introduce breakage, we do not and cannot
>
> Yes, understood.
>
> > I see two ways to improve this, tinc side.
> >
> > 1) Switch to another cipher. Blowfish uses a 64-bit block size which is
> > small and limits how much data can be safely encrypted with the same key
> > (
> > https://en.wikipedia.org/wiki/Blowfish_(cipher)#Weakness_and_successors
> > ). I guess this requires cooperation from the server which you might not
> > control but it is the best long-term solution (and would also help
> [...]
> > 2) Modify tinc because there's apparently a portable work-around as I've
>
> Blowfish is only needed for backward compatibility with now-obsolete
> versions of Tinc, so I don't think making changes on the Tinc side is
> feasible/helpful. (More-recent versions of Tinc have switched to using
> a newer algorythm.)
>
> What would be helpful from a Tinc-network-administrator standpoint is
> some easier way to install a fixed version of the libssl3 legacy.so
> file, though I am not sure what the best way to get there would be.
>
> (I would imagine having such an option could also be helpful to any
> users who have custom applications affected by this Blowfish bug as they
> prepare to upgrade from Jammy to later releases of Ubuntu.)

[Nathan Stratton Treadway (nathanst)]

On Fri, Nov 24, 2023 at 05:39:24PM -0000, Jeremy Sowden wrote:
> On 2023-11-24, at 17:25:11 -0000, Nathan Stratton Treadway wrote:
> > On Fri, Nov 24, 2023 at 12:04:59PM -0000, Adrien Nader wrote:
> > > FWIW, there's just been another report of the same issue with a
> > > different scenario but that's half-way between the "streaming" case and
> > > the "data at rest" one.
> >
> > Is this report you mention an LP bug? I look through the bug list for
> > the OpenSSL package but didn't immediately see a bug that seemed related
> > to this.
>
> I created this report a couple of days ago:
>
> https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2044391
>
> J.

Ah, thanks, got it. (I see now that it's not included in the
default listing for OpenSSL bugs that I was looking through since
it's already been marked as a duplicate.)

(Am I correct in understanding that if you didn't implement a fixed
version of Blowfish on your Jammy systems, your application
would also have the same trouble migrating from Jammy to the next
generation of Linux distribution that you chose in the future?)

In any case, it sounds like it could help your site, too, if
Ubuntu somehow provided an alternative legacy.so file with the
Blowfish fix already included, thus saving you from having to
maintain your own such package?

                                                Nathan

[Adrien Nader (adrien)]

Sometimes I don't understand what happens when I attempt to reply by mail...

Anyway...
The affected code is in libcrypto which I think sees fewer important security fixes. Therefore it's possible to build it and put it in your library search path. This should fix the issue without being too terrible wrt security updates.

I still think that setting the key size in the caller program is better if possible however. It's slightly more involved upfront though (but it could pay up in the end).

I'll prepare a PPA on tomorrow with only this change on tomorrow; I will probably add a few things to prevent using the package directly however since I expect the proper approach would be to 'dpkg -x' the package or equivalent and use its libcrypto.so and LD_LIBRARY_PATH or similar specifically when needed.

[Nathan Stratton Treadway (nathanst)]

On Mon, Dec 04, 2023 at 04:42:48PM -0000, Adrien Nader wrote:
> The affected code is in libcrypto which I think sees fewer important security
> fixes. Therefore it's possible to build it and put it in your library search
> path. This should fix the issue without being too terrible wrt security
> updates.
[...]
> I'll prepare a PPA on tomorrow with only this change on tomorrow; I will
> probably add a few things to prevent using the package directly however
> since I expect the proper approach would be to 'dpkg -x' the package or
> equivalent and use its libcrypto.so and LD_LIBRARY_PATH or similar
> specifically when needed.

Note that as far as I understand from my own testing, the library file
that is affected by this particular upstream bugfix (OpenSSL
issue #18359/PR #18362) is ossl-modules/legacy.so ...

... and as such it is possible to work around the bug in the
systemwide libssl3 package copy of that library by
setting the OPENSSL_MODULES environment variable (rather than
touching LD_LIBRARY_PATH).

(See LP: #1972939 comment 11 for a brief bit of additional detail.)

If you are considering building a PPA to resolve this in Jammy,
here are a few possible approaches I can envision:

A) a "full" libssl3 package that contains the upstream patch applied
   directly to ossl-modules/legacy.so , and would be installed
   in place of the standard libssl3 package.

B) a "full" libssl3 package (installed in place of the standard
   one) that contains the standard legacy.so file and then also a
   ossl-modules/legacy-fixedBF.so file with a copy of the
   legacy.so library patched by the fix.

C) a "special" libssl3 package (e.g. named libssl3-fixedbf) which
   would contain only the ossl-modules/legacy-fixedBF.so file, and
   would be installed alongside the standard libssl3 package.

The idea behind options B and C is that both the
broken/original-Jammy behavior and the fixed behavior would be
available simultaneously on the system, with the user selecting
which behavior they are looking for based on which module they
load at run time. In both cases the ossl-modules/ directory
would contain both legacy.so and legacy-fixedBF.so files (once
the PPA package in question was installed).

With option A only the corrected library would be available
(at run time with the PPA package installed), and it would be
found under the standard "legacy" module name. This would
certainly be the easiest-to-use approach in the Tinc-interop
case, but limits flexibility for sites which need compatibility
with both the default-Jammy and corrected behavior at the same
time.

(The names "legacy-fixBF.so" and "libssl3-fixedbf" are just names I
made up to provide these examples; some other scheme like
"legacy-issue18359.so" or whatever would work similarly from the
user's standpoint.)

                                        Nathan