Jammy tinc incompatibile with older (e.g. Xenial) tinc nodes

[Nathan Stratton Treadway]

The tinc included in Jammy (1.0.36-2build1 linked with libssl3) cannot connect to tinc nodes running e.g. tinc from Xenial (1.0.26-1).

(Tinc from Impish, which is also v1.0.36-2 but is linked to libssl1.1, can connect to these nodes without problems.)

The symptom is a log message (on the system running Jammy) during the metadata channel negotiation (with debug level set to 5):

Error during initialisation of cipher from tinc_xenial [...] error:0308010C:digital envelope routines::unsupported

[Nathan Stratton Treadway (nathanst)]

Since the tinc version number in Focal/Impish and Jammy are the same, it might be worth adding a warning to the release notes to people don't unexpectedly loose VPN access by upgrading to Jammy. (Or explaining a workaround there, if one can be determined.)

[Simon Chopin (schopin)]

I'm guessing there are some SSL certificates involved? If so, this issue
is mentioned in the release notes: certificates that use e.g. SHA1 as
the digest algorithm should be re-issued by your provider with a
stronger hash algorithm.

Would you be able to check that it is the correct diagnostic?
If you have a PEM file, you can see mentions of the hash algorithms in
the "Signature Algorithm" fields when using the following command:

openssl x509 -in cert.pem -noout -text

[Nathan Stratton Treadway (nathanst)]

On Wed, May 18, 2022 at 07:42:04 -0000, Simon Chopin wrote:
> I'm guessing there are some SSL certificates involved? If so, this issue

Tinc uses openssl's implementations of specific alogorithms, but does not
use either TLS or SSL certificates. (So I don't think the Tinc situation
is covered by the existing OpenSSL 3.0 section of the Release Notes
document.)

The Xenial version of Tinc uses the Blowfish algorithm for the metadata
connection, which openssl3 does move to the legacy provider -- but even
though enabling the legacy provider on the Jammy node allows the
connenction setup to get further along, it's not sufficient to get a
working connection -- the libssl3 transition seems to have affected some
other aspect of the connection as well...

[Simon Chopin (schopin)]

Could you give more details about what happens when using the legacy providers?

[Simon Chopin (schopin)]

Also, does tinc work in a purely Jammy context? :-)

[Nathan Stratton Treadway (nathanst)]

On Wed, May 18, 2022 at 13:41:06 -0000, Simon Chopin wrote:
> Also, does tinc work in a purely Jammy context? :-)

As far as I can determine the issue relates to compatibility between
libssl3 and the algorithms used by the Xenial-era tinc, and thus I can't
imagine Jammy-to-Jammy would be a problem....

[Nathan Stratton Treadway (nathanst)]

On Wed, May 18, 2022 at 13:37:46 -0000, Simon Chopin wrote:
> Could you give more details about what happens when using the legacy
> providers?

The short version is that by enabling the legacy provider and setting
SECLEVEL to 1, I'm able to get past the "digital envelope
routines::unsupported" error during the tinc metadata channel setup...
but the Jammy node still (just a step or two later in the negotiation
process) reports a "Bogus data received from" error and then aborts the
connection.

The "Bogus data received from" error is a tinc error message, but as far
as I can tell the likely trigger for that message is some sort of
failure to decrypt incoming data by the OpenSSL library -- and since
Focal, Impish and Jammy all have exactly the same tinc version, it would
seem the issue is libssl3-related... but I am not sure precisely how....

You can find additional details in this tinc-mailing-list thread:
  https://www.tinc-vpn.org/pipermail/tinc/2022-May/005598.html
(but so far the discussion there hasn't managed to narrow down the exact
interaction between tinc and libssl that's causing the problem).

[Nathan Stratton Treadway (nathanst)]

On Wed, May 18, 2022 at 13:41:06 -0000, Simon Chopin wrote:
> Also, does tinc work in a purely Jammy context? :-)

Sorry, I just realized that I had not mentioned here on this bug the
results of my tests between various Ubuntu versions. I didn't test
Jammy-to-Jammy, but (briefly):

  * Jammy (1.0.36/libssl3) to Xenial (1.0.26/libssl1.0.0) fails

  * Impish (1.0.36/libssl1.1) works to both Jammy and Xenial (no
    openssl.cnf changes needed on any node)

  * Focal (also 1.0.36/libssl1.1]) worked to Xenial. (I did not
    test that to Jammy.)

  * Jammy to Bionic (1.0.33/libssl1.1) works (no openssl.cnf changes
    needed)

(I did not test point-releases between Xenial and Bionic.)

[Don (speedy-carboncode)]

It appears the issue is resolved in libssl3 3.0.4-1ubuntu1 from kinetic (in addition to enabling the legacy providers)

[Nathan Stratton Treadway (nathanst)]

On Fri, Aug 05, 2022 at 00:35:32 -0000, Don wrote:
> It appears the issue is resolved in libssl3 3.0.4-1ubuntu1 from kinetic
> (in addition to enabling the legacy providers)

Thanks for that hint.

Can you provide any additional details on your Tinc environment and what
exactly allowed the connection to start working?

For example, did you previously attempt to connect a Tinc node running
Kinentic to a Xenial node and have it fail, but then see it start
working once you upgraded libssl3 to the 3.0.4-1ubuntu1 release?

       Nathan

[Nathan Stratton Treadway (nathanst)]

On Fri, Aug 05, 2022 at 00:35:32 -0000, Don wrote:
> It appears the issue is resolved in libssl3 3.0.4-1ubuntu1 from kinetic
> (in addition to enabling the legacy providers)

I installed a Kinetic test environment, and confirmed that I was able to
connect to my Xenial tinc (1.0.26-1) instance successfully (with the
legacy provider enabled).

I noticed that Jammy and Kinetic actually have the same exact tinc
package, so I figure the difference in functionality must be in libssl3:
  Jammy: pool/universe/t/tinc/tinc_1.0.36-2build1_amd64.deb
  Kinetic: pool/universe/t/tinc/tinc_1.0.36-2build1_amd64.deb

I experimented with downgrading the libssl3 package:

  libssl3 3.0.5-2ubuntu1 (current latest version): worked

  3.0.4-1ubuntu1: worked

  3.0.3-5ubuntu3: got "Bogus data received from" error message again

Further experimentation running tinc with the OPENSSL_MODULES environment
variable set confirmed that the tinc connection succeeds if libssl3
3.0.3-5ubuntu3 is installed but the ossl-modules/legacy.so file from
3.0.4-1ubuntu1 is used by the tincd process.

Cross-referencing the commit history for legacyprov.c
with the the git commit logs for changes between 3.0.3 and .4:
     https://github.com/openssl/openssl/compare/openssl-3.0.3...openssl-3.0.4

, I found the commit "Fix regression in default key length for Blowfish
CFB and OFB ciphers"... which would seem to be the change allows Tinc to
work again (since Tinc 1.0.26 uses the Blowfish algorithm for the
metadata connection).

  https://github.com/openssl/openssl/commit/1b8ef23e68b273bb5e59f60df62251153f24768d

  https://github.com/openssl/openssl/issues/18359
    "OpenSSL 3 cannot decrypt data encrypted with OpenSSL 1.1 with
    blowfish in OFB or CFB modes"

Finally, going back to the original issue on Jammy: I copied the
ossl-modules/legacy.so taken from libssl3 3.0.5-2ubuntu1 over to my
Jammy instance and pointed OPENSSL_MODULES to that file (in
/etc/default/tinc)... and sure enough that allowed my Jammy Tinc node to
connect to the Xenial Tinc node successfully as well....

[Nathan Stratton Treadway (nathanst)]

On Wed, May 18, 2022 at 15:36:30 -0000, Nathan Stratton Treadway wrote:
> On Wed, May 18, 2022 at 13:37:46 -0000, Simon Chopin wrote:
> > Could you give more details about what happens when using the legacy
> > providers?
>
> The short version is that by enabling the legacy provider and setting
> SECLEVEL to 1, I'm able to get past the "digital envelope

(With the fixed version of OpenSSL's legacy.so, the SECLEVEL=1
configuration change is no longer needed -- tincd's openssl.cnf only
needs to activate the "legacy" provider.)

[Nathan Stratton Treadway (nathanst)]

(I've opened LP bug #1990216 to request that the fix for upstream "OpenSSL 3 cannot decrypt data encrypted with OpenSSL 1.1 with blowfish in OFB or CFB modes #18359" be backported to libssl3 in Jammy.
  https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1990216
)

[Nathan Stratton Treadway (nathanst)]

Just to have links in both directions between various bug trackers:
"connecting tinc 1.0.36/libssl3 to older nodes #414"
  https://github.com/gsliepen/tinc/issues/414