openssl: After symbol versioning, distributed pkgs are missing API symbols (e.g. EVP_PKEY_asn1_set_item)

[Nicola Tuveri]

I'm developing an ENGINE for OpenSSL, and close to release, I noticed
that in Ubuntu 16.04 LTS and in Debian oldstable-backports the build fails with the following output:

```
/usr/bin/cc -fPIC -g -shared -Wl,-soname,liblibsuola.so -o liblibsuola.so CMakeFiles/suola.dir/suola.c.o CMakeFiles/suola.dir/suola_keypair.c.o CMakeFiles/suola.dir/debug/debug.c.o CMakeFiles/suola.dir/meths/X25519_meth.c.o CMakeFiles/suola.dir/meths/ed25519_meth.c.o CMakeFiles/suola.dir/meths/suola_asn1_meth.c.o CMakeFiles/suola.dir/meths/suola_md_identity_meth.c.o CMakeFiles/suola.dir/ossl/ossl_compat.c.o CMakeFiles/suola.dir/ossl/suola_err.c.o CMakeFiles/suola.dir/ossl/suola_objects.c.o CMakeFiles/suola.dir/providers/libsodium/base.c.o CMakeFiles/suola.dir/providers/libsodium/curve25519.c.o CMakeFiles/suola.dir/providers/libsodium/ed25519.c.o -lssl -lcrypto /opt/libsodium-stable/lib/libsodium.so -Wl,-z,defs -Wl,-rpath,/opt/libsodium-stable/lib:
CMakeFiles/suola.dir/meths/suola_asn1_meth.c.o: In function `suola_register_asn1_meth':
/usr/local/src/libsuola/meths/suola_asn1_meth.c:505: undefined reference to `EVP_PKEY_asn1_set_item'
collect2: error: ld returned 1 exit status
make[2]: *** [liblibsuola.so] Error 1
CMakeFiles/suola.dir/build.make:412: recipe for target 'liblibsuola.so' failed
make[2]: Leaving directory '/usr/local/src/libsuola/build'
make[1]: *** [CMakeFiles/suola.dir/all] Error 2
make: *** [all] Error 2
```

This does not happen linking against the same exact release of openssl
compiled from source on the same system.

I then learned that one of the patches applied by Debian and inherited also by Ubuntu has the goal of versioning library symbols to avoid conflicts.

Unfortunately said patch is not updated regularly with each release of
OpenSSL, resulting, like in my case, in symbols available in the public
header files but masked through versioning in the shared library binary.

The attached patch fixes my need by adding `EVP_PKEY_asn1_set_item` to
the list, but you might consider an internal review of your release
process to make sure that the list of symbols is updated whenever a new
upstream releases makes new functions publicly available.

I believe this bug is important, as it stops everyone using official
packages from using third-party ENGINEs that require to use that
function to set special handling of ASN.1 format, which basically
includes every ENGINE that would add support for cryptosystems that
upstream OpenSSL does not support (defying the purpose of using some
ENGINEs).

The patch I propose covers my use case, but basically the package as is results
unusable to any user of any application that may require functions
available in the public headers but accidentally masked in the symbol
versioning step.

The ideal outcome of fixing this issue would consist in making the
versioning patch dynamic, checking when symbols are added (or removed)
in newer releases and updating the list accordingly.

You might have the same versioning patch applied in other releases, so it's
worth tagging this bug also for those to make the handling of the exposed symbols consistent.

Finally I also opened a similar bug against the equivalent Debian oldstable-backports package, but I opened an Ubuntu-specific bug report because 16.04 ships a more recent version of the package and the inherited symbol versioning patch has already been modified compared with the original Debian one.
For reference this is the Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895547

-- System information --

# lsv_release -rd
Description: Ubuntu 16.04.4 LTS
Release: 16.04

# apt-cache policy libssl-dev
libssl-dev:
  Installed: 1.0.2g-1ubuntu4.11
  Candidate: 1.0.2g-1ubuntu4.11
  Version table:
 *** 1.0.2g-1ubuntu4.11 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.2g-1ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: libssl-dev 1.0.2g-1ubuntu4.11
ProcVersionSignature: Ubuntu 4.13.0-36.40~16.04.1-generic 4.13.13
Uname: Linux 4.13.0-36-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.16
Architecture: amd64
Date: Fri Apr 13 21:10:58 2018
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
SourcePackage: openssl
UpgradeStatus: No upgrade log present (probably fresh install)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Patch for the symbol versioning patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Nicola Tuveri (nic-tuv)]

I can confirm that while this is "fixed" by updating to a newer release of Ubuntu (or Debian, as can be read in the Debian bug tracker), this problem is still present in Xenial.

The reason it is fixed in newer releases is that they ship OpenSSL >= 1.1.0 where upstream takes care of symbol versioning for the public symbols in the library.

Xenial uses OpenSSL 1.0.2, and the package maintainers have a custom script to tag known symbols in the library with a version number, and mask all the symbols not explicitly whitelisted in the script.

As reported more than 1 year ago, this script has not been properly maintained, so in Xenial, today, there are still symbols like `EVP_PKEY_asn1_set_item` that are exposed to application developers as part of the public API in the headers contained in the latest package version for libssl-dev, but that are not linkable because they have never been whitelisted by the package maintainers.

In Debian this is now considered Fixed, as the currently supported releases all ship OpenSSL >=1.1.0, but Ubuntu Xenial is supposed to be supported until April 2021.

What can I do to get some attention to this issue and fix the problem in Xenial?

[Nicola Tuveri (nic-tuv)]

I can also further confirm that even on the latest LTS, bionic, the same problem still exists today when using the latest version of the official packages `openssl1.0`, `libssl1.0.0`, and `libssl1.0-dev` to have access OpenSSL 1.0.2.

So I would also formally request to reevaluate the "fixed" status for Ubuntu releases after xenial that are still offering official packages for OpenSSL <1.1.0.

[Nicola Tuveri (nic-tuv)]

Here is a minimal working example to test the issue: https://gist.github.com/romen/b95e99b3563a8ba4c27d88512c7932ff

[Nicola Tuveri (nic-tuv)]

Is there anything I can do to accelerate the resolution of this bug for Xenial and Bionic?

I already posted a proposed solution for the specific symbol that I am missing, but in theory fixing this should be as easy as running whatever script the package mantainers used in the first place to generate the file debian/patches/version-script.patch for this package.