openssl engine error if trying to exploit hw crypto on z due to library issue

[Frank Heimes]

[Testcase]
* configure ibmca engine as per below instructions
* execute openssl engine -c -vvvv
* it should complete without any loading errors

[Impact]
* Out of the box stock configuration results in non-usable engine which errors out
* Thus currently, without workarounds, the acceleration engine does not work. Meaning regression potential is low

Please note this is the first time we are integrating openssl-ibmca, and it is not enabled by default. Hopefully things will be better / more stable going forward.

openssl-ibmca usually requires libica2 and libica-utils for proper functioning and all required tooling (like icainfo, icastats, etc.)

But after the installation of these packages and the configuration, with is like this:
sudo tee -a /etc/ssl/openssl.cnf < /usr/share/doc/openssl-ibmca/examples/openssl.cnf.sample
sudo vi /etc/ssl/openssl.cnf
adding the following line as the first active one:
openssl_conf = openssl_def
and removing or commenting all other occurrences of that line in the config file
and saving and closing the openssl.cnf file
this output of the openssl engine command is expected:

$ openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support

or even more precise these chiphers should be listed in case of "-c":

$ openssl engine -c
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
 [RAND, DES-ECB, DES-CBC, DES-OFB, DES-CFB, DES-EDE3, DES-EDE3-CBC, DES-EDE3-OFB, DES-EDE3-CFB, AES-128-ECB, AES-192-ECB, AES-256-ECB, AES-128-CBC, AES-192-CBC, AES-256-CBC, AES-128-OFB, AES-192-OFB, AES-256-OFB, AES-128-CFB, AES-192-CFB, AES-256-CFB, SHA1, SHA256, SHA512]

But instead openssl is giving this error, due to a missing "libica.so":
$ openssl engine
Error configuring OpenSSL
4395950360208:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(libica.so): libica.so: cannot open shared object file: No such file or directory
4395950360208:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:233:
4395950360208:error:80066068:lib(128):IBMCA_INIT:dso failure:e_ibmca.c:1286:
4395950360208:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(libica.so): libica.so: cannot open shared object file: No such file or directory
4395950360208:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:233:
4395950360208:error:80066068:lib(128):IBMCA_INIT:dso failure:e_ibmca.c:1286:
4395950360208:error:260BC066:engine routines:INT_ENGINE_CONFIGURE:engine configuration error:eng_cnf.c:191:section=ibmca_section, name=init, value=1
4395950360208:error:0E07606D:configuration file routines:MODULE_RUN:module initialization error:conf_mod.c:223:module=engines, value=engine_section, retcode=-1
$

There is no libica.so that is shipped with any of the above packages (verified with dpkg -l) or otherwise available in the filesystem:
$ sudo find / -name "libica.so" 2>/dev/null
ubuntu@HWE0001:~$

But there is a different verison of that libica:
$ sudo find / -name "*libica.so*" 2>/dev/null
/usr/lib/s390x-linux-gnu/libica.so.2
/usr/lib/s390x-linux-gnu/libica.so.2.6.1
$

So there are right now two workarounds:
1)
creating a (symbolic) link from libica.so.2 to libica.so, like
$ sudo ln -s /usr/lib/s390x-linux-gnu/libica.so.2 /usr/lib/s390x-linux-gnu/libica.so
that allows openssl to find a library named 'libica.so':
18:15:00: <email address hidden>: ubuntu@HWE0001:~$ openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
But this could lead to issues in case of any potential functions or interface changes there we introduced with libica.so.2
2)
installation of the "libica-dev" package that provides a (development) version of libica.so:
$ dpkg -L libica-dev | grep libica.so
/usr/lib/s390x-linux-gnu/libica.so
$

But the hardware crypto exploitation should work out of the box w/o the link or the libica-dev package.

Either libica.so should be shipped (in addition to libica.so.2) with the proper dependency to openssl-ibmca - openssh-ibmca should make use of libica2 instead of libica.so.2...

[Dimitri John Ledkov (xnox)]

Please try packages from:

sudo add-apt-repository ppa:ci-train-ppa-service/2043
sudo apt update
sudo apt full-upgrade

With packages there, libica.so is still attempted to be loaded first. If that fails, libica.so.2 is attempted to be loaded, and that should succeed.

Before:
$ openssl engine ibmca
(ibmca) Ibmca hardware engine support
4396855273104:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(libica.so): libica.so: cannot open shared object file: No such file or directory
4396855273104:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:233:
4396855273104:error:80066068:lib(128):IBMCA_INIT:dso failure:e_ibmca.c:1286:

After:
$ openssl engine ibmca
(ibmca) Ibmca hardware engine support
4396661810832:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(libica.so): libica.so: cannot open shared object file: No such file or directory
4396661810832:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:233:

Which is still a bit ugly messaging, but should work out of the box.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl-ibmca - 1.3.0-0ubuntu3

---------------
openssl-ibmca (1.3.0-0ubuntu3) zesty; urgency=medium

  * Attempt to dlopen libica.so.2, if libica.so (or ctrl provided one)
    fails. LP: #1605511
  * Add depends on libica2.

 -- Dimitri John Ledkov <email address hidden> Tue, 04 Oct 2016 15:25:59 +0100

[Frank Heimes (fheimes)]

Being curious I tried "openssl-ibmca_1.3.0-0ubuntu3_s390x.deb" on yakkety and xenial, too.
It looks good - the error message is gone.
(Even if I do not yet know how to interpret 'built on: reproducible build, date unspecified'...)

Yakkety:
========

>>> openssl-ibmca prior to (1.3.0-0ubuntu3):

ubuntu@s1lp14:~$ openssl speed -evp des-ede3-cbc
Doing des-ede3-cbc for 3s on 16 size blocks: 23686887 des-ede3-cbc's in 2.99s
Doing des-ede3-cbc for 3s on 64 size blocks: 16020848 des-ede3-cbc's in 3.00s
Doing des-ede3-cbc for 3s on 256 size blocks: 6971169 des-ede3-cbc's in 3.00s
Doing des-ede3-cbc for 3s on 1024 size blocks: 2154635 des-ede3-cbc's in 3.00s
Doing des-ede3-cbc for 3s on 8192 size blocks: 287230 des-ede3-cbc's in 3.00s
OpenSSL 1.0.2g 1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,char) des(idx,cisc,16,int) aes(partial) blowfish(idx)
compiler: cc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DB_ENDIAN -g -O2 -fdebug-prefix-map=/build/openssl-tmX0Mb/openssl-1.0.2g=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DAES_CTR_ASM -DAES_XTS_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
des-ede3-cbc 126752.57k 341778.09k 594873.09k 735448.75k 784329.39k
4396106589840:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(libica.so): libica.so: cannot open shared object file: No such file or directory
4396106589840:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:233:
4396106589840:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(libica.so): libica.so: cannot open shared object file: No such file or directory
4396106589840:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:233:

>>> openssl-ibmca (1.3.0-0ubuntu3):

ubuntu@s1lp14:~$ sudo dpkg -i ./openssl-ibmca_1.3.0-0ubuntu3_s390x.deb
(Reading database ... 91267 files and directories currently installed.)
Preparing to unpack .../openssl-ibmca_1.3.0-0ubuntu3_s390x.deb ...
Unpacking openssl-ibmca (1.3.0-0ubuntu3) over (1.3.0-0ubuntu3) ...
Setting up openssl-ibmca (1.3.0-0ubuntu3) ...
Processing triggers for man-db (2.7.5-1) ...
ubuntu@s1lp14:~$ openssl speed -evp des-ede3-cbc
Doing des-ede3-cbc for 3s on 16 size blocks: 24062744 des-ede3-cbc's in 2.99s
Doing des-ede3-cbc for 3s on 64 size blocks: 16179261 des-ede3-cbc's in 3.00s
Doing des-ede3-cbc for 3s on 256 size blocks: 7044115 des-ede3-cbc's in 3.00s
Doing des-ede3-cbc for 3s on 1024 size blocks: 2157283 des-ede3-cbc's in 3.00s
Doing des-ede3-cbc for 3s on 8192 size blocks: 287455 des-ede3-cbc's in 3.00s
OpenSSL 1.0.2g 1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,char) des(idx,cisc,16,int) aes(partial) blowfish(idx)
com...

[Brian Murray (brian-murray)]

Hello Frank, or anyone else affected,

Accepted openssl-ibmca into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl-ibmca/1.3.0-0ubuntu2.16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Frank, or anyone else affected,

Accepted openssl-ibmca into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl-ibmca/1.3.0-0ubuntu2.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Frank Heimes (fheimes)]

Yepp both works for me - in Xenial and Yakkety:

Xenial:
-------

ubuntu@s1lp14:~$ sudo apt-cache policy openssl-ibmca
openssl-ibmca:
  Installed: (none)
  Candidate: 1.3.0-0ubuntu2.16.04.1
  Version table:
     1.3.0-0ubuntu2.16.04.1 500
        500 http://ports.ubuntu.com xenial-proposed/universe s390x Packages
     1.3.0-0ubuntu2 500
        500 http://ports.ubuntu.com xenial/universe s390x Packages
ubuntu@s1lp14:~$

ubuntu@s1lp14:~$ sudo apt --yes install openssl-ibmca libica-utils
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libica2
The following NEW packages will be installed:
  libica-utils libica2 openssl-ibmca
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 92.3 kB of archives.
After this operation, 333 kB of additional disk space will be used.
Get:1 http://ports.ubuntu.com xenial/universe s390x libica2 s390x 2.6.1-3 [60.0 kB]
Get:2 http://ports.ubuntu.com xenial/universe s390x libica-utils s390x 2.6.1-3 [15.2 kB]
Get:3 http://ports.ubuntu.com xenial-proposed/universe s390x openssl-ibmca s390x 1.3.0-0ubuntu2.16.10.1 [17.1 kB]
Fetched 92.3 kB in 0s (287 kB/s)
Selecting previously unselected package libica2:s390x.
(Reading database ... 44591 files and directories currently installed.)
Preparing to unpack .../0-libica2_2.6.1-3_s390x.deb ...
Unpacking libica2:s390x (2.6.1-3) ...
Selecting previously unselected package libica-utils.
Preparing to unpack .../1-libica-utils_2.6.1-3_s390x.deb ...
Unpacking libica-utils (2.6.1-3) ...
Selecting previously unselected package openssl-ibmca.
Preparing to unpack .../2-openssl-ibmca_1.3.0-0ubuntu2.16.10.1_s390x.deb ...
Unpacking openssl-ibmca (1.3.0-0ubuntu2.16.10.1) ...
Processing triggers for libc-bin (2.24-3ubuntu2) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up libica2:s390x (2.6.1-3) ...
Setting up openssl-ibmca (1.3.0-0ubuntu2.16.10.1) ...
Setting up libica-utils (2.6.1-3) ...
Processing triggers for libc-bin (2.24-3ubuntu2) ...
ubuntu@s1lp14:~$

ubuntu@s1lp14:~$ sudo apt-cache policy openssl-ibmca
openssl-ibmca:
  Installed: 1.3.0-0ubuntu2.16.04.1
  Candidate: 1.3.0-0ubuntu2.16.04.1
  Version table:
 *** 1.3.0-0ubuntu2.16.04.1 500
        500 http://ports.ubuntu.com xenial-proposed/universe s390x Packages
        100 /var/lib/dpkg/status
     1.3.0-0ubuntu2 500
        500 http://ports.ubuntu.com xenial/universe s390x Packages
ubuntu@s1lp14:~$

ubuntu@s1lp14:~$ sudo cp -p /etc/ssl/openssl.cnf /etc/ssl/openssl.cnf_`date +%Y%m%d`.backup
ubuntu@s1lp14:~$ ls -la /etc/ssl/openssl.cnf*
-rw-r--r-- 1 root root 10835 Nov 18 15:28 /etc/ssl/openssl.cnf
-rw-r--r-- 1 root root 10835 Sep 23 08:22 /etc/ssl/openssl.cnf_20161118.backup
ubuntu@s1lp14:~$

ubuntu@s1lp14:~$ sudo tee -a /etc/ssl/openssl.cnf < /usr/share/doc/openssl-ibmca/examples/openssl.cnf.sample
...
ubuntu@s1lp14:~$ ls -la /etc/ssl/openssl.cnf*
-rw-r--r-- 1 root root 12251 Nov 18 15:33 /etc/ssl/openssl.cnf
-rw-r--r-- 1 root root 10835 Sep 23 08:22 /etc/ssl/openssl.cnf_20161118.backup
ubuntu@s1lp14:~$

ubuntu@s1lp14:~$ sudo vi /etc/ssl/openssl.cnf
357: openssl_conf = openssl_def
=>
357: # open...

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl-ibmca - 1.3.0-0ubuntu2.16.04.1

---------------
openssl-ibmca (1.3.0-0ubuntu2.16.04.1) xenial; urgency=medium

  * Attempt to dlopen libica.so.2 by default, libica.so (or ctrl provided one)
    when the default fails. LP: #1605511
  * Add depends on libica2.

 -- Dimitri John Ledkov <email address hidden> Tue, 04 Oct 2016 15:25:59 +0100

[Chris J Arges (arges)]

The verification of the Stable Release Update for openssl-ibmca has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl-ibmca - 1.3.0-0ubuntu2.16.10.1

---------------
openssl-ibmca (1.3.0-0ubuntu2.16.10.1) yakkety; urgency=medium

  * Attempt to dlopen libica.so.2 by default, libica.so (or ctrl provided one)
    when the default fails. LP: #1605511
  * Add depends on libica2.

 -- Dimitri John Ledkov <email address hidden> Tue, 04 Oct 2016 15:25:59 +0100