Ubuntu 17.10 - opencryptoki 3.7.0 segmentation fault on pkcsconf -t

[bugproxy]

[Impact]

 * Impossible to use multiple different token types in opencryptoki.

[Test Case]

 * Ensure one has a system with multiple tokens configured. E.g. s390x with ICA and SoftTok.
 * Execute $ pkcsconf -t
 * The model for two tokens should be different e.g.:
$ pkcsconf -t | grep Model
 Model: IBM ICA
 Model: IBM SoftTok

On broken systems, whichever token is loaded first is repeated for all subsequent tokens.

[Regression Potential]

 * No code changes are done. It appears that the code relies on dynamically loading and rebinding functions, yet that is not possible to do with distribution default linker flag -Wl,-Bsymbolic-functions. Recompiling the software with this flag stripped makes the software operate as expected.

[Other Info]

 * The fix for this issue is similar to what has been employed previously. E.g. in https://bugs.launchpad.net/ubuntu/+source/xfe/+bug/644645

[Original Bug report]

Running Ubuntu 17.10 on a zVM s390x environment with opencryptoki 3.7 installed via apt, the command pkcsconf -t gives a segmentation fault.

We did some tests (all on a zVM):
1. On a debian testing installation: the opencryptoki 3.7.0+dfsg-4 package works as expected.
2. On a ubuntu 17.04 installation: the opencryptoki (3.6.2+dfsg-1 that is available on the repo) package works as expected.
3. On a ubuntu 17.10 installation: opencryptoki 3.7.0+dfsg-4 gives segmentation fault (pkcsconf -t)
4. On a ubuntu 17.10 installation: downloading building opencryptoki 3.7.0 from Github manually and installing it, works as expected.
5. On a ubuntu 17.10 installation: installing opencryptoki 3.7.0+dfsg-4 package from debian testing repository, work as expected.

It seems that opencryptoki 3.7.0+dfsg-4 package from Ubuntu was built differently compared to Debian. We believe that the build was done incorrectly and is causing the pkcsconf -t command to segfault.

Could you guys verify how the package is being built and compare it to debian's opencryptoki package?

Machine Type = zVM s390x

---Steps to Reproduce---
 #apt-get install opencryptoki

#pkcsconf -t
Segmentation fault

---Patches Installed---
na

---uname output---
Linux 4.13.0-16-generic #19-Ubuntu SMP Wed Oct 11 18:33:05 UTC 2017 s390x s390x s390x GNU/Linux

---Debugger---
A debugger is not configured

Userspace tool common name: opencryptoki

Userspace rpm: opencryptoki_3.7.0+dfsg-4

The userspace tool has the following bit modes: 64-bit

Userspace tool obtained from project website: na

-Attach ltrace and strace of userspace application.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-10-20 09:58 EDT-------
Hello Frank,

you might want to run 'id' to see if your user is member of the pkcs11 group. Further ensure the pkcsslotd is started.

Thanks,
Christian

[Frank Heimes (frank-heimes)]

Hi, yepp - in between I followed the steps of the crypto paper ...

[Frank Heimes (frank-heimes)]

after initial install and config of pkcs11:

$ sudo apt install opencryptoki libtspi1

$ sudo usermod -aG pkcs11 ubuntu

$ grep pkcs11 /etc/group
pkcs11:x:118:root,ubuntu

$ sudo systemctl enable pkcsslotd.service
Synchronizing state of pkcsslotd.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable pkcsslotd

$ sudo systemctl start pkcsslotd.service

$ sudo pkcsconf
usage: pkcsconf [-itsmIupPh] [-c slotnumber -U userPIN -S SOPin -n newpin]
 -i display PKCS11 info
 -t display token info
 -s display slot info
 -m display mechanism list
 -l display slot description
 -I initialize token
 -u initialize user PIN
 -p set the user PIN
 -P set the SO PIN
 -h show this help

$ sudo pkcsconf -t
Segmentation fault

$ sudo pkcsconf -i
PKCS#11 Info
 Version 2.20
 Manufacturer: IBM
 Flags: 0x0
 Library Description: Meta PKCS11 LIBRARY
 Library Version 3.7
Segmentation fault

$ sudo pkcsconf -s
Slot #1 Info
 Description: Linux
 Manufacturer: IBM
 Flags: 0x1 (TOKEN_PRESENT)
 Hardware Version: 0.0
 Firmware Version: 0.0
Slot #2 Info
 Description: Linux
 Manufacturer: IBM
 Flags: 0x1 (TOKEN_PRESENT)
 Hardware Version: 0.0
 Firmware Version: 0.0
Slot #3 Info
 Description: Linux
 Manufacturer: IBM
 Flags: 0x1 (TOKEN_PRESENT)
 Hardware Version: 0.0
 Firmware Version: 0.0
Segmentation fault

[Frank Heimes (frank-heimes)]

16.04.3 and 17.04 are not affected

[Dimitri John Ledkov (xnox)]

18.04 has 3.8.1+dfsg-3 now, does that one work ok? And thus is this fix released in 18.04 bionic?

[Frank Heimes (frank-heimes)]

pkcsconf on bionic works - see attachment
so a no-change rebuild on artful should be fine

[Dimitri John Ledkov (xnox)]

Could you please if upgrading Artful to this PPA https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3083 resolves the issue for you on Artful too?

[Frank Heimes (frank-heimes)]

unfortunately I still get the segfault with the package from the PPA
for details see attached file

[Frank Heimes (frank-heimes)]

bisect required from 3.7.0 to 3.8.1 to identify when the fix got introduced.

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-14 12:01 EDT-------
(In reply to comment #21)
> bisect required from 3.7.0 to 3.8.1 to identify when the fix got introduced.

Hi Frank,

Do we have any update on this issue?

It just came to my knowledge that it is also happening on Ubuntu 18.04.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-14 12:37 EDT-------
Hi Frank,
I've found something interesting.
On Ubuntu 18.04, if you comment ep11 (slot 4) from the /etc/opencryptoki.conf, and restart pkcsslotd, then pkcsconf -t works.

But even commenting ep11 from the configuration file, the output of pkcsconf -t is not consistent. It is showing ica for both slot 1 and 2, where 2 should be CCA.
# pkcsconf -t
Token #1 Info:
Label: IBM ICA PKCS #11
Manufacturer: IBM Corp.
Model: IBM ICA
Serial Number: 123
Flags: 0x880045 (RNG|LOGIN_REQUIRED|CLOCK_ON_TOKEN|USER_PIN_TO_BE_CHANGED|SO_PIN_TO_BE_CHANGED)
Sessions: 0/18446744073709551614
R/W Sessions: 18446744073709551615/18446744073709551614
PIN Length: 4-8
Public Memory: 0xFFFFFFFFFFFFFFFF/0xFFFFFFFFFFFFFFFF
Private Memory: 0xFFFFFFFFFFFFFFFF/0xFFFFFFFFFFFFFFFF
Hardware Version: 1.0
Firmware Version: 1.0
Time: 18:17:30
Token #2 Info:
Label: IBM ICA PKCS #11
Manufacturer: IBM Corp.
Model: IBM ICA
Serial Number: 123
Flags: 0x880045 (RNG|LOGIN_REQUIRED|CLOCK_ON_TOKEN|USER_PIN_TO_BE_CHANGED|SO_PIN_TO_BE_CHANGED)
Sessions: 0/18446744073709551614
R/W Sessions: 18446744073709551615/18446744073709551614
PIN Length: 4-8
Public Memory: 0xFFFFFFFFFFFFFFFF/0xFFFFFFFFFFFFFFFF
Private Memory: 0xFFFFFFFFFFFFFFFF/0xFFFFFFFFFFFFFFFF
Hardware Version: 1.0
Firmware Version: 1.0
Time: 18:17:30
Token #3 Info:
Label: IBM ICA PKCS #11
Manufacturer: IBM Corp.
Model: IBM ICA
Serial Number: 123
Flags: 0x880045 (RNG|LOGIN_REQUIRED|CLOCK_ON_TOKEN|USER_PIN_TO_BE_CHANGED|SO_PIN_TO_BE_CHANGED)
Sessions: 0/18446744073709551614
R/W Sessions: 18446744073709551615/18446744073709551614
PIN Length: 4-8
Public Memory: 0xFFFFFFFFFFFFFFFF/0xFFFFFFFFFFFFFFFF
Private Memory: 0xFFFFFFFFFFFFFFFF/0xFFFFFFFFFFFFFFFF
Hardware Version: 1.0
Firmware Version: 1.0
Time: 18:17:30
I'll be doing some debugging, but something really wrong is on the deb package.

Remembering that this is not happening on any other distro, or debian.

[Frank Heimes (frank-heimes)]

I can re-create and confirm this ...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-16 10:20 EDT-------
Hi Frank,

As we mentioned in the description, if you manually build and install opencryptoki, everything works. So, it is clear that is something wrong in the .deb package.

Could it be possible for you guys to generate a new opencryptoki .deb package, from scratch, so we can see if the error persists?

[Dimitri John Ledkov (xnox)]

> Remembering that this is not happening on any other distro, or debian.

Note that all other distributions use lower minimum ISA level, and possibly have less hardening enabled. Ubuntu toolchain defaults to -march=zEC12 & we do have PIE enabled, fortify sources, etc. So it may make sense to recompile the stack with optimisations turned off and using lower instruction sets, to see if there is a toolchain bug / incompatibility lurking somewhere.

It was previously confirmed that everything works fine, in bionic during development. But I think validation was done in a z/VM at the time, not an LPAR. It would be nice to document the configs / underlying lpar/z/vm configs too, which trigger the crashes. To insure we actually find the root cause of the issue at hand.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-16 11:43 EDT-------
We reproduced the problem on a zVM Guest UBUNTU 18.04 ( 4.15.0-20-generic ) instance running on IBM Z 14 machine (IBM Type: 3906 Model: 703 M03) using opencryptoki version 3.9.0+dfsg-0ubuntu1.
~# lscpu
Architecture: s390x
...
Hypervisor: z/VM 6.4.0
Hypervisor vendor: IBM
Virtualization type: full

# lszcrypt -V
CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT PENDINGQ_CNT REQUESTQ_CNT HW_TYPE Q_DEPTH FUNCTIONS
------------------------------------------------------------------------------------------------------
01 CEX5C CCA-Coproc online 1 0 0 11 08 0x92800000
01.0018 CEX5C CCA-Coproc online 1 0 0 11 08 0x92800000
02 CEX5C CCA-Coproc online 0 0 0 11 08 0x92800000
02.0018 CEX5C CCA-Coproc online 0 0 0 11 08 0x92800000
03 CEX5P EP11-Coproc online 0 0 0 11 08 0x06800000
03.0018 CEX5P EP11-Coproc online 0 0 0 11 08 0x06800000
05 CEX6C CCA-Coproc online 0 0 0 12 08 0x92800000
05.0018 CEX6C CCA-Coproc online 0 0 0 12 08 0x92800000

gdb output:
Program received signal SIGSEGV, Segmentation fault.
0x000003fffc9de6c0 in ?? ()
(gdb) bt
#0 0x000003fffc9de6c0 in ?? ()
#1 0x000003fffdb86f4c in C_GetTokenInfo (slotID=1, pInfo=0x3fffffff210) at api_interface.c:2491
#2 0x000002aa00005ba0 in display_token_info (slot_id=<optimized out>) at pkcsconf.c:839
#3 0x000002aa00002e16 in main (argc=<optimized out>, argv=0x3fffffff5d8) at pkcsconf.c:209

Please let us know if we can test a differently built package to track down - in that case please attach to defect or offer download location.
Thanks.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-16 12:13 EDT-------
To help in narrowing down In a next step I installed the libraries for CCA (csulcca_6.0.9-11_s390x.deb) and EP11 (libep11_1.3.2-1_s390x.deb) which are distributed by IBM. The pacakge-provided /etc/opencryptoki/opencryptoki.conf file is providing the TPM token in slot 0, ICA Token as slot 1, CCA token as slot 2, Soft Token as slot 3, and EP11 token in slot 4.
After restarting the pkcsslotd.service 'pkcsconf -t' did finish without segmentation fault, RC=0, but still display all tokens as ICA tokens.
Next, I tried to display the mechanism list, but got segfault for all but slot 4. Sample for slot 1:

# gdb /usr/sbin/pkcsconf
(gdb) run -m -c 1
Starting program: /usr/sbin/pkcsconf -m -c 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x000003fffc9eb33e in ep11tok_get_mechanism_list (tokdata=<optimized out>, pMechanismList=pMechanismList@entry=0x0,
pulCount=pulCount@entry=0x3fffffff118) at ep11_specific.c:4732
4732 ep11_specific.c: No such file or directory.
(gdb) bt
#0 0x000003fffc9eb33e in ep11tok_get_mechanism_list (tokdata=<optimized out>, pMechanismList=pMechanismList@entry=0x0,
pulCount=pulCount@entry=0x3fffffff118) at ep11_specific.c:4732
#1 0x000003fffc9de882 in SC_GetMechanismList (tokdata=<optimized out>, sid=sid@entry=1, pMechList=pMechList@entry=0x0,
count=count@entry=0x3fffffff118) at new_host.c:346
#2 0x000003fffdb87634 in C_GetMechanismList (slotID=1, pMechanismList=0x0, pulCount=0x3fffffff118) at api_interface.c:1997
#3 0x000002aa000050b0 in print_mech_info (slot_id=<optimized out>) at pkcsconf.c:597
#4 0x000002aa0000344a in display_mechanism_info (slot_id=<optimized out>) at pkcsconf.c:655
#5 main (argc=<optimized out>, argv=0x3fffffff438) at pkcsconf.c:224

[Dimitri John Ledkov (xnox)]

Timeline wise, my debugging below was performed before seeing comment #19.

So, running $ pkcsconf -t, under gdb is not helpful, as the segfault is in the non-existant/non-loaded object.
Running same command like so:
 LD_DEBUG=all LD_BIND_NOT=1 pkcsconf -t

Reveals in the debug output that:
    187548: file=libep11.so [0]; dynamically loaded by /usr/lib/s390x-linux-gnu/libpkcs11_ep11.so [0]
    187548: find library=libep11.so [0]; searching
    187548: search cache=/etc/ld.so.cache

However, libep11.so does not exist on Ubuntu. I would call this "dependency hell". It appears that libpkcs11_ep11.so requires dependencies that are not declared by the package in the Ubuntu archive, and which do not appear to be satisfiable from the Ubuntu archive. This error path is not nice, so I guess libpkcs11_ep11.so does not handle lack of libep11.so gracefully?

What is libep11.so and where is it supposed to come from? Since we do not appear like we have it in the Ubuntu archive, I guess we should not be shipping libpkcs11_ep11.so as it cannot be used at all =/ and causes crashes. I wonder if it is my mistake for packaging libpkcs11_ep11.so in the first place.

[Dimitri John Ledkov (xnox)]

(where comment #19 is launchpad bug number sequence, meaning comment from Christian.Rund @ 2018-05-16 11:43 EDT)

[Dimitri John Ledkov (xnox)]

Ah, and of course all the logging was in the journalctl!

May 16 12:27:02 s1lp14 pkcsconf[187270]: ep11_specific.c ep11tok_init: Error loading shared library 'libep11.so' [libep11.so: cannot open shared object file: No such file or directory]
May 16 12:27:02 s1lp14 kernel: User process fault: interruption code 0010 ilc:2 in var.lib.opencryptoki.lite[3ffbb500000+15000]
May 16 12:27:02 s1lp14 kernel: Failing address: 000003ffbb488000 TEID: 000003ffbb488800
May 16 12:27:02 s1lp14 kernel: Fault in primary space mode while using user ASCE.
May 16 12:27:02 s1lp14 kernel: AS:0000000478a081c7 R3:00000004d6dc4007 S:0000000000000020
May 16 12:27:02 s1lp14 kernel: CPU: 0 PID: 187270 Comm: pkcsconf Not tainted 4.15.0-20-generic #21-Ubuntu
May 16 12:27:02 s1lp14 kernel: Hardware name: IBM 2964 N63 400 (LPAR)
May 16 12:27:02 s1lp14 kernel: User PSW : 000000002896ffe7 000000006a87c495
May 16 12:27:02 s1lp14 kernel: R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:1 AS:0 CC:2 PM:0 RI:0 EA:3
May 16 12:27:02 s1lp14 kernel: User GPRS: 0000000000000000 000003ffbb4886c0 000002aa54515ad0 0000000000000001
May 16 12:27:02 s1lp14 kernel: 000003fff28fe6d8 00000000000009a9 000003ffbc614dbc 000003fff28fead0
May 16 12:27:02 s1lp14 kernel: 000002aa3da82a28 0000000000000000 0000000000000002 000003fff28fe538
May 16 12:27:02 s1lp14 kernel: 000003ffbca26000 0000000000000000 000003ffbc60bcca 000003fff28fe538
May 16 12:27:02 s1lp14 kernel: User Code: Bad PSW.
May 16 12:27:02 s1lp14 kernel: Last Breaking-Event-Address:
May 16 12:27:02 s1lp14 kernel: [<000003ffbc60bcc8>] 0x3ffbc60bcc8

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-16 13:14 EDT-------
Hi Dimitri,

No, libep11 is not the problem and the message that you are seeing on gdb is not the main reason.

This error message:
pkcsconf[187270]: ep11_specific.c ep11tok_init: Error loading shared library 'libep11.so' [libep11.so: cannot open shared object file: No such file or directory]

This is a common error that opencryptoki throws everytime that you have ep11 described on your /etc/opencryptoki/opencryptoki.conf. And throwing this error message doesn't cause segfault or other issue, this is just for logging so we know that you have ep11 in your conf file and that you don't have support of it.

Again, you can remove ep11 from the opencryptoki.conf file, or if you have a machine with the EP11 token and install the libep11 from IBM website, you will see that the output of pkcsconf -t is still wrong, as Christian pointed out.

To me, the problem is in the build process. Something is being done incorrectly or maybe the specific flags that you use on Ubuntu are affecting it somehow and making all the tokens to be recognized as ICA token.

If you want to be sure about what I am talking about EP11, build the project manually from source and install it, you will see the libep11 error message and you will see that everything still works.

I hope I was clear, if not please let me know.

[Dimitri John Ledkov (xnox)]

Three bugs.

bug 1 - on Ubuntu, pkcsconf -t results in segfault in default configuration, when executed without libep11.so installed. At least on my s1lp14 I see that. This is not good.

bug 2 - with ep11 token removed from the opencryptoki config, slots 1 2 3 are recognized as ICA; instead of ICA/CCA/SW. And no segfaults.

Which is very weird, because I would have expected the slot 2 to be empty for me, given that libcsulcca.so is not installed on my system, and thus it should not have been loaded at all....

bug 3 - even with extra deps from IBM, there are segfaults, e.g. run -m -c 1 (based on the message above from 2018-05-16 12:13 EDT by Christian Rund).

Some of the above may be duplicates.

Playing around with these things, it appears that whichever first token that is loaded "wins" and the rest of tokens duplicated it. with the following config:

version opencryptoki-3.1
slot 0
{
stdll = libpkcs11_tok.so
}
slot 1
{
stdll = libpkcs11_cca.so
}
slot 2
{
stdll = libpkcs11_cca.so
}
slot 3
{
stdll = libpkcs11_sw.so
}
slot 4
{
stdll = libpkcs11_ep11.so
confname = ep11tok.conf
}
slot 5
{
stdll = libpkcs11_ica.so
}

My output is:
sudo pkcsconf -t
Token #3 Info:
 Label: IBM OS PKCS#11
 Manufacturer: IBM Corp.
 Model: IBM SoftTok
 Serial Number: 123
 Flags: 0x880045 (RNG|LOGIN_REQUIRED|CLOCK_ON_TOKEN|USER_PIN_TO_BE_CHANGED|SO_PIN_TO_BE_CHANGED)
 Sessions: 0/18446744073709551614
 R/W Sessions: 18446744073709551615/18446744073709551614
 PIN Length: 4-8
 Public Memory: 0xFFFFFFFFFFFFFFFF/0xFFFFFFFFFFFFFFFF
 Private Memory: 0xFFFFFFFFFFFFFFFF/0xFFFFFFFFFFFFFFFF
 Hardware Version: 1.0
 Firmware Version: 1.0
 Time: 14:08:50
Token #5 Info:
 Label: IBM OS PKCS#11
 Manufacturer: IBM Corp.
 Model: IBM SoftTok
 Serial Number: 123
 Flags: 0x880045 (RNG|LOGIN_REQUIRED|CLOCK_ON_TOKEN|USER_PIN_TO_BE_CHANGED|SO_PIN_TO_BE_CHANGED)
 Sessions: 0/18446744073709551614
 R/W Sessions: 18446744073709551615/18446744073709551614
 PIN Length: 4-8
 Public Memory: 0xFFFFFFFFFFFFFFFF/0xFFFFFFFFFFFFFFFF
 Private Memory: 0xFFFFFFFFFFFFFFFF/0xFFFFFFFFFFFFFFFF
 Hardware Version: 1.0
 Firmware Version: 1.0
 Time: 14:08:50

Making "all tokens" to be SoftTok.

I'll debug stuff more, looks like initialisation of tokens is broken/not-cleared/reset correctly or something =/

[Dimitri John Ledkov (xnox)]

re: simple rebuilds
rebuilding opencryptoki, on ubuntu, using ubuntu toolchain & dependencies, does not improve things, and the crashes / missidentification of tokens is still reproducible for me.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-17 04:54 EDT-------
(In reply to comment #32)
Hello Dimitri,

as Eduardo also pointed out the ep11 lib is not causing this problem.
In a comment before I already mentioned where the CCA and EP11 libraries come from, please take into account.

> However, libep11.so does not exist on Ubuntu.
> ...
> What is libep11.so and where is it supposed to come from?
As already pointed out that library is distributed by IBM.
system # find / -name libep11.so
/usr/lib/libep11.so
system # dpkg -S /usr/lib/libep11.so
libep11: /usr/lib/libep11.so
system # dpkg -l libep11
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-========================-=================-=================-=====================================================
ii libep11 1.3.2-1 s390x EP11 host code
> I would call this "dependency hell". It appears that libpkcs11_ep11.so requires dependencies that are not
> declared by the package in the Ubuntu archive, and which do not appear to be
> satisfiable from the Ubuntu archive. This error path is not nice, so I guess
> libpkcs11_ep11.so does not handle lack of libep11.so gracefully?
...
> Since we do not appear like we have it in the Ubuntu archive, I guess we should not be
> shipping libpkcs11_ep11.so as it cannot be used at all =/ and causes
> crashes. I wonder if it is my mistake for packaging libpkcs11_ep11.so in the
> first place.
This is not true, I just cross-checked that Ubuntu 16.04.04 ( ) works with by the default configured CCA and EP11 tokens in the opencryptoki.conf file, when libep11 and csulcca are not installed. In that case one syslog entry per library appears to inform about that fact, which is not a problem.

------- Comment From <email address hidden> 2018-05-17 04:55 EDT-------
... Ubuntu 16.04.04 ( 4.4.0-124-generic ) that is.

[Dimitri John Ledkov (xnox)]

Rebuilding ubuntu's opencryptoki in debian chroot, appears to yield correctly working .debs
Starting to building ubuntu's opencryptoki in ubuntu chroot, and piece by piece cross-grading it into debian chroot (e.g. by piece-wise upgrading toolchain stack / dependencies / compilers / linkers / etc...) appears to yield this difference between working and non-working builds.

In ubuntu, dpkg-buildflags by default set LDFLAGS -Wl,-Bsymbolic-functions, whilst debian do not.
(e.g. observe the difference between, on any Ubuntu/Debian system:
DEB_VENDOR=Ubuntu dpkg-buildflags --query
DEB_VENDOR=Debian dpkg-buildflags --query
).

I guess this flag is incompatible with opencryptoki and how plugins/tokens are loaded. I will provide sample builds of opencryptoki with symbolic-functions LDFLAG stripped shortly.

[Dimitri John Ledkov (xnox)]

Proposed change:
https://launchpadlibrarian.net/370642983/opencryptoki_3.9.0+dfsg-0ubuntu1_3.9.0+dfsg-0ubuntu1.1.diff.gz

PPA for testing on bionic:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3264

To test:
sudo add-apt-repository ppa:ci-train-ppa-service/3264
sudo apt update
sudo apt install opencryptoki

And redo testing, as affected. For me, this now yields no crashes / correct token identification of ICA & SoftTok.

Direct links to debs can be found on e.g. s390x build page at:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3264/+build/14893360

Plain http files are also available from:
http://ppa.launchpad.net/ci-train-ppa-service/3264/ubuntu/pool/main/o/opencryptoki/

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-17 09:40 EDT-------
Hi Dimitri,

Here worked as well on my tests.

Thanks for fixing it!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-18 03:59 EDT-------
So did I get it right ? It was just the
-Wl,-Bsymbolic-functions
linker option ?

Even if this is fixed for now, we should spend some time on finding out why this linker option breaks the opencryptoki build. The man page says:

-Bsymbolic-functions
When creating a shared library, bind references to global function symbols to the definition within the shared library, if any....

This may lead to confusions when one token shared lib provides a function which is also provided by another token shared lib and/or the ock common code.
Eduardo, the practice in the tokens to build in some of the ock common files and to 'reimplement' some of these functions may need a review. Long term there may be another layering required.

regards
Harald Freudenberger

[Dimitri John Ledkov (xnox)]

Herald,

I've spoken to our toolchain maintainer ~doko. As far as I understood (which is very little, to be honest) this linker option prevents one from rebinding and changing symbol definitions once one implementation has been loaded. There have been previous cases of code being incompatible in Ubuntu. For example, https://bugs.launchpad.net/ubuntu/+source/xfe/+bug/644645. Also doko cited some python code, which a plugin wound re-bind a new definition of the function, which the original process would not start using as intended. So the fact that "first token implementation wins" and takes over all slots sounds like, opencrypoki is prevented to load and use the functions from subsequent token implementations. I'm not sure if it is reasonable to force tokens to use unique functions names, as most of token code is probably cargo-culted. I thought that it should be possible to rename/prefix the symbols at linktime or at dlopen time or some such. Cause things like fakeroot/faketime do have access to original function names, and new ones, and are able to use both simultaneously. Tracking this thing down, and improving opencryptoki to support linking with -Bsymbolic-functions is imho out of scope to be tracked here, and maybe we should move this to the upstream bug-tracker / upstream mailing list?

For Ubuntu, fixes to make opencryptoki support -Bsymbolic-functions is too invasive for an SRU, and for us, it is preferred to simply ship the "build opencryptoki without -Bsymbolic-functions" as a much smaller/safer SRU.

Regards,

Dimitri.

[Dimitri John Ledkov (xnox)]

opencryptoki (3.9.0+dfsg-0ubuntu2) cosmic; urgency=medium

  * Build without symbolic-functions, as that makes opencryptoki fail to
    load multiple tokens/plugins correctly.

 -- Dimitri John Ledkov <email address hidden> Thu, 17 May 2018 12:01:17 +0100

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-18 09:33 EDT-------
Hello Dimitri

clearly this bug is not the right place to track a re-evaluation and possible re-design of opencryptoki.

So far I understand this option it prevents some kind of attacks with preloaded shard libs which provide some functions and thus in fact act like a man-in-the-middle (e.g. think of overwriting exec or fork). So no wonder why opencryptoki has problems with this as some tokens 'overwrite' some of the opencrptoki common code.

However, we had problems with this already in the past as you have to be carefull with linking the objects together for the token shared lib. It would be nice to have a better solution which does not collide with the -Bsymbolic-functions linker option but as of now I have only some vague ideas.

regards
Harald

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-18 10:41 EDT-------
Hi Harald,

I agree that we should investigate this, so let's keep this discussion going outside here.

@Dimitri, just out of curiosity, that Xfe ticket that you mentioned is from 2010. So the "-Bsymbolic-functions" is on Ubuntu since this time right?

Then, and just to confirm, is it right to assume that this linking option was included in opencryptoki package only in Ubuntu 17.10?

I am asking this because I want to be sure that we are not missing any other factor.

[Steve Langasek (vorlon)]

On Fri, May 18, 2018 at 09:34:42AM -0000, Dimitri John Ledkov wrote:

> I'm not sure if it is reasonable to force tokens to use unique functions
> names, as most of token code is probably cargo-culted. I thought that it
> should be possible to rename/prefix the symbols at linktime or at dlopen
> time or some such. Cause things like fakeroot/faketime do have access to
> original function names, and new ones, and are able to use both
> simultaneously.

This is generally done by accessing the symbols exclusively via a dlopen
handle, instead of loading two sets of overlapping symbols into the global
namespace.

[Adam Conrad (adconrad)]

Hello bugproxy, or anyone else affected,

Accepted opencryptoki into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/opencryptoki/3.7.0+dfsg-4ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Conrad (adconrad)]

Hello bugproxy, or anyone else affected,

Accepted opencryptoki into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/opencryptoki/3.9.0+dfsg-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Frank Heimes (frank-heimes)]

Verification successfully done - set the tags accordingly.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for opencryptoki has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package opencryptoki - 3.7.0+dfsg-4ubuntu1

---------------
opencryptoki (3.7.0+dfsg-4ubuntu1) artful; urgency=medium

  * Build without symbolic-functions, as that makes opencryptoki fail to
    load multiple tokens/plugins correctly. LP: #1725250

 -- Dimitri John Ledkov <email address hidden> Fri, 25 May 2018 13:14:12 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package opencryptoki - 3.9.0+dfsg-0ubuntu1.1

---------------
opencryptoki (3.9.0+dfsg-0ubuntu1.1) bionic; urgency=medium

  * Build without symbolic-functions, as that makes opencryptoki fail to
    load multiple tokens/plugins correctly. LP: #1725250

 -- Dimitri John Ledkov <email address hidden> Thu, 17 May 2018 12:01:17 +0100

[Dimitri John Ledkov (xnox)]

On 18 May 2018 at 15:51, bugproxy <email address hidden> wrote:
> ------- Comment From <email address hidden> 2018-05-18 10:41 EDT-------
> Hi Harald,
>
> I agree that we should investigate this, so let's keep this discussion
> going outside here.
>
> @Dimitri, just out of curiosity, that Xfe ticket that you mentioned is
> from 2010. So the "-Bsymbolic-functions" is on Ubuntu since this time
> right?
>

No.

> Then, and just to confirm, is it right to assume that this linking
> option was included in opencryptoki package only in Ubuntu 17.10?
>

No.

> I am asking this because I want to be sure that we are not missing any
> other factor.
>

I believe -Bsymbolic-functions was on since Ubuntu's inception in
2004. But I'm failing to find documentation of this flag on e.g.
https://wiki.ubuntu.com/ToolChain/CompilerFlags

@ matthias, could you please provide comments about
symbolic-functions? Should it be added to
https://wiki.ubuntu.com/ToolChain/CompilerFlags ?

--
Regards,

Dimitri.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-02 05:10 EDT-------
IBM Bugzilla status -> closed, Fix Released for Artful, Bionic, Cosmic