[UBUNTU 22.10] opencryptoki C_GenerateKeyPair() fails after generating > 500 RSA keys with CEX7 and CEX8 crypto cards

[bugproxy]

---Problem Description---
opencryptoki C_GenerateKeyPair() fails after generating > 500 RSA keys with CEX7 crypto cards

Running a program that generates RSA keys and tests encyrption/decryption on Crypto cards, is failing after about 500 iterations. The program makes the following opencryptoki calls in a loop:

C_GenerateKeyPair()
C_EncryptInit()
C_Encrypt()
C_DecryptInit()
C_Decrypt()
C_SignInit()
C_Sign() (RSA Sign)
C_VerifyInit()
C_Verify() (RSA Verify)
C_SignInit()
C_SignUpdate() (RSA Multipart sign)
C_VerifyInit()
C_VerifyUpdate() (RSA multi-part verify)
C_VerifyFinal()
C_DestroyObject() (Destroy RSA public key)
C_DestroyObject() (Destroy RSA private key)

When running single-threaded, it fails consistently on loop 504. When running multi-threaded it fails on about the 506th iteration across all threads (ie with 5 threads, each thread fails after about 100 loops).

On the 504th loop, the C_GenerateKeyPair() call fails with return code 6. Looking at the /var/log/opencryptoki/trace.* logfile, I found these error messages which seem to correspond to the failure:

```
09/06/2022 15:37:09 6099 [usr/lib/api/api_interface.c:1106 api] DEVEL: fcn->ST_DestroyObject returned:0x0
09/06/2022 15:37:09 6099 [usr/lib/api/api_interface.c:1996 api] INFO: C_GenerateKeyPair
09/06/2022 15:37:09 6099 [usr/lib/api/api_interface.c:2019 api] INFO: Valid Session handle id: 2
09/06/2022 15:37:09 6099 [usr/lib/ep11_stdll/ep11_specific.c:5977 ep11tok] INFO: rsa_ec_generate_keypair m_GenerateKeyPair rc=0x0 spki_len=0x1a6 privkey_blob_len=0x970 mech='CKM_RSA_PKCS_KEY_PAIR_GEN'
09/06/2022 15:37:09 6099 [usr/lib/ep11_stdll/ep11_specific.c:6612 ep11tok] INFO: ep11tok_generate_key_pair rc=0x0 hpubkey=0x1 hprivkey=0x2 pub_name='' priv_name='' pub_obj=0x3ff7001d210 priv_obj=0x3ff70068460
09/06/2022 15:37:09 6099 [usr/lib/common/loadsave.c:149 ep11tok] DEVEL: Unable to set permissions on file.
09/06/2022 15:37:09 6099 [usr/lib/common/loadsave.c:149 ep11tok] DEVEL: Unable to set permissions on file.
09/06/2022 15:37:09 6099 [usr/lib/common/loadsave.c:149 ep11tok] DEVEL: Unable to set permissions on file.
09/06/2022 15:37:09 6099 [usr/lib/common/obj_mgr.c:613 ep11tok] DEVEL: Object created: handle: 2
09/06/2022 15:37:09 6099 [usr/lib/common/loadsave.c:2409 ep11tok] ERROR: fopen(/var/lib/opencryptoki/ep11tok/TOK_OBJ/OBxIH3zb): Too many open files
09/06/2022 15:37:09 6099 [usr/lib/ep11_stdll/ep11_specific.c:6749 ep11tok] DEVEL: ep11tok_generate_key_pair Object mgr create final failed
09/06/2022 15:37:09 6099 [usr/lib/common/obj_mgr.c:991 ep11tok] DEVEL: Object found: handle: 2
09/06/2022 15:37:09 6099 [usr/lib/common/loadsave.c:224 ep11tok] ERROR: fopen failed
09/06/2022 15:37:09 6099 [usr/lib/ep11_stdll/new_host.c:3837 ep11tok] DEVEL: ep11tok_generate_key_pair() failed.
09/06/2022 15:37:09 6099 [usr/lib/ep11_stdll/new_host.c:3840 ep11tok] INFO: C_GenerateKeyPair: rc = 0x00000006, sess = 2, mech = 0x0
09/06/2022 15:37:09 6099 [usr/lib/api/api_interface.c:2041 api] DEVEL: fcn->ST_GenerateKeyPair returned:0x6
09/06/2022 15:37:09 6099 [usr/lib/api/api_interface.c:1653 api] INFO: C_Finalize
```

I tested it on both z15 and z16 (with different crypto cards) and got the same failure. In all cases, I am running on a KVM guest and using CEX7 crypto cards. I have not tested yet on CEX8 crypto cards.

---
External link: https://warthogs.atlassian.net/browse/PEI-29

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2022-09-14 07:52 EDT-------
====== Comment by Ingo: ========
For the record, this bug was introduced with the following commit
https://github.com/opencryptoki/opencryptoki/commit/5d0d588e391ab2f69bc6218b921bca092055606e
which went into OCK 3.18.0. So only Distros with 3.18.0 are affected by this problem.

------- Comment From <email address hidden> 2022-09-14 07:53 EDT-------
== Comment: <email address hidden> ==
I would assume that this is fixed by the following commit:
https://github.com/opencryptoki/opencryptoki/commit/d5ccb00e52f5b0c66533f085cda36f63f7583d44

Without this fix, any creation of a new token object will leak one open file. So after so many new token objects, the limit of open files is reached, and it starts to fail.

[Frank Heimes (fheimes)]

Hello and thx for reporting this.

There is a snapshot version in jammy/22.04:
3.17.0+dfsg+20220202.b40982e-0ubuntu1.1
This is obviously based on 3.17, BUT has a lot of commits on top (until date 20220202 and commit b40982e).
So I looked up both commits manually (that's why I don't like packages based on snapshots) and both commits are NOT in - hence jammy is not affected.

For kinetic/22.10 things are much clearer.
Commit 5d0d588 is in, thus d5ccb00 is needed.

So I worked on an updated package for kinetic that is currently building here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1989558

[Frank Heimes (fheimes)]

Builds (and their logs) are looking good and package installs fine.

Uploading the debdiff (see attachment).

[Simon Chopin (schopin)]

Uploaded. I just changed the bug number in the changelog, it was pointing to another opencryptoki bug :)

[Frank Heimes (fheimes)]

Many thx Simon - ops, probably a c&p issue - thx for having that fixed!

[Launchpad Janitor (janitor)]

This bug was fixed in the package opencryptoki - 3.18.0+dfsg-0ubuntu2

---------------
opencryptoki (3.18.0+dfsg-0ubuntu2) kinetic; urgency=medium

  * Add quilt patch
    d/p/lp-1989558-common-fix-memory-leak-in-save_private_token_object.patch
    to fix a failure of opencryptoki C_GenerateKeyPair() that occurs after
    having generated more that 500 RSA keys with CEX7 and CEX8 crypto cards.
    (LP: #1989558)

 -- Frank Heimes <email address hidden> Wed, 14 Sep 2022 15:01:02 +0200