segfault from aes ccm encryption after RSA key generation and EVP_PKEY_assign_RSA()

[Jens Hiller]

1) lsb_release -rd
Description: Ubuntu 12.04.4 LTS
Release: 12.04

2) apt-cache policy libssl1.0.0
libssl1.0.0:
  Installed: 1.0.1-4ubuntu5.13
  Candidate: 1.0.1-4ubuntu5.13
  Version table:
 *** 1.0.1-4ubuntu5.13 0
        500 http://be.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.1-4ubuntu3 0
        500 http://be.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

For the sourcecode that triggers the segfault see the attachement.

3/4)

I was testing aes ccm encryption when I stumbled over a segmentation fault.
I was able to reproduce this error using code from the openssl demos at openssl.org.

I started with demos/evp/aesccm.c and added rsa key generation as used in 'demos/tunala/cb.c' and convert this rsa key into an EVP_PKEY key as done in 'demos/selfsign.c'. Then I added this rsa key generation function in front of the aes ccm encryption and decryption.
Finally, a for loop repeatedly performs the keygeneration, aes ccm encryption and aes ccm decryption. This eventually results in a segmentation fault during aes ccm encryption (see gdb output below) on a x64 Ubuntu 12.04 with latest openssl version as provided by ubuntu package system (1.0.1-4ubuntu5.13).

Note that the segfault only occurs if the rsa key is assigned to an EVP_PKEY. Otherwise, if only the RSA key is generated, the segfault does not occur.
Furthermore, the segfault does not occur if I use the standard openssl libraries from openssl.org.

When encountering this error in my own code I could observe that the error occurred more often on a machine that only runs the standard processes and is accessed remotely by ssh, compared to a local workstation with running webbrowser, development IDE, etc., where the error occurred rather seldom. Hence, I have the feeling that this could be related to too little
randomness for the RNG, but I do not have any idea how to debug this.

=== gdb backtrace ===

(gdb) run
Starting program: /home/hiller/openssl_bug/aesccm
AES CCM Encrypt:
Plaintext:
0000 - c8 d2 75 f9 19 e1 7d 7f-e6 9c 2a 1f 58 93 9d fe ..u...}...*.X...
0010 - 4d 40 37 91 b5 df 13 10- M@7.....
Ciphertext:
0000 - 8a 0f 3d 82 29 e4 8e 74-87 fd 95 a2 8a d3 92 c8 ..=.)..t........
0010 - 0b 36 81 d4 fb c7 bb fd- .6......
Tag:
0000 - 2d d6 ef 1c 45 d4 cc b7-23 dc 07 44 14 db 50 6d -...E...#..D..Pm
AES CCM Derypt:
Ciphertext:
0000 - 8a 0f 3d 82 29 e4 8e 74-87 fd 95 a2 8a d3 92 c8 ..=.)..t........
0010 - 0b 36 81 d4 fb c7 bb fd- .6......
Plaintext:
0000 - c8 d2 75 f9 19 e1 7d 7f-e6 9c 2a 1f 58 93 9d fe ..u...}...*.X...
0010 - 4d 40 37 91 b5 df 13 10- M@7.....
AES CCM Encrypt:
[ the output above is repeated several times ]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000090 in ?? ()
(gdb) backtrace
#0 0x0000000000000090 in ?? ()
#1 0x00007ffff7a948d4 in CRYPTO_ccm128_encrypt_ccm64 (ctx=0x604fd0,
inp=0x401240
"\310\322u\371\031\341}\177\346\234*\037X\223\235\376M@7\221\265\337\023\020",

    out=0x7fffffffe0c0
"\310\322u\371\031\341}\177\346\234*\037X\223\235\376M@7\221\265\337\023\020",
len=24, stream=<optimized out>) at ccm128.c:354
#2 0x00007ffff7af1688 in aes_ccm_cipher (ctx=0x604e10,
out=0x7fffffffe0c0
"\310\322u\371\031\341}\177\346\234*\037X\223\235\376M@7\221\265\337\023\020",

    in=0x401240
"\310\322u\371\031\341}\177\346\234*\037X\223\235\376M@7\221\265\337\023\020",
len=24) at e_aes.c:1275
#3 0x00007ffff7aedaa2 in EVP_EncryptUpdate (ctx=0x604e10,
out=0x7fffffffe0c0
"\310\322u\371\031\341}\177\346\234*\037X\223\235\376M@7\221\265\337\023\020",
outl=0x7fffffffe0bc,
    in=0x401240
"\310\322u\371\031\341}\177\346\234*\037X\223\235\376M@7\221\265\337\023\020",
inl=<optimized out>) at evp_enc.c:314
#4 0x0000000000400e37 in aes_ccm_encrypt () at aesccm.c:106
#5 0x00000000004010ce in main (argc=1, argv=0x7fffffffe5e8) at aesccm.c:161

[Adrien Nader (adrien)]

Thanks for the report and for the reproducer. I haven't been able to trigger a segfault despite numerous attempts. I'll therefore mark this bug as Incomplete for now.

[Launchpad Janitor (janitor)]

[Expired for openssl (Ubuntu) because there has been no activity for 60 days.]