[SRU] segmentation fault for all https operations in libcrypto.so.1.0.0 on 'legacy' Intel Xeon CPUs

[Darik Horn]

[IMPACT]

https operations that make use of openssl/libcrypto on specific legacy Intel CPUs results in SIGSEGV. This includes everything from utilities like wget and curl, to mail servers and everything in between.

[TESTCASE]

curl or wget a URL via the HTTPS, observe segfault. One user reports this is reproducible on Intel Xeon(TM) CPU 3.40GHz cpu family: 15.

[DEVELOPMENT FIX]

This bug has been fixed upstream since April 2012, see http://cvs.openssl.org/chngview?cn=22415. The upstream fix has been applied and fixed in Debain since openssl-1.0.1a-3, and in Ubuntu 12.10 since the 1.0.1c-3ubuntu1 sync.

[Regression Potential]
Low, relatively trivial patch that only uses 4_hmac_md56_cipher if the architecture is capable.

>> Original Bug <<

All `s3cmd` commands fail with a Segmentation Fault on Ubuntu 12.04 Precise Pangolin if `use_https = True` is set in the `$HOME/.s3cfg` file.

$ apt-cache policy s3cmd
s3cmd:
  Installed: 1.0.0-1
  Candidate: 1.0.0-1
  Version table:
 *** 1.0.0-1 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages
        100 /var/lib/dpkg/status

$ s3cmd du
DEBUG: Updating Config.Config encoding -> UTF-8
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 30
DEBUG: Unicodising 'du' using UTF-8
DEBUG: Command: du
DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Wed, 04 Apr 2012 20:33:06 +0000\n/'
DEBUG: CreateRequest: resource[uri]=/
DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Wed, 04 Apr 2012 20:33:06 +0000\n/'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(None): s3.amazonaws.com
DEBUG: format_uri(): /
Segmentation fault

$ gdb python
GNU gdb (Ubuntu/Linaro 7.4-2012.02-0ubuntu2) 7.4-2012.02
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /usr/bin/python...(no debugging symbols found)...done.
(gdb) run /usr/bin/s3cmd du
Starting program: /usr/bin/python /usr/bin/s3cmd du
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff721b031 in RC4 () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(gdb) backtrace
#0 0x00007ffff721b031 in RC4 () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#1 0x00000000000000db in ?? ()
#2 0x000000000000009f in ?? ()
#3 0x0000000000ed91f0 in ?? ()
#4 0x00007ffff7281609 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#5 0x00007ffff7583b0f in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#6 0x00007ffff757adb8 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#7 0x00007ffff757b0e4 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#8 0x000000000056f7ec in ?? ()
#9 0x0000000000562ad2 in PyEval_EvalFrameEx ()
#10 0x000000000056a166 in PyEval_EvalCodeEx ()
#11 0x000000000056312e in PyEval_EvalFrameEx ()
#12 0x000000000056a166 in PyEval_EvalCodeEx ()
#13 0x000000000056312e in PyEval_EvalFrameEx ()
#14 0x0000000000563070 in PyEval_EvalFrameEx ()
#15 0x000000000056a166 in PyEval_EvalCodeEx ()
#16 0x000000000056312e in PyEval_EvalFrameEx ()
#17 0x000000000056a166 in PyEval_EvalCodeEx ()
#18 0x000000000056312e in PyEval_EvalFrameEx ()
#19 0x0000000000563070 in PyEval_EvalFrameEx ()
#20 0x000000000056a166 in PyEval_EvalCodeEx ()
#21 0x000000000056312e in PyEval_EvalFrameEx ()
#22 0x000000000056a166 in PyEval_EvalCodeEx ()
#23 0x000000000056312e in PyEval_EvalFrameEx ()
#24 0x0000000000563070 in PyEval_EvalFrameEx ()
#25 0x0000000000563070 in PyEval_EvalFrameEx ()
#26 0x0000000000563070 in PyEval_EvalFrameEx ()
#27 0x0000000000563070 in PyEval_EvalFrameEx ()
#28 0x000000000056a166 in PyEval_EvalCodeEx ()
#29 0x000000000055f10b in ?? ()
#30 0x000000000055f9d4 in PyRun_FileExFlags ()
#31 0x000000000055fc41 in PyRun_SimpleFileExFlags ()
#32 0x000000000056102d in Py_Main ()
#33 0x00007ffff68e576d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#34 0x000000000041b971 in _start ()

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Roman Vorobets (roman-vorobets)]

The problem is actually in the libssl version that ships with Ubuntu 12.04.
I also have segfault in libcrypto when using ejabberd with SSL turned on.

[Brett Anderson (bzdzb)]

I am getting a strikingly similar backtrace when doing an https post using php5-curl.

Program received signal SIGSEGV, Segmentation fault.
0x00007f67f2db9031 in RC4 () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(gdb) backtrace
#0 0x00007f67f2db9031 in RC4 () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#1 0x00000000000003a4 in ?? ()
#2 0x000000000000035f in ?? ()
#3 0x00007f67fa75e020 in ?? ()
#4 0x00007f67f2e1f629 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#5 0x00007f67f3121bcf in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#6 0x00007f67f3118e04 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#7 0x00007f67f3119134 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#8 0x00007f67f14fc1fe in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#9 0x00007f67f14ee0e1 in Curl_write () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#10 0x00007f67f14eacb6 in Curl_add_buffer_send () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#11 0x00007f67f14ec87d in Curl_http () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#12 0x00007f67f14fb238 in Curl_do () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#13 0x00007f67f15062e7 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#14 0x00007f67f17443f2 in ?? () from /usr/lib/php5/20090626/curl.so
#15 0x00007f67f3e8b81d in ?? () from /usr/lib/apache2/modules/libphp5.so
#16 0x00007f67f3e3c38b in execute () from /usr/lib/apache2/modules/libphp5.so
#17 0x00007f67f3e178f0 in zend_execute_scripts () from /usr/lib/apache2/modules/libphp5.so
#18 0x00007f67f3dc3fa3 in php_execute_script () from /usr/lib/apache2/modules/libphp5.so
#19 0x00007f67f3ea6dfd in ?? () from /usr/lib/apache2/modules/libphp5.so
#20 0x00007f67f839a508 in ap_run_handler ()
#21 0x00007f67f839a97e in ap_invoke_handler ()
#22 0x00007f67f83aa570 in ap_process_request ()
#23 0x00007f67f83a7398 in ?? ()
#24 0x00007f67f83a0fa8 in ap_run_process_connection ()
#25 0x00007f67f83af1d0 in ?? ()
#26 0x00007f67f83af93a in ?? ()
#27 0x00007f67f83b04e7 in ap_mpm_run ()
#28 0x00007f67f83854a4 in main ()

[Brett Anderson (bzdzb)]

Both curl and wget segfault:
$ wget https://secure.authorize.net/gateway/transact.dll
$ curl https://secure.authorize.net/gateway/transact.dll

They both use libcrypto.so.1.0.0 (/lib/x86_64-linux-gnu/libcrypto.so.1.0.0) and have similar backtraces:
Program received signal SIGSEGV, Segmentation fault.
RC4 () at rc4-x86_64.s:343
343 rc4-x86_64.s: No such file or directory.
(gdb) backtra
#0 RC4 () at rc4-x86_64.s:343
#1 0x000000000000009c in ?? ()
#2 0x000000000000005f in ?? ()
#3 0x0000000000697bb0 in ?? ()
#4 0x00007ffff78a5629 in rc4_hmac_md5_cipher

This seems related to http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666405
Also, it looks like there may be an upstream patch according to http://old.nabble.com/-openssl.org--2792--Crash-in-rc4-on-x86_64-td33702502.html

[Brett Anderson (bzdzb)]

I compiled openssl-1.0.1c and have verified that this issue is resolved upstream.

[Ante Karamatić (ivoks)]

Link to an upstream fix: http://cvs.openssl.org/chngview?cn=22415

[Ante Karamatić (ivoks)]

This is not s3cmd, php or dovecot bug. It's a bug in openssl.

[Nicola (nicola)]

hi,

any updates about this? Will a new openssl package be included in 12.04.1? For now I'm using my custom openssl package that apply the needed patch and they works fine (my problem was with dovecot), I think would be useful for most users to apply the patch in the official LTS packages, any eta?

thanks
Nicola

[Adam Gandelman (gandelman-a)]

Attached is a debdiff against openssl 1.0.1-4ubuntu5.3 with the upstream fix applied.

[Clint Byrum (clint-fewbar)]

I uploaded this to precise-proposed yesterday BTW, forgot to comment. Its waiting for SRU team review.

[Clint Byrum (clint-fewbar)]

Hello Darik, or anyone else affected,

Accepted openssl into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/openssl/1.0.1-4ubuntu5.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Nicola (nicola)]

Hi,

it works for me thanks! My problem was with dovecot,

Nicola

[Adam Conrad (adconrad)]

Removing this SRU from -proposed, it broke on all non-x86 builds.

[Adam Conrad (adconrad)]

A more complete fix (that would work on non-x86 builds) could be the diff between 1.2 and 1.5:

http://cvs.openssl.org/filediff?f=openssl/crypto/evp/e_rc4_hmac_md5.c&v1=1.2&v2=1.5

[Adam Gandelman (gandelman-a)]

Attached is a new debdiff with the full change history applied. Built successfully on ARM and PPC, as well as built and tested on amd64. Please accept into -proposed. Thanks.

[Adam Conrad (adconrad)]

Hello Darik, or anyone else affected,

Accepted openssl into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/openssl/1.0.1-4ubuntu5.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Conrad (adconrad)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 1.0.1-4ubuntu5.5

---------------
openssl (1.0.1-4ubuntu5.5) precise-proposed; urgency=low

  * debian/patches/lp973741.patch: Apply complete and more recent changeset,
    which fixes original issue on Intel CPUs and fixes FTBFS on non-x86
    architectures. (LP: #973741)

openssl (1.0.1-4ubuntu5.4) precise-proposed; urgency=low

  * debian/patches/lp973741.patch: Avoid segfault on legacy Intel CPUs
    by using correct cypher. (LP: #973741)
 -- Adam Gandelman <email address hidden> Tue, 14 Aug 2012 17:44:51 +0100

[Richard Huddleston (rhuddusa)]

just information for other people, I just encountered this bug in deluge (deluged)

gdb output

Core was generated by `/usr/bin/python /usr/bin/deluged -p 9998 --loglevel=error --logfile=/dev/null -'.
Program terminated with signal 11, Segmentation fault.
#0 RC4 () at rc4-x86_64.s:309
309 rc4-x86_64.s: No such file or directory.

uname -a
Linux box01 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

dpkg -l | grep deluge
ii deluge-common 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (common files)
ii deluged 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (daemon)

lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 23
Stepping: 10
CPU MHz: 1998.000
BogoMIPS: 4987.54
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 6144K
NUMA node0 CPU(s): 0-7

[Richard Huddleston (rhuddusa)]

forgot to add
Intel(R) Xeon(R) CPU E5420 @ 2.50GHz

[Richard Huddleston (rhuddusa)]

i believe i am getting this error through python 2.7 and libssl0.9.8

apt-cache rdepends --installed libssl0.9.8
libssl0.9.8
Reverse Depends:
  python2.7-minimal
  libpython2.7

[Matthew White (mtw)]

I'm experiencing this bug with deluge on AMD:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff2bd0700 (LWP 30364)]
RC4 () at rc4-x86_64.s:128
128 rc4-x86_64.s: No such file or directory.

$ uname -a
Linux kuma 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 16
Model: 6
Stepping: 3
CPU MHz: 800.000
BogoMIPS: 2995.07
Virtualization: AMD-V
L1d cache: 64K
L1i cache: 64K
L2 cache: 1024K
NUMA node0 CPU(s): 0,1

$ dpkg -l | egrep '(deluge|libssl)'
ii deluge 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK
ii deluge-common 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (common files)
ii deluge-console 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (console ui)
ii deluge-gtk 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (GTK+ ui)
ii deluge-web 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (web ui)
ii deluged 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (daemon)
ii libssl1.0.0 1.0.1-4ubuntu5.5 SSL shared libraries

[Friedrich Gonzalez (friedrichg)]

I am getting the same error with deluge in Intel 64 bits

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff2e0c700 (LWP 22903)]
0x00007ffff721adde in RC4 () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(gdb) bt
#0 0x00007ffff721adde in RC4 () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#1 0x0000000000000000 in ?? ()

$ uname -a
Linux hp 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 15
Stepping: 11
CPU MHz: 1600.000
BogoMIPS: 4799.89
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
NUMA node0 CPU(s): 0-3

$ dpkg -l | egrep '(deluge|libssl)'
ii deluge 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK
ii deluge-common 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (common files)
ii deluge-gtk 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (GTK+ ui)
ii deluge-web 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (web ui)
ii deluged 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (daemon)
ii libssl-dev 1.0.1-4ubuntu5.5 SSL development libraries, header files and documentation
ii libssl-doc 1.0.1-4ubuntu5.5 SSL development documentation documentation
ii libssl0.9.8 0.9.8o-7ubuntu3.1 SSL shared libraries
ii libssl0.9.8:i386 0.9.8o-7ubuntu3.1 SSL shared libraries
ii libssl1.0.0 1.0.1-4ubuntu5.5 SSL shared libraries
ii libssl1.0.0:i386 1.0.1-4ubuntu5.5 SSL shared libraries

[claw (claw-111)]

Same segfault with deluge involving libcrypto and RC4(). What's worse is I think ubuntu-bug isn't reporting the regression on this bug at all?!

$ uname -a
Linux atrus 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 37
Stepping: 5
CPU MHz: 1199.000
BogoMIPS: 5319.88
Virtualisation: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 3072K
NUMA node0 CPU(s): 0-3

$ dpkg -l | grep -E '(deluge|openssl|libssl)'
ii deluge-common 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (common files)
ii deluge-gtk 1.3.5-0ubuntu2~precise2 bittorrent client written in Python/PyGTK (GTK+ ui)
ii libgnutls-openssl27 2.12.14-5ubuntu3.1 GNU TLS library - OpenSSL wrapper
ii libssl1.0.0 1.0.1-4ubuntu5.5 SSL shared libraries
ii openssl 1.0.1-4ubuntu5.5 Secure Socket Layer (SSL) binary and related cryptographic tools
ii python-openssl 0.12-1ubuntu2 Python wrapper around the OpenSSL library

[Arceliar (littlebuzz71)]

Just wanted to confirm that this is still an issue (at least on my end)

Also, it's yet another CPU family: 6 GenuineIntel system. Seems like there's a pattern.

Linux Laptop-E530 3.2.0-39-generic #62-Ubuntu SMP Thu Feb 28 00:28:53 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Stepping: 9
CPU MHz: 1200.000
BogoMIPS: 4389.83
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-7

$ dpkg -l | grep -E '(openssl|libssl)'
ii libssl-dev 1.0.1-4ubuntu5.8 SSL development libraries, header files and documentation
ii libssl-doc 1.0.1-4ubuntu5.8 SSL development documentation documentation
ii libssl0.9.8 0.9.8o-7ubuntu3.1 SSL shared libraries
ii libssl1.0.0 1.0.1-4ubuntu5.8 SSL shared libraries
ii libssl1.0.0:i386 1.0.1-4ubuntu5.8 SSL shared libraries
ii openssl 1.0.1-4ubuntu5.8 Secure Socket Layer (SSL) binary and related cryptographic tools
ii python-openssl 0.12-1ubuntu2 Python wrapper around the OpenSSL library

[claw (claw-111)]

For anyone using deluge and still experiencing this regression, I have filed bug 1232311.

[mmalmeida (mmalmeida)]

claw - that bug doesn't exist.