stunnel4: "INTERNAL ERROR: Bad magic at ssl.c, line 117" - DoS vulnerability

Bug #1847275 reported by Malcolm Scott on 2019-10-08
322
This bug affects 13 people
Affects Status Importance Assigned to Milestone
stunnel4 (Ubuntu)
Undecided
Unassigned

Bug Description

On multiple machines running Ubuntu 18.04 (stunnel4 3:5.44-1ubuntu3), I am experiencing stunnel crashes seemingly caused by an attacker sending an invalid handshake of some sort.

Aug 23 14:23:23 callisto stunnel[6302]: LOG5[599]: Service [btsync] accepted connection from ::ffff:23.225.177.161:61844
Aug 23 14:23:24 callisto stunnel[6302]: INTERNAL ERROR: Bad magic at ssl.c, line 117

Oct 07 18:21:10 elara stunnel[5718]: LOG5[1173]: Service [btsync] accepted connection from ::ffff:172.247.55.206:52036
Oct 07 18:21:11 elara stunnel[5718]: INTERNAL ERROR: Bad magic at ssl.c, line 117

Oct 07 21:07:40 callisto stunnel[15207]: LOG5[343]: Service [btsync] accepted connection from ::ffff:23.225.121.126:58374
Oct 07 21:07:40 callisto stunnel[15207]: INTERNAL ERROR: Bad magic at ssl.c, line 117

I suspect this to be an intentional (and successful) denial-of-service attack.

Please let me know what other information I can usefully provide.

Malcolm Scott (malcscott) wrote :

(Report made public since it's being actively exploited.)

information type: Private Security → Public Security
Marc Deslauriers (mdeslaur) wrote :

Thanks for reporting this issue.

What's the output of stunnel4 -v?
Which openssl packages are you using?

Changed in stunnel4 (Ubuntu):
status: New → Incomplete
Malcolm Scott (malcscott) wrote :

$ stunnel4 -v
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g 2 Nov 2017
[.] Running with OpenSSL 1.1.1 11 Sep 2018
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)

libssl1.1 version: 1.1.1-1ubuntu2.1~18.04.4

Marc Deslauriers (mdeslaur) wrote :

Do you know if you had the problem with the openssl 1.1.0 packages that were in bionic before we released 1.1.1?

Changed in stunnel4 (Ubuntu):
status: Incomplete → New
Malcolm Scott (malcscott) wrote :

Hmm, good question. 1.1.1 was pushed to bionic in June? Unfortunately I don't think I have syslog going back that far on any affected machine. I only recall this problem happening in the last few months.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in stunnel4 (Ubuntu):
status: New → Confirmed
David Smoot (davidsmoot) wrote :

I can confirm I am also experiencing this issue on 18.04.

 stunnel4 -v
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g 2 Nov 2017
[.] Running with OpenSSL 1.1.1b 26 Feb 2019
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)

$ sudo more stunnel.log.1
2019.10.25 05:06:34 LOG3[217]: TLS socket closed (SSL_write) with 18432 unsent byte(s)
2019.10.25 05:25:04 LOG3[221]: TLS socket closed (SSL_write) with 2274 unsent byte(s)
2019.10.25 06:21:56 LOG3[231]: TLS socket closed (SSL_write) with 6993 unsent byte(s)
2019.10.25 06:39:36 LOG3[233]: TLS socket closed (SSL_read) with 15138 unsent byte(s)
2019.10.25 07:59:49 LOG3[241]: TLS socket closed (SSL_write) with 18432 unsent byte(s)
2019.10.25 08:22:01 LOG3[243]: TLS socket closed (SSL_write) with 1117 unsent byte(s)
2019.10.25 08:37:57 LOG3[245]: TLS socket closed (SSL_write) with 1591 unsent byte(s)
2019.10.25 09:37:20 LOG3[251]: TLS socket closed (SSL_write) with 18432 unsent byte(s)
2019.10.25 09:50:53 LOG3[253]: TLS socket closed (SSL_write) with 11950 unsent byte(s)
INTERNAL ERROR: Bad magic at ssl.c, line 117

I do not know any history of when the bug started, I only recently set up stunnel in the last couple months.

John KLO (johnklo) wrote :

I have the same problem on two Ubuntu 18.04 hosts.
The error does not occur immediately. Sometimes it takes several days.

Reboot (systemctl restart stunnel4.service) helps.

stunnel.log: INTERNAL ERROR: Bad magic at ssl.c, line 117

# stunnel4 -v
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g 2 Nov 2017
[.] Running with OpenSSL 1.1.1 11 Sep 2018
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)

ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.8
Architecture: amd64
DistroRelease: Ubuntu 18.04
Package: stunnel4 3:5.44-1ubuntu3
PackageArchitecture: amd64
ProcEnviron:
 LANG=C.UTF-8
 TERM=xterm
 PATH=(custom, no user)
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 4.15.0-65.74-generic 4.15.18
Tags: bionic uec-images
Uname: Linux 4.15.0-65-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
modified.conffile..etc.default.stunnel4: [modified]
mtime.conffile..etc.default.stunnel4: 2019-11-04T17:53:44.054718

tags: added: apport-collected bionic uec-images

apport information

apport information

Brett Keller (blkeller) wrote :
Download full text (11.4 KiB)

Stacktrace:
 #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
         set = {__val = {0, 140337533684800, 140725801169072, 94865733453904, 140725801169056, 15442105031219824384, 4, 94865733453904, 0, 15442105031219824384, 4, 1, 94865733453904, 1, 4, 94865708102691}}
         pid = <optimized out>
         tid = <optimized out>
         ret = <optimized out>
 #1 0x00007fa2e09d5801 in __GI_abort () at abort.c:79
         save_stage = 1
         act = {__sigaction_handler = {sa_handler = 0x5647a656bd10, sa_sigaction = 0x5647a656bd10}, sa_mask = {__val = {94865708083362, 1572911363, 206158430224, 140725801169664, 94865708014912, 17179869207, 17179869201, 511101108234, 1318554959873, 94862942666752, 18446744073709530016, 94865733327264, 94865707998636, 94865708111963, 140725801169488, 94865708083362}}, sa_flags = -1529528963, sa_restorer = 0x75}
         sigs = {__val = {32, 0 <repeats 15 times>}}
         __cnt = <optimized out>
         __set = <optimized out>
         __cnt = <optimized out>
         __set = <optimized out>
 #2 0x00005647a4d3e3d0 in ?? ()
 No symbol table info available.
 #3 0x00005647a4d395b3 in ?? ()
 No symbol table info available.
 #4 0x00005647a4d39809 in ?? ()
 No symbol table info available.
 #5 0x00005647a4d39e44 in ?? ()
 No symbol table info available.
 #6 0x00007fa2e17ab7c8 in CRYPTO_free_ex_data (class_index=class_index@entry=2, obj=obj@entry=0x7fa2d0019f90, ad=ad@entry=0x7fa2d001a0e0) at ../crypto/ex_data.c:354
         mx = <optimized out>
         i = 2
         ip = 0x7fa2e1aff6f0 <ex_data+16>
         ptr = <optimized out>
         f = 0x5647a656b430
         stack = {0x0, 0x5647a656b3a0, 0x5647a656b430, 0xd64d60b5ca214700, 0x7fa2d000bc80, 0x5647a65715f0, 0x7fa2d0019f90, 0x7fa2e17ab2c8 <CRYPTO_get_ex_data+24>, 0x5, 0x5647a65715f0}
         storage = 0x7ffd4763b370
 #7 0x00007fa2e1b3fe58 in SSL_SESSION_free (ss=0x7fa2d0019f90) at ../ssl/ssl_sess.c:787
         i = <optimized out>
         ss = 0x7fa2d0019f90
         i = <optimized out>
         i = <optimized out>
 #8 0x00007fa2e17af3fc in doall_util_fn (arg=0x7ffd4763b460, arg@entry=0x7ffd4763b420, func_arg=func_arg@entry=0x7fa2e1b406f0 <timeout_cb>, func=0x0, use_arg=1, lh=0x5647a6551d40) at ../crypto/lhash/lhash.c:196
         i = 5
         a = <optimized out>
         n = 0x7fa2c8012800
         i = <optimized out>
         a = <optimized out>
         n = <optimized out>
 #9 OPENSSL_LH_doall_arg (lh=0x5647a6551d40, func=func@entry=0x7fa2e1b406f0 <timeout_cb>, arg=arg@entry=0x7ffd4763b460) at ../crypto/lhash/lhash.c:211
 No locals.
 #10 0x00007fa2e1b414e7 in lh_SSL_SESSION_doall_TIMEOUT_PARAM (arg=0x7ffd4763b460, fn=0x7fa2e1b406f0 <timeout_cb>, lh=<optimized out>) at ../ssl/ssl_sess.c:1104
 No locals.
 #11 SSL_CTX_flush_sessions (s=0x5647a65715f0, t=<optimized out>) at ../ssl/ssl_sess.c:1119
         i = <optimized out>
         tp = {ctx = 0x5647a65715f0, time = 1572911664, cache = 0x5647a6551d40}
 #12 0x00005647a4d4d080 in ?? ()
 No symbol table info available.
 #13 0x00005647a4d4d109 in ?? ()
 No symbol table info available.
 #14 0x00005647a4d38c56 in ?? ()
 No symbol table info available.
 #15 0x00007fa2e09b6b97 in __...

John KLO (johnklo) wrote :

Hello!
Is there a chance that this error will be fixed?
Or is it better to upgrade to Ubuntu 19.10?

PETITJEAN Valentin (vpetitje) wrote :

Same problem on Ubuntu 18.04 host :

stunnel4 -v
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g 2 Nov 2017
[.] Running with OpenSSL 1.1.1 11 Sep 2018
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)

In the log file :

INTERNAL ERROR: Bad magic at ssl.c, line 117

Yasunobu Nohara (yasunohara) wrote :

I had same problem on Ubuntu 18.04 host.

stunnel4 -v
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g 2 Nov 2017
[.] Running with OpenSSL 1.1.1 11 Sep 2018
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)

I found "Double free attempt" messages before INTERNAL ERROR in the log file.

2019.12.09 21:21:06 LOG5[ui]: Terminated
2019.12.09 21:21:06 LOG2[ui]: Double free attempt: ptr=0x7f48b000e680 alloc=client.c:1398 free#1=client.c:1410 free#2=ssl.c:117
2019.12.09 21:21:06 LOG2[ui]: Double free attempt: ptr=0x7f48ac007610 alloc=client.c:1398 free#1=client.c:1410 free#2=ssl.c:117
INTERNAL ERROR: Bad magic at ssl.c, line 117
-----
2019.12.10 12:26:02 LOG5[ui]: Terminated
2019.12.10 12:26:02 LOG2[ui]: Double free attempt: ptr=0x7f8e3c005480 alloc=../crypto/evp/digest.c:51 free#1=../crypto/evp/digest.c:57 free#2=ssl.c:117
INTERNAL ERROR: Bad magic at ssl.c, line 117

Brett Keller (blkeller) wrote :

I too see this problem with stunnel on a public-facing Ubuntu 18.04 server. In addition to the stack trace with debugging symbols that I provided earlier, I now have a packet capture of one of the TLS sessions that caused a crash.

I was lucky enough to catch the attacker in the act, and in addition to capturing packets on the wire, I was also logging TLS session premaster keys. I've injected the relevant ephemeral key material (*not* my server private key!) into the capture so that we can all see inside the TLS tunnel.

I'm not a TLS expert, but nothing jumped out at me as being obviously malformed or exploitative in the TLS negotiation, though second opinions are very welcome. Looking inside the tunnel, the attacker appears to be trying to begin some sort of WordPress-based vulnerability test. Stunnel crashed before it could connect to the local service I have it in front of, so this HTTP GET request was never delivered downstream. For what it's worth, I'm not running WordPress, so that vulnerability scan would have come up empty anyway.

Please let me know if there's any more information I can provide regarding this packet capture or my server configuration.

Scott Ogrin (tcott) wrote :

I've been seeing the same problem for months. Same exact stunnel4 -v as posted above.

Brett Keller (blkeller) wrote :

In addition to my earlier packet capture and stack trace, I can now add a detailed debug log from a different occurrence of this crash. See attachment.

I turned the log level on stunnel all the way up to the maximum (debug=7) and left things running until another attack hit my server. This log only covers 30 seconds, but in that time stunnel handled four different incoming connections, the last of which ended in a crash.

It's probably worth noting that right before the crash there's a big chunk of repetitive message pairs: "Remove session callback" followed by "Deallocating application specific data for session connect address", repeated in tandem until the crash with "INTERNAL ERROR: Bad magic at ssl.c, line 117".

Hope that helps, and please let me know if there's any more information I can provide.

Juan Amores (juanamm) wrote :

This thread takes several months without solution for Ubuntu 18.04 users.
I recently requested a backport to be able to update Stunnel to its latest version.
Stunnel crash continuously and I think the update would help fix this problem.
Please Bionic Backports I request to give priority to my request, which will benefit all of us who continue to use Bionic. https://bugs.launchpad.net/bionic-backports/+bug/1884279

Scott Ogrin (tcott) wrote :

Just install stunnel 5.56 from source. It's actually pretty painless. I got tired of waiting and wrote a quick guide:
https://scottiestech.info/2020/06/23/fix-stunnel-bad-magic-at-ssl-c-error-crash-on-ubuntu/

Juan Amores (juanamm) wrote :

I made a more radical decision, I upgraded to 20.04 LTS (focal), installed Stunnel 5.56 and so far I have not had any crashes.
For those who want and can do this upgrade, I recommend them, since not only is Stunnel updated, but also OpenSSL.
For newbies like me I share the commands:

sudo apt update
sudo apt upgrade
sudo do-release-upgrade -d

After that the Stunnel looks like this:

root@ip-xxxxx:/home/ubuntu# stunnel -v
[ ] Clients allowed=500
[.] stunnel 5.56 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.1c 28 May 2019
[.] Running with OpenSSL 1.1.1f 31 Mar 2020
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)
[ ] Deallocating section defaults

Nick (ngaugler) wrote :

Did everyone die? How is it that this is marked as undecided and not fixed? The software crashes and fails to function. If SSH crashed and failed to function I am sure that this would not be left undecided. So is the answer to uninstall Ubuntu and install CentOS or Debian, maybe even Windows?

To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers