stunnel4: "INTERNAL ERROR: Bad magic at ssl.c, line 117" - DoS vulnerability

[Malcolm Scott]

On multiple machines running Ubuntu 18.04 (stunnel4 3:5.44-1ubuntu3), I am experiencing stunnel crashes seemingly caused by an attacker sending an invalid handshake of some sort.

Aug 23 14:23:23 callisto stunnel[6302]: LOG5[599]: Service [btsync] accepted connection from ::ffff:23.225.177.161:61844
Aug 23 14:23:24 callisto stunnel[6302]: INTERNAL ERROR: Bad magic at ssl.c, line 117

Oct 07 18:21:10 elara stunnel[5718]: LOG5[1173]: Service [btsync] accepted connection from ::ffff:172.247.55.206:52036
Oct 07 18:21:11 elara stunnel[5718]: INTERNAL ERROR: Bad magic at ssl.c, line 117

Oct 07 21:07:40 callisto stunnel[15207]: LOG5[343]: Service [btsync] accepted connection from ::ffff:23.225.121.126:58374
Oct 07 21:07:40 callisto stunnel[15207]: INTERNAL ERROR: Bad magic at ssl.c, line 117

I suspect this to be an intentional (and successful) denial-of-service attack.

Please let me know what other information I can usefully provide.

[Malcolm Scott (malcscott)]

(Report made public since it's being actively exploited.)

[Marc Deslauriers (mdeslaur)]

Thanks for reporting this issue.

What's the output of stunnel4 -v?
Which openssl packages are you using?

[Malcolm Scott (malcscott)]

$ stunnel4 -v
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g 2 Nov 2017
[.] Running with OpenSSL 1.1.1 11 Sep 2018
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)

libssl1.1 version: 1.1.1-1ubuntu2.1~18.04.4

[Marc Deslauriers (mdeslaur)]

Do you know if you had the problem with the openssl 1.1.0 packages that were in bionic before we released 1.1.1?

[Malcolm Scott (malcscott)]

Hmm, good question. 1.1.1 was pushed to bionic in June? Unfortunately I don't think I have syslog going back that far on any affected machine. I only recall this problem happening in the last few months.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[David Smoot (davidsmoot)]

I can confirm I am also experiencing this issue on 18.04.

 stunnel4 -v
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g 2 Nov 2017
[.] Running with OpenSSL 1.1.1b 26 Feb 2019
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)

$ sudo more stunnel.log.1
2019.10.25 05:06:34 LOG3[217]: TLS socket closed (SSL_write) with 18432 unsent byte(s)
2019.10.25 05:25:04 LOG3[221]: TLS socket closed (SSL_write) with 2274 unsent byte(s)
2019.10.25 06:21:56 LOG3[231]: TLS socket closed (SSL_write) with 6993 unsent byte(s)
2019.10.25 06:39:36 LOG3[233]: TLS socket closed (SSL_read) with 15138 unsent byte(s)
2019.10.25 07:59:49 LOG3[241]: TLS socket closed (SSL_write) with 18432 unsent byte(s)
2019.10.25 08:22:01 LOG3[243]: TLS socket closed (SSL_write) with 1117 unsent byte(s)
2019.10.25 08:37:57 LOG3[245]: TLS socket closed (SSL_write) with 1591 unsent byte(s)
2019.10.25 09:37:20 LOG3[251]: TLS socket closed (SSL_write) with 18432 unsent byte(s)
2019.10.25 09:50:53 LOG3[253]: TLS socket closed (SSL_write) with 11950 unsent byte(s)
INTERNAL ERROR: Bad magic at ssl.c, line 117

I do not know any history of when the bug started, I only recently set up stunnel in the last couple months.

[John KLO (johnklo)]

I have the same problem on two Ubuntu 18.04 hosts.
The error does not occur immediately. Sometimes it takes several days.

Reboot (systemctl restart stunnel4.service) helps.

stunnel.log: INTERNAL ERROR: Bad magic at ssl.c, line 117

# stunnel4 -v
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g 2 Nov 2017
[.] Running with OpenSSL 1.1.1 11 Sep 2018
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)

[Brett Keller (blkeller)]

ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.8
Architecture: amd64
DistroRelease: Ubuntu 18.04
Package: stunnel4 3:5.44-1ubuntu3
PackageArchitecture: amd64
ProcEnviron:
 LANG=C.UTF-8
 TERM=xterm
 PATH=(custom, no user)
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 4.15.0-65.74-generic 4.15.18
Tags: bionic uec-images
Uname: Linux 4.15.0-65-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
modified.conffile..etc.default.stunnel4: [modified]
mtime.conffile..etc.default.stunnel4: 2019-11-04T17:53:44.054718

[Brett Keller (blkeller)]

apport information

[Brett Keller (blkeller)]

apport information

[Brett Keller (blkeller)]

Stacktrace:
 #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
         set = {__val = {0, 140337533684800, 140725801169072, 94865733453904, 140725801169056, 15442105031219824384, 4, 94865733453904, 0, 15442105031219824384, 4, 1, 94865733453904, 1, 4, 94865708102691}}
         pid = <optimized out>
         tid = <optimized out>
         ret = <optimized out>
 #1 0x00007fa2e09d5801 in __GI_abort () at abort.c:79
         save_stage = 1
         act = {__sigaction_handler = {sa_handler = 0x5647a656bd10, sa_sigaction = 0x5647a656bd10}, sa_mask = {__val = {94865708083362, 1572911363, 206158430224, 140725801169664, 94865708014912, 17179869207, 17179869201, 511101108234, 1318554959873, 94862942666752, 18446744073709530016, 94865733327264, 94865707998636, 94865708111963, 140725801169488, 94865708083362}}, sa_flags = -1529528963, sa_restorer = 0x75}
         sigs = {__val = {32, 0 <repeats 15 times>}}
         __cnt = <optimized out>
         __set = <optimized out>
         __cnt = <optimized out>
         __set = <optimized out>
 #2 0x00005647a4d3e3d0 in ?? ()
 No symbol table info available.
 #3 0x00005647a4d395b3 in ?? ()
 No symbol table info available.
 #4 0x00005647a4d39809 in ?? ()
 No symbol table info available.
 #5 0x00005647a4d39e44 in ?? ()
 No symbol table info available.
 #6 0x00007fa2e17ab7c8 in CRYPTO_free_ex_data (class_index=class_index@entry=2, obj=obj@entry=0x7fa2d0019f90, ad=ad@entry=0x7fa2d001a0e0) at ../crypto/ex_data.c:354
         mx = <optimized out>
         i = 2
         ip = 0x7fa2e1aff6f0 <ex_data+16>
         ptr = <optimized out>
         f = 0x5647a656b430
         stack = {0x0, 0x5647a656b3a0, 0x5647a656b430, 0xd64d60b5ca214700, 0x7fa2d000bc80, 0x5647a65715f0, 0x7fa2d0019f90, 0x7fa2e17ab2c8 <CRYPTO_get_ex_data+24>, 0x5, 0x5647a65715f0}
         storage = 0x7ffd4763b370
 #7 0x00007fa2e1b3fe58 in SSL_SESSION_free (ss=0x7fa2d0019f90) at ../ssl/ssl_sess.c:787
         i = <optimized out>
         ss = 0x7fa2d0019f90
         i = <optimized out>
         i = <optimized out>
 #8 0x00007fa2e17af3fc in doall_util_fn (arg=0x7ffd4763b460, arg@entry=0x7ffd4763b420, func_arg=func_arg@entry=0x7fa2e1b406f0 <timeout_cb>, func=0x0, use_arg=1, lh=0x5647a6551d40) at ../crypto/lhash/lhash.c:196
         i = 5
         a = <optimized out>
         n = 0x7fa2c8012800
         i = <optimized out>
         a = <optimized out>
         n = <optimized out>
 #9 OPENSSL_LH_doall_arg (lh=0x5647a6551d40, func=func@entry=0x7fa2e1b406f0 <timeout_cb>, arg=arg@entry=0x7ffd4763b460) at ../crypto/lhash/lhash.c:211
 No locals.
 #10 0x00007fa2e1b414e7 in lh_SSL_SESSION_doall_TIMEOUT_PARAM (arg=0x7ffd4763b460, fn=0x7fa2e1b406f0 <timeout_cb>, lh=<optimized out>) at ../ssl/ssl_sess.c:1104
 No locals.
 #11 SSL_CTX_flush_sessions (s=0x5647a65715f0, t=<optimized out>) at ../ssl/ssl_sess.c:1119
         i = <optimized out>
         tp = {ctx = 0x5647a65715f0, time = 1572911664, cache = 0x5647a6551d40}
 #12 0x00005647a4d4d080 in ?? ()
 No symbol table info available.
 #13 0x00005647a4d4d109 in ?? ()
 No symbol table info available.
 #14 0x00005647a4d38c56 in ?? ()
 No symbol table info available.
 #15 0x00007fa2e09b6b97 in __...

[John KLO (johnklo)]

Hello!
Is there a chance that this error will be fixed?
Or is it better to upgrade to Ubuntu 19.10?

[PETITJEAN Valentin (vpetitje)]

Same problem on Ubuntu 18.04 host :

stunnel4 -v
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g 2 Nov 2017
[.] Running with OpenSSL 1.1.1 11 Sep 2018
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)

In the log file :

INTERNAL ERROR: Bad magic at ssl.c, line 117

[Yasunobu Nohara (yasunohara)]

I had same problem on Ubuntu 18.04 host.

stunnel4 -v
[ ] Clients allowed=500
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.0g 2 Nov 2017
[.] Running with OpenSSL 1.1.1 11 Sep 2018
[.] Update OpenSSL shared libraries or rebuild stunnel
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)

I found "Double free attempt" messages before INTERNAL ERROR in the log file.

2019.12.09 21:21:06 LOG5[ui]: Terminated
2019.12.09 21:21:06 LOG2[ui]: Double free attempt: ptr=0x7f48b000e680 alloc=client.c:1398 free#1=client.c:1410 free#2=ssl.c:117
2019.12.09 21:21:06 LOG2[ui]: Double free attempt: ptr=0x7f48ac007610 alloc=client.c:1398 free#1=client.c:1410 free#2=ssl.c:117
INTERNAL ERROR: Bad magic at ssl.c, line 117
-----
2019.12.10 12:26:02 LOG5[ui]: Terminated
2019.12.10 12:26:02 LOG2[ui]: Double free attempt: ptr=0x7f8e3c005480 alloc=../crypto/evp/digest.c:51 free#1=../crypto/evp/digest.c:57 free#2=ssl.c:117
INTERNAL ERROR: Bad magic at ssl.c, line 117

[Brett Keller (blkeller)]

I too see this problem with stunnel on a public-facing Ubuntu 18.04 server. In addition to the stack trace with debugging symbols that I provided earlier, I now have a packet capture of one of the TLS sessions that caused a crash.

I was lucky enough to catch the attacker in the act, and in addition to capturing packets on the wire, I was also logging TLS session premaster keys. I've injected the relevant ephemeral key material (*not* my server private key!) into the capture so that we can all see inside the TLS tunnel.

I'm not a TLS expert, but nothing jumped out at me as being obviously malformed or exploitative in the TLS negotiation, though second opinions are very welcome. Looking inside the tunnel, the attacker appears to be trying to begin some sort of WordPress-based vulnerability test. Stunnel crashed before it could connect to the local service I have it in front of, so this HTTP GET request was never delivered downstream. For what it's worth, I'm not running WordPress, so that vulnerability scan would have come up empty anyway.

Please let me know if there's any more information I can provide regarding this packet capture or my server configuration.

[Scott Ogrin (tcott)]

I've been seeing the same problem for months. Same exact stunnel4 -v as posted above.

[Brett Keller (blkeller)]

In addition to my earlier packet capture and stack trace, I can now add a detailed debug log from a different occurrence of this crash. See attachment.

I turned the log level on stunnel all the way up to the maximum (debug=7) and left things running until another attack hit my server. This log only covers 30 seconds, but in that time stunnel handled four different incoming connections, the last of which ended in a crash.

It's probably worth noting that right before the crash there's a big chunk of repetitive message pairs: "Remove session callback" followed by "Deallocating application specific data for session connect address", repeated in tandem until the crash with "INTERNAL ERROR: Bad magic at ssl.c, line 117".

Hope that helps, and please let me know if there's any more information I can provide.

[Juan Amores (juanamm)]

This thread takes several months without solution for Ubuntu 18.04 users.
I recently requested a backport to be able to update Stunnel to its latest version.
Stunnel crash continuously and I think the update would help fix this problem.
Please Bionic Backports I request to give priority to my request, which will benefit all of us who continue to use Bionic. https://bugs.launchpad.net/bionic-backports/+bug/1884279

[Scott Ogrin (tcott)]

Just install stunnel 5.56 from source. It's actually pretty painless. I got tired of waiting and wrote a quick guide:
https://scottiestech.info/2020/06/23/fix-stunnel-bad-magic-at-ssl-c-error-crash-on-ubuntu/

[Juan Amores (juanamm)]

I made a more radical decision, I upgraded to 20.04 LTS (focal), installed Stunnel 5.56 and so far I have not had any crashes.
For those who want and can do this upgrade, I recommend them, since not only is Stunnel updated, but also OpenSSL.
For newbies like me I share the commands:

sudo apt update
sudo apt upgrade
sudo do-release-upgrade -d

After that the Stunnel looks like this:

root@ip-xxxxx:/home/ubuntu# stunnel -v
[ ] Clients allowed=500
[.] stunnel 5.56 on x86_64-pc-linux-gnu platform
[.] Compiled with OpenSSL 1.1.1c 28 May 2019
[.] Running with OpenSSL 1.1.1f 31 Mar 2020
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[!] Invalid configuration file name "-v"
[!] realpath: No such file or directory (2)
[ ] Deallocating section defaults

[Nick (ngaugler)]

Did everyone die? How is it that this is marked as undecided and not fixed? The software crashes and fails to function. If SSH crashed and failed to function I am sure that this would not be left undecided. So is the answer to uninstall Ubuntu and install CentOS or Debian, maybe even Windows?

[Steve Arnold (nerdboy)]

Seriously, this is almost a year old. What is the resolution? Where are the upgraded pkgs? Leaving it to some unknown number of users to devise their own out-of-band solution seems like a really bad idea when all we really need is a current stunnel package built against the current openssl lib. At least one helpful user has documented their workaround above, but the method is not what I wanted when I installed an LTS (bionic) release. Now I get to go hand-build pkgs on a production server. yay.

[Chul-Woong Yang (cwyang)]

Hi. I've been encoutering this issue on my OpenSSL-1.1.1 application.
Do you have any workaround? I found that SSL_SESSION list is corrupted
but I don't have any SSL_SESSSION related code in my app.

You can notice that symptom is same from the following backtrace:

/usr/local/bin/worker[0x509a21]
/usr/local/bin/worker(OPENSSL_LH_doall_arg+0x4f)[0x62e65f]
/usr/local/bin/worker(SSL_CTX_flush_sessions+0x57)[0x50abb7]
/usr/local/bin/worker(tls_finish_handshake+0x193)[0x520763]
/usr/local/bin/worker[0x5160c2]
/usr/local/bin/worker(async_start_func+0x2c)[0x55b58c]

[Steve Arnold (nerdboy)]

Would anyone like to test the 5.56 version with upstream patches? I have it backported to bionic and focal here:

https://launchpad.net/~nerdboy/+archive/ubuntu/embedded/+packages

[Sam Stenvall (negge)]

Ran into the same problem. The service never recovers after this error, because the package is not shipped with a systemd unit, but with an old /etc/init.d/stunnel4 script.

I ended up uninstalling the Ubuntu version, compling 5.60 (latest at the time) from source and then running it with a makeshift systemd service (available at https://gist.github.com/Jalle19/60532513bbcec2fa630a9d8b576d91e3).

[Malcolm Scott (malcscott)]

I'm not convinced that CVE-2021-20230 is the same bug.

[Lars Kollstedt (lk-x)]

I also think CVE-2021-20230 and this bug are probably two different things. But Steve Arnold is also addressing CVE-2021-20230 in Comment#25, and it's still considered unfixed on https://ubuntu.com/security/CVE-2021-20230. So there is a a relation to this CVE, but CVE-2021-20230 is not describing this bug.

This Bug should be worth a CVE, but I did't find one really describing this, yet. I'm trying one of Steves Arnolds Packages, now. Since I was experiencing crashes due this bug almost every day.

[Juan Amores (juanamm)]

Hello everyone and especially the admins of this list.
I have tried to unsubscribe from this list and I can't do it because it
asks me for the password.
I have already requested a reminder of my password but the email does not
arrive.

Please tell me the steps to follow with this.

Thanks a lot.

El mar, 21 dic 2021 a las 11:05, Lars Kollstedt (<email address hidden>)
escribió:

> I also think CVE-2021-20230 and this bug are probably two different
> things. But Steve Arnold is also addressing CVE-2021-20230 in
> Comment#25, and it's still considered unfixed on
> https://ubuntu.com/security/CVE-2021-20230. So there is a a relation to
> this CVE, but CVE-2021-20230 is not describing this bug.
>
> This Bug should be worth a CVE, but I did't find one really describing
> this, yet. I'm trying one of Steves Arnolds Packages, now. Since I was
> experiencing crashes due this bug almost every day.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1847275
>
> Title:
> stunnel4: "INTERNAL ERROR: Bad magic at ssl.c, line 117" - DoS
> vulnerability
>
> Status in stunnel4 package in Ubuntu:
> Confirmed
>
> Bug description:
> On multiple machines running Ubuntu 18.04 (stunnel4 3:5.44-1ubuntu3),
> I am experiencing stunnel crashes seemingly caused by an attacker
> sending an invalid handshake of some sort.
>
> Aug 23 14:23:23 callisto stunnel[6302]: LOG5[599]: Service [btsync]
> accepted connection from ::ffff:23.225.177.161:61844
>
> Aug 23 14:23:24 callisto stunnel[6302]: INTERNAL ERROR: Bad magic at
> ssl.c, line 117
>
> Oct 07 18:21:10 elara stunnel[5718]: LOG5[1173]: Service [btsync]
> accepted connection from ::ffff:172.247.55.206:52036
>
> Oct 07 18:21:11 elara stunnel[5718]: INTERNAL ERROR: Bad magic at ssl.c,
> line 117
>
> Oct 07 21:07:40 callisto stunnel[15207]: LOG5[343]: Service [btsync]
> accepted connection from ::ffff:23.225.121.126:58374
>
> Oct 07 21:07:40 callisto stunnel[15207]: INTERNAL ERROR: Bad magic at
> ssl.c, line 117
>
> I suspect this to be an intentional (and successful) denial-of-service
> attack.
>
> Please let me know what other information I can usefully provide.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/stunnel4/+bug/1847275/+subscriptions
>
>

[Lars Kollstedt (lk-x)]

Steve Arnolds package for bionic from https://bugs.launchpad.net/ubuntu/+source/stunnel4/+bug/1847275/comments/25 works for me.

[David Tauriainen (david-tauriainen)]

Can confirm that stunnel4 (5.44-1ubuntu3) crashes every few hours on a 18.04 LTS system (about to be upgraded) when connected from a 22.04 LTS system (5.63-1build1). The network between the two only allows their respective packets, and the 22.04 LTS system was recently a 18.04 LTS system. The 18.04 LTS didn't have stunnel4 crash ever until the other system upgraded. I was going to upgrade the 18.04 LTS anyway, but this bug report thread convinced me to upgrade ASAP (I'd rather not rely on restarting after every crash via systemd unit file).
If you want to reproduce the error, try an 18.04 VM and 22.04, and start a data transfer. Shouldn't take too long.