Bug #1951279 “OpenSSL 1.1.1f raise a segmentation faults on Arm6...” : Focal (20.04) : Bugs : openssl package : Ubuntu

Revision history for this message

Juan (juanparati) wrote on 2021-11-17:

#1

I observed that many users are affected by this bug. (See: https://github.com/curl/curl/issues/8024)

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2021-11-17:

#2

Download full text (6.4 KiB)

Can you provide more information on your environment and how to reproduce this? I wasn't able to reproduce this on my rpi3b+ running focal, with either libssl1.1 1.1.1f-1ubuntu2.8 or 1.1.1f-1ubuntu2.9:

First, 1.1.1f-1ubuntu2.8 installed:

$ curl -v https://graph.facebook.com/v12.0/act_111/
* Trying 157.240.3.20:443...
* TCP_NODELAY set
* Connected to graph.facebook.com (157.240.3.20) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=Menlo Park; O=Facebook, Inc.; CN=*.facebook.com
* start date: Nov 4 00:00:00 2021 GMT
* expire date: Feb 2 23:59:59 2022 GMT
* subjectAltName: host "graph.facebook.com" matched cert's "*.facebook.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xaaaac4c9dee0)
> GET /v12.0/act_111/ HTTP/2
> Host: graph.facebook.com
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 403
< vary: Origin
< x-ad-account-usage: {"acc_id_util_pct":0}
< x-fb-rlafr: 0
< content-type: application/json; charset=UTF-8
< www-authenticate: OAuth "Facebook Platform" "insufficient_scope" "(#200) Provide valid app ID"
< access-control-allow-origin: *
< facebook-api-version: v12.0
< strict-transport-security: max-age=15552000; preload
< pragma: no-cache
< cache-control: no-store
< expires: Sat, 01 Jan 2000 00:00:00 GMT
< x-fb-request-id: AYFxZKGuw4Uidu_b6_RsyRn
< x-fb-trace-id: C1HBc2Oi1S3
< x-fb-rev: 1004746171
< x-fb-debug: yza+SwSrqD6mY1INQSyb5rcHmU89PziSoE3txYwg1BjWybYcgB36mUMVxq9bsRAJXZGkc34nNcSps5APpyG8QA==
< content-length: 125
< date: Wed, 17 Nov 2021 20:48:02 GMT
< alt-svc: h3=":443"; ma=3600, h3-29=":443"; ma=3600
<
* Connection #0 to host graph.facebook.com left intact
{"error":{"message":"(#200) Provide valid app ID","type":"OAuthException","code":200,"fbtrace_id":"AYFxZKGuw4Uidu_b6_RsyRn"}}ubuntu@ubuntu:~ $ wget https://graph.facebook.com/v12.0/act_111/
--2021-11-17 20:48:16-- https://graph.facebook.com/v12.0/act_111/
Resolving graph.facebook.com (graph.facebook.com)... 157.240.3.20, 2a03:2880:f001:6:face:b00c:0:2
Connecting to graph.facebook.com (graph.facebook.com)|157.240.3.20|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021...

Can you provide more information on your environment and how to reproduce this? I wasn't able to reproduce this on my rpi3b+ running focal, with either libssl1.1 1.1.1f-1ubuntu2.8 or 1.1.1f-1ubuntu2.9:

First, 1.1.1f-1ubuntu2.8 installed:

$ curl -v https://graph.facebook.com/v12.0/act_111/
*   Trying 157.240.3.20:443...
* TCP_NODELAY set
* Connected to graph.facebook.com (157.240.3.20) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Menlo Park; O=Facebook, Inc.; CN=*.facebook.com
*  start date: Nov  4 00:00:00 2021 GMT
*  expire date: Feb  2 23:59:59 2022 GMT
*  subjectAltName: host "graph.facebook.com" matched cert's "*.facebook.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xaaaac4c9dee0)
> GET /v12.0/act_111/ HTTP/2
> Host: graph.facebook.com
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 403 
< vary: Origin
< x-ad-account-usage: {"acc_id_util_pct":0}
< x-fb-rlafr: 0
< content-type: application/json; charset=UTF-8
< www-authenticate: OAuth "Facebook Platform" "insufficient_scope" "(#200) Provide valid app ID"
< access-control-allow-origin: *
< facebook-api-version: v12.0
< strict-transport-security: max-age=15552000; preload
< pragma: no-cache
< cache-control: no-store
< expires: Sat, 01 Jan 2000 00:00:00 GMT
< x-fb-request-id: AYFxZKGuw4Uidu_b6_RsyRn
< x-fb-trace-id: C1HBc2Oi1S3
< x-fb-rev: 1004746171
< x-fb-debug: yza+SwSrqD6mY1INQSyb5rcHmU89PziSoE3txYwg1BjWybYcgB36mUMVxq9bsRAJXZGkc34nNcSps5APpyG8QA==
< content-length: 125
< date: Wed, 17 Nov 2021 20:48:02 GMT
< alt-svc: h3=":443"; ma=3600, h3-29=":443"; ma=3600
< 
* Connection #0 to host graph.facebook.com left intact
{"error":{"message":"(#200) Provide valid app ID","type":"OAuthException","code":200,"fbtrace_id":"AYFxZKGuw4Uidu_b6_RsyRn"}}ubuntu@ubuntu:~ $ wget https://graph.facebook.com/v12.0/act_111/
--2021-11-17 20:48:16--  https://graph.facebook.com/v12.0/act_111/
Resolving graph.facebook.com (graph.facebook.com)... 157.240.3.20, 2a03:2880:f001:6:face:b00c:0:2
Connecting to graph.facebook.com (graph.facebook.com)|157.240.3.20|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-11-17 20:48:16 ERROR 403: Forbidden.

ubuntu@ubuntu:~ 8 $

Next, 1.1.1f-1ubuntu2.9 installed:

ubuntu@ubuntu:~ 10s $ curl -v https://graph.facebook.com/v12.0/act_111/
*   Trying 157.240.3.20:443...
* TCP_NODELAY set
* Connected to graph.facebook.com (157.240.3.20) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Menlo Park; O=Facebook, Inc.; CN=*.facebook.com
*  start date: Nov  4 00:00:00 2021 GMT
*  expire date: Feb  2 23:59:59 2022 GMT
*  subjectAltName: host "graph.facebook.com" matched cert's "*.facebook.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xaaaaf7766ee0)
> GET /v12.0/act_111/ HTTP/2
> Host: graph.facebook.com
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 403 
< vary: Origin
< x-ad-account-usage: {"acc_id_util_pct":0}
< x-fb-rlafr: 0
< content-type: application/json; charset=UTF-8
< www-authenticate: OAuth "Facebook Platform" "insufficient_scope" "(#200) Provide valid app ID"
< access-control-allow-origin: *
< facebook-api-version: v12.0
< strict-transport-security: max-age=15552000; preload
< pragma: no-cache
< cache-control: no-store
< expires: Sat, 01 Jan 2000 00:00:00 GMT
< x-fb-request-id: Am3RN54patCCpaHOyAFFei2
< x-fb-trace-id: DRBLeslKDkd
< x-fb-rev: 1004746932
< x-fb-debug: uKL59lodhRXYgSVNGEttmwHpFrCHYdUtuRqAl0zFKuCA70xBHp365dz/H7gg2MFE4/qQaY7d4AlhjpSynjKa3A==
< content-length: 125
< date: Wed, 17 Nov 2021 21:35:20 GMT
< priority: u=3,i
< alt-svc: h3=":443"; ma=3600, h3-29=":443"; ma=3600
< 
* Connection #0 to host graph.facebook.com left intact
{"error":{"message":"(#200) Provide valid app ID","type":"OAuthException","code":200,"fbtrace_id":"Am3RN54patCCpaHOyAFFei2"}}ubuntu@ubuntu:~ $ wget https://graph.facebook.com/v12.0/act_111/ 
ubuntu@ubuntu:~ $ wget https://graph.facebook.com/v12.0/act_111/
--2021-11-17 21:35:33--  https://graph.facebook.com/v12.0/act_111/
Resolving graph.facebook.com (graph.facebook.com)... 157.240.3.20, 2a03:2880:f001:6:face:b00c:0:2
Connecting to graph.facebook.com (graph.facebook.com)|157.240.3.20|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-11-17 21:35:33 ERROR 403: Forbidden.

ubuntu@ubuntu:~ 8 $

If you're able to reproduce this problem, please install the debug symbol packages needed to get useful backtraces: https://wiki.ubuntu.com/Debug%20Symbol%20Packages

Thanks

Changed in openssl (Ubuntu):
status:	New → Incomplete

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2021-11-17:

#3

Hmm, something else to keep in mind: many aarch64 systems run on SD cards or USB memory sticks and those are notorious garbage.

Is this a reasonable hard drive or is this cheap flash storage? Are there messages in dmesg that might indicate filesystem or block storage errors?

If this isn't a real hard drive then your debugging time is probably better spent replacing the storage as a first effort.

Thanks

Revision history for this message

Hendrawan Kuncoro (pentatonicfunk) wrote on 2021-11-18:

#4

same issue
my environment is inside docker

------
docker run -it --platform linux/arm64 ubuntu:20.04 /bin/bash

Linux 888c7f7b294c 5.10.47-linuxkit #1 SMP PREEMPT Sat Jul 3 21:50:16 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

curl 7.68.0 (aarch64-unknown-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSocket

------

maybe that will help finding the root cause.

Revision history for this message

Juan (juanparati) wrote on 2021-11-18:

#5

@Seth: The Ubuntu 20.04 that where I tested this issue was a virtualized environments that runs Parallels Desktop over a Mac OS.

I cannot reproduce this issue on Debian Buster 10 (Arm64) with OpenSSL using the same virtualized environment.

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2021-11-18:

#6

Ah, that's good for the health of your storage :)

Please follow up with the debug symbols and reproduction instructions.

Thanks

Revision history for this message

Juan (juanparati) wrote on 2021-11-19:

#7

More information about the OpenSSL version:

Package: openssl
Architecture: arm64
Version: 1.1.1f-1ubuntu2.9
Multi-Arch: foreign
Priority: important
Section: utils
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Debian OpenSSL Team <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 1213
Depends: libc6 (>= 2.17), libssl1.1 (>= 1.1.1)
Suggests: ca-certificates
Filename: pool/main/o/openssl/openssl_1.1.1f-1ubuntu2.9_arm64.deb
Size: 598980
MD5sum: da89b21f3a0fe0fb5742b406ddcfe3f0
SHA1: 46000c169dc62b33e5a5cf0775597382576de1d3
SHA256: 62ccb4f98929011145f9d49cefa23a21388ee72aab46b304ad05fec6d46d7d2e
SHA512: 27058d8acf628ad2b26926c779c444c7393d44c897f6d23b97c7fc89ae4b0af7dd6ef8d9c28d0aca87fde107235da
940c8a0cb068e5274d87776c44ecd9e399a

Revision history for this message

Juan (juanparati) wrote on 2021-11-19:

#8

Another more accurate way of reproduce this bug:

Execute:

openssl s_client -showcerts -connect graph.facebook.com:443 </dev/null

Result:

CONNECTED(00000003)
[1] 4427 segmentation fault openssl s_client -showcerts -connect graph.facebook.com:443 < /dev/null

Revision history for this message

Juan (juanparati) wrote on 2021-11-19:

#9

I installed the debug symbols and run OpenSSL however GDB is not returned valuable information about the backtrace.

This is what I received:

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from openssl...
Reading symbols from /usr/lib/debug/.build-id/a2/f3e269767a7410ab51fafa0461e7f051144517.debug...
(gdb) run s_client -showcerts -connect graph.facebook.com:443
Starting program: /usr/bin/openssl s_client -showcerts -connect graph.facebook.com:443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
CONNECTED(00000003)

Program received signal SIGSEGV, Segmentation fault.
0x0020fffff7e0809c in ?? ()
(gdb) bt
#0 0x0020fffff7e0809c in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) frame 0
#0 0x0020fffff7e0809c in ?? ()

Revision history for this message

Bento (bdg01) wrote on 2021-11-23:

#10

I am encountering the same issue. IMHO there needs to be a newer OpenSSL release for 20.04 LTS included in the repos.

Revision history for this message

Craig Anderson (intrepidws) wrote on 2021-12-07:

#11

I have the same issue. It's preventing me from doing some fairly important things, without an obvious workaround at the moment.

Revision history for this message

Bento (bdg01) wrote on 2021-12-08:

#12

1.1.1f-1ubuntu2.09 was just updated in the repos to 1.1.1f-1ubuntu2.10 but no fix for this issue yet :-(

Revision history for this message

David Hess (dhess-8) wrote on 2021-12-25:

#13

Download full text (8.3 KiB)

Here's an interesting data point. If I run this under valgrind:

$ valgrind openssl s_client -showcerts -connect graph.facebook.com:443
==36982== Memcheck, a memory error detector
==36982== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==36982== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==36982== Command: openssl s_client -showcerts -connect graph.facebook.com:443
==36982==
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
verify return:1
depth=0 C = US, ST = California, L = Menlo Park, O = "Facebook, Inc.", CN = *.facebook.com
verify return:1
---
Certificate chain
0 s:C = US, ST = California, L = Menlo Park, O = "Facebook, Inc.", CN = *.facebook.com
i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
-----BEGIN CERTIFICATE-----
MIIGkTCCBXmgAwIBAgIQCivCRFoplTX8XaBRF4f4wDANBgkqhkiG9w0BAQsFADBw
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMS8wLQYDVQQDEyZEaWdpQ2VydCBTSEEyIEhpZ2ggQXNz
dXJhbmNlIFNlcnZlciBDQTAeFw0yMTEwMDMwMDAwMDBaFw0yMjAxMDEyMzU5NTla
MGkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRMwEQYDVQQHEwpN
ZW5sbyBQYXJrMRcwFQYDVQQKEw5GYWNlYm9vaywgSW5jLjEXMBUGA1UEAwwOKi5m
YWNlYm9vay5jb20wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQ9cpj/+XqZgH6d
qstfu/ZUoA+jNBS8A2pE9naG2+/1fDQbFhAK/4N1wNQ69xh2o1KbUId1Qi9sDCWk
Fu6s9XAqo4ID9zCCA/MwHwYDVR0jBBgwFoAUUWj/kK8CB3U8zNllZGKiErhZcjsw
HQYDVR0OBBYEFLARiFqEkLy4TD1U7suyuVQuqD6/MIG1BgNVHREEga0wgaqCDiou
ZmFjZWJvb2suY29tgg4qLmZhY2Vib29rLm5ldIILKi5mYmNkbi5uZXSCCyouZmJz
YnguY29tghAqLm0uZmFjZWJvb2suY29tgg8qLm1lc3Nlbmdlci5jb22CDioueHgu
ZmJjZG4ubmV0gg4qLnh5LmZiY2RuLm5ldIIOKi54ei5mYmNkbi5uZXSCDGZhY2Vi
b29rLmNvbYINbWVzc2VuZ2VyLmNvbTAOBgNVHQ8BAf8EBAMCB4AwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMHUGA1UdHwRuMGwwNKAyoDCGLmh0dHA6Ly9j
cmwzLmRpZ2ljZXJ0LmNvbS9zaGEyLWhhLXNlcnZlci1nNi5jcmwwNKAyoDCGLmh0
dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9zaGEyLWhhLXNlcnZlci1nNi5jcmwwPgYD
VR0gBDcwNTAzBgZngQwBAgIwKTAnBggrBgEFBQcCARYbaHR0cDovL3d3dy5kaWdp
Y2VydC5jb20vQ1BTMIGDBggrBgEFBQcBAQR3MHUwJAYIKwYBBQUHMAGGGGh0dHA6
Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBNBggrBgEFBQcwAoZBaHR0cDovL2NhY2VydHMu
ZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hBMkhpZ2hBc3N1cmFuY2VTZXJ2ZXJDQS5j
cnQwDAYDVR0TAQH/BAIwADCCAX0GCisGAQQB1nkCBAIEggFtBIIBaQFnAHYAKXm+
8J45OSHwVnOfY6V35b5XfZxgCvj5TV0mXCVdx4QAAAF8Q4ALtQAABAMARzBFAiAB
3r54JbfoFtjAqiLdaY6fXb/o1jjaT9u4XCdN5xu7NAIhAKGf3c1Jzs2cXQcAu422
KlF9P2lXrpZyMmP4rJYTuev2AHYAUaOw9f0BeZxWbbg3eI8MpHrMGyfL956IQpoN
/tSLBeUAAAF8Q4AL8AAABAMARzBFAiEAq7yuxzQO7e+txL6xITlC/qIHDYvapOTh
Tl4NHC+rOj0CIFHm8wXHI/F0gsWQVv5Ot8Ij4RuEZEVMMau/AEMroZ2aAHUAQcjK
sd8iRkoQxqE6CUKHXk4xixsD6+tLx2jwkGKWBvYAAAF8Q4ALyAAABAMARjBEAiAB
b4Q1lYSN7rcuXUxkIzbum5dVe0caAHuVTYgU00HYkQIgFtebuk3zr+NVZ520RCGZ
fetkFZv08wHf7TJK+JBMnBwwDQYJKoZIhvcNAQELBQADggEBAIcfi8CHi/+QH4cW
BjjDEZnQtPaOG77WtdFW2UK3xt2GKk4qY5rfVKLFGT9vAZRx4ZHFfK6c1IADkNMe
42lwc0+gTBrqrx1Uu+11vAIektX3Lx5xfeCTY604NK1qCC6ISYB/U2NVy2l0skT/
1xKLH87I+P7IwgQtE2en7yt9bu03GPIU0DVkYzqvBxGCq599ae2gYIVZVRSe2Xhp
WVeBB03fhf+NNcbR4LZK7n1ohrUBs7rurc0z6iv8V...

Here's an interesting data point. If I run this under valgrind:

$ valgrind openssl s_client -showcerts -connect graph.facebook.com:443
==36982== Memcheck, a memory error detector
==36982== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==36982== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==36982== Command: openssl s_client -showcerts -connect graph.facebook.com:443
==36982== 
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
verify return:1
depth=0 C = US, ST = California, L = Menlo Park, O = "Facebook, Inc.", CN = *.facebook.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Menlo Park, O = "Facebook, Inc.", CN = *.facebook.com
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
-----BEGIN CERTIFICATE-----
MIIGkTCCBXmgAwIBAgIQCivCRFoplTX8XaBRF4f4wDANBgkqhkiG9w0BAQsFADBw
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMS8wLQYDVQQDEyZEaWdpQ2VydCBTSEEyIEhpZ2ggQXNz
dXJhbmNlIFNlcnZlciBDQTAeFw0yMTEwMDMwMDAwMDBaFw0yMjAxMDEyMzU5NTla
MGkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRMwEQYDVQQHEwpN
ZW5sbyBQYXJrMRcwFQYDVQQKEw5GYWNlYm9vaywgSW5jLjEXMBUGA1UEAwwOKi5m
YWNlYm9vay5jb20wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQ9cpj/+XqZgH6d
qstfu/ZUoA+jNBS8A2pE9naG2+/1fDQbFhAK/4N1wNQ69xh2o1KbUId1Qi9sDCWk
Fu6s9XAqo4ID9zCCA/MwHwYDVR0jBBgwFoAUUWj/kK8CB3U8zNllZGKiErhZcjsw
HQYDVR0OBBYEFLARiFqEkLy4TD1U7suyuVQuqD6/MIG1BgNVHREEga0wgaqCDiou
ZmFjZWJvb2suY29tgg4qLmZhY2Vib29rLm5ldIILKi5mYmNkbi5uZXSCCyouZmJz
YnguY29tghAqLm0uZmFjZWJvb2suY29tgg8qLm1lc3Nlbmdlci5jb22CDioueHgu
ZmJjZG4ubmV0gg4qLnh5LmZiY2RuLm5ldIIOKi54ei5mYmNkbi5uZXSCDGZhY2Vi
b29rLmNvbYINbWVzc2VuZ2VyLmNvbTAOBgNVHQ8BAf8EBAMCB4AwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMHUGA1UdHwRuMGwwNKAyoDCGLmh0dHA6Ly9j
cmwzLmRpZ2ljZXJ0LmNvbS9zaGEyLWhhLXNlcnZlci1nNi5jcmwwNKAyoDCGLmh0
dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9zaGEyLWhhLXNlcnZlci1nNi5jcmwwPgYD
VR0gBDcwNTAzBgZngQwBAgIwKTAnBggrBgEFBQcCARYbaHR0cDovL3d3dy5kaWdp
Y2VydC5jb20vQ1BTMIGDBggrBgEFBQcBAQR3MHUwJAYIKwYBBQUHMAGGGGh0dHA6
Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBNBggrBgEFBQcwAoZBaHR0cDovL2NhY2VydHMu
ZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hBMkhpZ2hBc3N1cmFuY2VTZXJ2ZXJDQS5j
cnQwDAYDVR0TAQH/BAIwADCCAX0GCisGAQQB1nkCBAIEggFtBIIBaQFnAHYAKXm+
8J45OSHwVnOfY6V35b5XfZxgCvj5TV0mXCVdx4QAAAF8Q4ALtQAABAMARzBFAiAB
3r54JbfoFtjAqiLdaY6fXb/o1jjaT9u4XCdN5xu7NAIhAKGf3c1Jzs2cXQcAu422
KlF9P2lXrpZyMmP4rJYTuev2AHYAUaOw9f0BeZxWbbg3eI8MpHrMGyfL956IQpoN
/tSLBeUAAAF8Q4AL8AAABAMARzBFAiEAq7yuxzQO7e+txL6xITlC/qIHDYvapOTh
Tl4NHC+rOj0CIFHm8wXHI/F0gsWQVv5Ot8Ij4RuEZEVMMau/AEMroZ2aAHUAQcjK
sd8iRkoQxqE6CUKHXk4xixsD6+tLx2jwkGKWBvYAAAF8Q4ALyAAABAMARjBEAiAB
b4Q1lYSN7rcuXUxkIzbum5dVe0caAHuVTYgU00HYkQIgFtebuk3zr+NVZ520RCGZ
fetkFZv08wHf7TJK+JBMnBwwDQYJKoZIhvcNAQELBQADggEBAIcfi8CHi/+QH4cW
BjjDEZnQtPaOG77WtdFW2UK3xt2GKk4qY5rfVKLFGT9vAZRx4ZHFfK6c1IADkNMe
42lwc0+gTBrqrx1Uu+11vAIektX3Lx5xfeCTY604NK1qCC6ISYB/U2NVy2l0skT/
1xKLH87I+P7IwgQtE2en7yt9bu03GPIU0DVkYzqvBxGCq599ae2gYIVZVRSe2Xhp
WVeBB03fhf+NNcbR4LZK7n1ohrUBs7rurc0z6iv8V6qRU+aj8tS8606iQhcjEm9h
SN8G9/0JXXR5oWD/pAScLRtBwvkmDJEyw9lwxiErjgnfFHL0iDUZvqk0gXiQq54v
IK957WY=
-----END CERTIFICATE-----
 1 s:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
-----BEGIN CERTIFICATE-----
MIIEsTCCA5mgAwIBAgIQBOHnpNxc8vNtwCtCuF0VnzANBgkqhkiG9w0BAQsFADBs
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSswKQYDVQQDEyJEaWdpQ2VydCBIaWdoIEFzc3VyYW5j
ZSBFViBSb290IENBMB4XDTEzMTAyMjEyMDAwMFoXDTI4MTAyMjEyMDAwMFowcDEL
MAkGA1UEBhMCVVMxFTATBgNVBAoTDERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3
LmRpZ2ljZXJ0LmNvbTEvMC0GA1UEAxMmRGlnaUNlcnQgU0hBMiBIaWdoIEFzc3Vy
YW5jZSBTZXJ2ZXIgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC2
4C/CJAbIbQRf1+8KZAayfSImZRauQkCbztyfn3YHPsMwVYcZuU+UDlqUH1VWtMIC
Kq/QmO4LQNfE0DtyyBSe75CxEamu0si4QzrZCwvV1ZX1QK/IHe1NnF9Xt4ZQaJn1
itrSxwUfqJfJ3KSxgoQtxq2lnMcZgqaFD15EWCo3j/018QsIJzJa9buLnqS9UdAn
4t07QjOjBSjEuyjMmqwrIw14xnvmXnG3Sj4I+4G3FhahnSMSTeXXkgisdaScus0X
sh5ENWV/UyU50RwKmmMbGZJ0aAo3wsJSSMs5WqK24V3B3aAguCGikyZvFEohQcft
bZvySC/zA/WiaJJTL17jAgMBAAGjggFJMIIBRTASBgNVHRMBAf8ECDAGAQH/AgEA
MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
NAYIKwYBBQUHAQEEKDAmMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2Vy
dC5jb20wSwYDVR0fBEQwQjBAoD6gPIY6aHR0cDovL2NybDQuZGlnaWNlcnQuY29t
L0RpZ2lDZXJ0SGlnaEFzc3VyYW5jZUVWUm9vdENBLmNybDA9BgNVHSAENjA0MDIG
BFUdIAAwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQ
UzAdBgNVHQ4EFgQUUWj/kK8CB3U8zNllZGKiErhZcjswHwYDVR0jBBgwFoAUsT7D
aQP4v0cB1JgmGggC72NkK8MwDQYJKoZIhvcNAQELBQADggEBABiKlYkD5m3fXPwd
aOpKj4PWUS+Na0QWnqxj9dJubISZi6qBcYRb7TROsLd5kinMLYBq8I4g4Xmk/gNH
E+r1hspZcX30BJZr01lYPf7TMSVcGDiEo+afgv2MW5gxTs14nhr9hctJqvIni5ly
/D6q1UEL2tU2ob8cbkdJf17ZSHwD2f2LSaCYJkJA69aSEaRkCldUxPUd1gJea6zu
xICaEnL6VpPX/78whQYwvwt/Tv9XBZ0k7YXDK/umdaisLRbvfXknsuvCnQsH6qqF
0wGjIChBWUMo0oHjqvbsezt3tkBigAVBRQHvFwY+3sAzm2fTYS5yh+Rp/BIAV0Ae
cPUeybQ=
-----END CERTIFICATE-----
---
Server certificate
subject=C = US, ST = California, L = Menlo Park, O = "Facebook, Inc.", CN = *.facebook.com

issuer=C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3228 bytes and written 374 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_CHACHA20_POLY1305_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_CHACHA20_POLY1305_SHA256
    Session-ID: BA5E7D3EF0748870F4B94CC6EA59C5C0575EC6E8DBDDD0D9BEFCBDF543E26EC3
    Session-ID-ctx: 
    Resumption PSK: 9A9647E811A905233D1B66E668B6BC8775583D5453E3DBB68186B0E8BDE6C3DC
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 172800 (seconds)
    TLS session ticket:
    0000 - b4 c8 31 e9 50 5f 01 e3-7a 27 61 3c f1 14 92 e8   ..1.P_..z'a<....
    0010 - ab a3 15 9a e0 87 c3 0f-93 0e 56 7b 21 70 82 2b   ..........V{!p.+
    0020 - 00 00 00 00 90 38 f6 e1-5b 61 49 24 21 c5 50 c8   .....8..[aI$!.P.
    0030 - c7 cd 2c 0e 6d a0 5c 5c-19 3f 1d fe 94 fa 36 ef   ..,.m.\\.?....6.
    0040 - 1a 66 7c 33 e6 d8 47 db-46 ed e2 f9 cb 6f cf bb   .f|3..G.F....o..
    0050 - 9e 57 0d 6f 23 39 9f d0-74 9b 3b 8b 9c 5f 21 ed   .W.o#9..t.;.._!.
    0060 - 6e 86 fa 3b d2 99 c9 98-36 62 d8 84 38 aa 14 22   n..;....6b..8.."
    0070 - ec cc b7 ca b2 ee 8e 11-87 7e 1d 58 3b e6 a5 1a   .........~.X;...
    0080 - c5 34                                             .4

Start Time: 1640448351
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
^C==36982== 
==36982== Process terminating with default action of signal 2 (SIGINT)
==36982==    at 0x4C8B7BC: select (select.c:53)
==36982==    by 0x15DB47: s_client_main (s_client.c:2845)
==36982==    by 0x14DFD3: do_cmd (openssl.c:570)
==36982==    by 0x13AC03: main (openssl.c:189)
==36982== 
==36982== HEAP SUMMARY:
==36982==     in use at exit: 208,670 bytes in 4,256 blocks
==36982==   total heap usage: 6,290 allocs, 2,034 frees, 563,009 bytes allocated
==36982== 
==36982== LEAK SUMMARY:
==36982==    definitely lost: 0 bytes in 0 blocks
==36982==    indirectly lost: 0 bytes in 0 blocks
==36982==      possibly lost: 0 bytes in 0 blocks
==36982==    still reachable: 208,670 bytes in 4,256 blocks
==36982==         suppressed: 0 bytes in 0 blocks
==36982== Rerun with --leak-check=full to see details of leaked memory
==36982== 
==36982== For lists of detected and suppressed errors, rerun with: -s
==36982== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

it not only doesn't report any errors, it finishes and goes into waiting for input *without* a segmentation fault. Versus:

$ openssl s_client -showcerts -connect graph.facebook.com:443
CONNECTED(00000003)
Segmentation fault (core dumped)
$

This may be a newly discovered race condition of some sort that's only showing up on some Arm64 platforms (Apple Silicon M1 Max in my case).

Revision history for this message

David Hess (dhess-8) wrote on 2021-12-25:

#14

More info about my environment:

Running under Parallels 17.1.1 (51537) on macOS Monterey 12.1 on an Apple Silicon M1 Max. Ubuntu Ubuntu 20.04.3 LTS w/ libssl1.1:arm64 1.1.1f-1ubuntu2.10.

Revision history for this message

David Hess (dhess-8) wrote on 2021-12-25 (last edit on 2021-12-25):

#15

After a lot of sleuthing with gdb, I'm pretty confident this is the source of (and fix for) the crash we are seeing with libssl1.1:arm64 1.1.1f-1ubuntu2.10:

https://github.com/openssl/openssl/commit/fcf6e9d056162d5af64c6f7209388a5c3be2ce57

It's a bug fix for some pointer authentication assembly instructions for the Poly1305 arm64 assembly code. These instructions only execute (and crash) on Arm v8.3 64 bit processors - they NOOP on other processors that don't understand them.

Note, I have no idea why that code would not also be a problem and crash under valgrind, but I've definitely narrowed this particular crash outside of valgrind down to that location. Maybe valigrind disables pointer authentication....?

It appears the commit above was landed in OpenSSL 1.1.1i:

https://github.com/openssl/openssl/blob/OpenSSL_1_1_1i/crypto/poly1305/asm/poly1305-armv8.pl

Bottom line, in order to prevent crashes on Arm v8.3 processors I believe addressing this requires an upgrade of libssl1.1 to OpenSSL 1.1.1i or patching with that commit.

Revision history for this message

David Hess (dhess-8) wrote on 2021-12-25:

#16

Download full text (5.3 KiB)

To reproduce, be on an Arm v8.3 processor and do the following:

$ gdb $(which openssl)
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/openssl...
Reading symbols from /usr/lib/debug/.build-id/8c/c0ad363ae4508d48a68d9f9dafdbadf7bd264a.debug...
(gdb) break main
Breakpoint 1 at 0x32840: file ../apps/openssl.c, line 120.
(gdb) run s_client -showcerts -connect graph.facebook.com:443
Starting program: /usr/bin/openssl s_client -showcerts -connect graph.facebook.com:443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main (argc=5, argv=0xfffffffff478) at ../apps/openssl.c:120
120 ../apps/openssl.c: No such file or directory.
(gdb) break ../crypto/poly1305/poly1305.c:502
Breakpoint 2 at 0xfffff7e082c8: file ../crypto/poly1305/poly1305.c, line 502.
(gdb) c
Continuing.
CONNECTED(00000003)

Breakpoint 2, Poly1305_Update (ctx=ctx@entry=0xaaaaaaba97f0, inp=<optimized out>, inp@entry=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, len=992, len@entry=1001)
    at ../crypto/poly1305/poly1305.c:502
502 ../crypto/poly1305/poly1305.c: No such file or directory.
(gdb) s
poly1305_blocks_neon () at crypto/poly1305/poly1305-armv8.S:223
223 crypto/poly1305/poly1305-armv8.S: No such file or directory.
(gdb) bt
#0 poly1305_blocks_neon () at crypto/poly1305/poly1305-armv8.S:223
#1 0x0000fffff7e082dc in Poly1305_Update (ctx=ctx@entry=0xaaaaaaba97f0, inp=<optimized out>, inp@entry=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>,
    len=<optimized out>, len@entry=1001) at ../crypto/poly1305/poly1305.c:502
#2 0x0000fffff7dd7834 in chacha20_poly1305_cipher (ctx=0xaaaaaaba95b0, out=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>,
    in=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, len=1001) at ../crypto/evp/e_chacha20_poly1305.c:419
#3 0x0000fffff7ddc214 in EVP_DecryptUpdate (inl=1001, in=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, outl=0xffffffffe360,
    out=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, ctx=...

To reproduce, be on an Arm v8.3 processor and do the following:

$ gdb $(which openssl)
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/openssl...
Reading symbols from /usr/lib/debug/.build-id/8c/c0ad363ae4508d48a68d9f9dafdbadf7bd264a.debug...
(gdb) break main
Breakpoint 1 at 0x32840: file ../apps/openssl.c, line 120.
(gdb) run s_client -showcerts -connect graph.facebook.com:443
Starting program: /usr/bin/openssl s_client -showcerts -connect graph.facebook.com:443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main (argc=5, argv=0xfffffffff478) at ../apps/openssl.c:120
120	../apps/openssl.c: No such file or directory.
(gdb) break ../crypto/poly1305/poly1305.c:502
Breakpoint 2 at 0xfffff7e082c8: file ../crypto/poly1305/poly1305.c, line 502.
(gdb) c
Continuing.
CONNECTED(00000003)

Breakpoint 2, Poly1305_Update (ctx=ctx@entry=0xaaaaaaba97f0, inp=<optimized out>, inp@entry=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, len=992, len@entry=1001)
    at ../crypto/poly1305/poly1305.c:502
502	../crypto/poly1305/poly1305.c: No such file or directory.
(gdb) s
poly1305_blocks_neon () at crypto/poly1305/poly1305-armv8.S:223
223	crypto/poly1305/poly1305-armv8.S: No such file or directory.
(gdb) bt
#0  poly1305_blocks_neon () at crypto/poly1305/poly1305-armv8.S:223
#1  0x0000fffff7e082dc in Poly1305_Update (ctx=ctx@entry=0xaaaaaaba97f0, inp=<optimized out>, inp@entry=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, 
    len=<optimized out>, len@entry=1001) at ../crypto/poly1305/poly1305.c:502
#2  0x0000fffff7dd7834 in chacha20_poly1305_cipher (ctx=0xaaaaaaba95b0, out=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, 
    in=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, len=1001) at ../crypto/evp/e_chacha20_poly1305.c:419
#3  0x0000fffff7ddc214 in EVP_DecryptUpdate (inl=1001, in=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, outl=0xffffffffe360, 
    out=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, ctx=0xaaaaaaba95b0) at ../crypto/evp/evp_enc.c:498
#4  EVP_DecryptUpdate (ctx=0xaaaaaaba95b0, out=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, outl=0xffffffffe360, 
    in=0xaaaaaab9e098 "\362Hd\025\245\223\351f\027\265 b䓁\207<s\261\027\036\230\031Y/\031M\307D\"F\370", <incomplete sequence \356>, inl=1001) at ../crypto/evp/evp_enc.c:464
#5  0x0000fffff7f59d8c in tls13_enc (s=0xaaaaaab94ca0, recs=0xaaaaaab95a28, n_recs=<optimized out>, sending=0) at ../ssl/record/ssl3_record_tls13.c:173
#6  0x0000fffff7f58748 in ssl3_get_record (s=s@entry=0xaaaaaab94ca0) at ../ssl/record/ssl3_record.c:529
#7  0x0000fffff7f55fc0 in ssl3_read_bytes (s=0xaaaaaab94ca0, type=22, recvd_type=0xffffffffe5ec, buf=0xaaaaaab98b30 "\002", len=4, peek=0, readbytes=0xffffffffe5f0) at ../ssl/record/rec_layer_s3.c:1323
#8  0x0000fffff7f84800 in tls_get_message_header (s=s@entry=0xaaaaaab94ca0, mt=mt@entry=0xffffffffe68c) at ../ssl/statem/statem_lib.c:1160
#9  0x0000fffff7f7af74 in read_state_machine (s=0xaaaaaab94ca0) at ../ssl/statem/statem.c:579
#10 state_machine (s=0xaaaaaab94ca0, server=0) at ../ssl/statem/statem.c:434
#11 0x0000fffff7f55ce4 in ssl3_write_bytes (s=0xaaaaaab94ca0, type=23, buf_=0xaaaaaab89d90, len=0, written=0xffffffffe8e0) at ../ssl/record/rec_layer_s3.c:390
#12 0x0000fffff7f66b74 in ssl_write_internal (s=s@entry=0xaaaaaab94ca0, buf=buf@entry=0xaaaaaab89d90, num=num@entry=0, written=written@entry=0xffffffffe8e0) at ../ssl/ssl_lib.c:1958
#13 0x0000fffff7f66ca0 in SSL_write (s=s@entry=0xaaaaaab94ca0, buf=buf@entry=0xaaaaaab89d90, num=num@entry=0) at ../ssl/ssl_lib.c:1972
#14 0x0000aaaaaab00250 in s_client_main (argc=<optimized out>, argv=<optimized out>) at ../apps/s_client.c:2859
#15 0x0000aaaaaaaeffd4 in do_cmd (prog=0xaaaaaab84740, argc=4, argv=0xfffffffff480) at ../apps/openssl.c:570
#16 0x0000aaaaaaadcc04 in main (argc=4, argv=0xfffffffff480) at ../apps/openssl.c:189
(gdb) finish
Run till exit from #0  poly1305_blocks_neon () at crypto/poly1305/poly1305-armv8.S:223

Program received signal SIGSEGV, Segmentation fault.
0x0020fffff7e082dc in ?? ()
(gdb) bt
#0  0x0020fffff7e082dc in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

Revision history for this message

David Hess (dhess-8) wrote on 2021-12-27:

#17

If anybody needs a workaround, disable the CHACHA20 cipher suites which use Poly1305:

$ openssl s_client -debug -showcerts -connect graph.facebook.com:443 -ciphersuites TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256 -cipher 'ALL:!CHACHA20'

Unfortunately, it appears this can't be done system wide from /etc/ssl/openssl.conf - it needs to be done in a tool specific way for each tool using openssl (such as curl: https://curl.se/docs/ssl-ciphers.html).

Revision history for this message

Anders Kaseorg (andersk) wrote on 2022-02-05:

#18

David Hess’s analysis is correct. This is a bug in the ARM64 assembly for Poly1305, introduced in

https://github.com/openssl/openssl/commit/2cf7fd698ec1375421f91338ff8a44e7da5238b6 (OpenSSL_1_1_1b~37)

and fixed upstream in

https://github.com/openssl/openssl/commit/5795acffd8706e1cb584284ee5bb3a30986d0e75 (OpenSSL_1_1_1i~21).

This fix needs to be backported to focal.

Changed in openssl (Ubuntu):
status:	Incomplete → Confirmed

Revision history for this message

Anders Kaseorg (andersk) wrote on 2022-02-05:

#19

openssl_1.1.1f-1ubuntu2.10_lp1951279.debdiff Edit (2.7 KiB, text/plain)

Here’s a debdiff with the two-line upstream fix.

tags:

added: focal patch patch-accepted-upstream regression-release

Bug Watch Updater (bug-watch-updater) on 2022-02-05

Changed in openssl:
status:	Unknown → Fix Released

Bug Watch Updater (bug-watch-updater) on 2022-02-06

Changed in openssl (Debian):
status:	Unknown → New

Mathew Hodson (mhodson) on 2022-02-19

Changed in openssl (Ubuntu):
status:	Confirmed → Fix Released
importance:	Undecided → Medium
Changed in openssl (Ubuntu Focal):
importance:	Undecided → Medium

Revision history for this message

Launchpad Janitor (janitor) wrote on 2022-03-06:

#20

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openssl (Ubuntu Focal):
status:	New → Confirmed

Bug Watch Updater (bug-watch-updater) on 2022-03-17

Changed in openssl (Debian):
status:	New → Fix Released

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2022-08-05:

#21

Update: this has been fixed in Focal (same fix commit):

openssl (1.1.1f-1ubuntu2.11) focal; urgency=medium

  * Fixup pointer authentication for armv8 systems that support it when
    using the poly1305 MAC, preventing segmentation faults. (LP: #1960863)
    - d/p/lp-1960863-crypto-poly1305-asm-fix-armv8-pointer-authenticat.patch

-- Matthew Ruffell <email address hidden> Tue, 15 Feb 2022 10:10:01 +1300

Changed in openssl (Ubuntu Focal):
status:	Confirmed → Fix Released

Ubuntu
openssl package

OpenSSL 1.1.1f raise a segmentation faults on Arm64 builds

Bug Description

Duplicates of this bug

Other bug subscribers

Patches

Remote bug watches

	Status	Importance	Assigned to
OpenSSL	Fix Released	Unknown	auto-github-openssl-openssl #13256
openssl (Debian)	Fix Released	Unknown	debbugs #989604
openssl (Ubuntu)	Fix Released	Medium	Unassigned
Focal	Fix Released	Medium	Unassigned

Ubuntuopenssl package

OpenSSL 1.1.1f raise a segmentation faults on Arm64 builds

Bug Description

Duplicates of this bug

Other bug subscribers

Patches

Remote bug watches

Ubuntu
openssl package