sshd crashed on s390x with hw crypto enabled

[Frank Heimes]

While using today's daily image of disco (either in z/VM or on LPAR) and enabling s309x hardware crypto support in openssh, sshd crashes with the following messages:

$ ssh ubuntu@localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:KoTYC0jCQPtmsOMmBW9DrCiBbkrKTL0leQ/zoIaInNw.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
ubuntu@localhost's password:
packet_write_wait: Connection to ::1 port 22: Broken pipe
(local session is sufficient to reproduce)

Steps to reproduce - on disco daily:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Disco Dingo (development branch)
Release: 19.04
Codename: disco
$ uname -a
Linux zlin42 5.0.0-7-generic #8-Ubuntu SMP Mon Mar 4 16:25:21 UTC 2019 s390x s390x s390x GNU/Linux

- enable z hw crypto support for openssh on an Ubuntu host (zlin42) on s390x like this:
- sudo apt-get install openssl-ibmca libica-utils libica3
- sudo tee -a /etc/ssl/openssl.cnf < /usr/share/doc/openssl-ibmca/examples/openssl.cnf.sample
- sudo sed -i 's/^\(openssl_conf = openssl_def.*$\)/# \1/g' /etc/ssl/openssl.cnf
- sudo sed -i '10i openssl_conf = openssl_def' /etc/ssl/openssl.cnf
- afterwards ssh login attempts fail (existing session are unaffected):
   $ ssh ubuntu@localhost
   The authenticity of host 'localhost (::1)' can't be established.
   ECDSA key fingerprint is SHA256:KoTYC0jCQPtmsOMmBW9DrCiBbkrKTL0leQ/zoIaInNw.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
   ubuntu@localhost's password:
   packet_write_wait: Connection to ::1 port 22: Broken pipe

[94629.032586] User process fault: interruption code 003b ilc:2 in libpthread-2.29.so[3ff7d480000+1c000]
[94629.032597] Failing address: 0000000000000000 TEID: 0000000000000800
[94629.032598] Fault in primary space mode while using user ASCE.
[94629.032601] AS:00000007450281c7 R3:0000000000000024
[94629.032606] CPU: 0 PID: 8183 Comm: sshd Not tainted 5.0.0-7-generic #8-Ubuntu
[94629.032607] Hardware name: IBM 2964 N63 400 (LPAR)
[94629.032608] User PSW : 0705200180000000 000003ff7d48e954
[94629.032610] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:1 AS:0 CC:2 PM:0 RI:0 EA:3
[94629.032611] User GPRS: 0000000000000000 0000000000000000 0000000000000000 000003ff7e0085c8
[94629.032612] 000003ff7e510108 000003ff7db1c3a8 0000000000000000 000003fff857eea0
[94629.032613] 000003ff7e525040 000002aa3f5ec090 000003ff7e4916f0 000003ff7e4921a8
[94629.032614] 000003ff7db17c18 0000000000000002 000003ff7e07238a 000003fff857ea20
[94629.032622] User Code: 000003ff7d48e946: b9040012 lgr %r1,%r2
                          000003ff7d48e94a: e3f0ff60ff71 lay %r15,-160(%r15)
                         #000003ff7d48e950: 47000000 bc 0,0
                         >000003ff7d48e954: 58202018 l %r2,24(%r2)
                          000003ff7d48e958: b24f00b0 ear %r11,%a0
                          000003ff7d48e95c: ebbb0020000d sllg %r11,%r11,32
                          000003ff7d48e962: b24f00b1 ear %r11,%a1
                          000003ff7d48e966: 5920b0d0 c %r2,208(%r11)
[94629.032634] Last Breaking-Event-Address:
[94629.032638] [<000003ff7df773b4>] 0x3ff7df773b4

For more details see attachments ...
---
ProblemType: Bug
ApportVersion: 2.20.10-0ubuntu23
Architecture: s390x
DistroRelease: Ubuntu 19.04
Package: openssh-server 1:7.9p1-9
PackageArchitecture: s390x
ProcVersionSignature: Ubuntu 5.0.0-7.8-generic 5.0.0
SSHDConfig: Error: command ['/usr/sbin/sshd', '-T'] failed with exit code -11:
Tags: disco
Uname: Linux 5.0.0-7-generic s390x
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: pkcs11
_MarkForUpload: True

[Frank Heimes (fheimes)]

GNU gdb (Ubuntu 8.2.90-0ubuntu1) 8.2.90
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "s390x-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/sshd...
Reading symbols from /usr/lib/debug/.build-id/18/2cb9bda0be98e50d87a4104872aead696f1998.debug...
[New LWP 27159]
Core was generated by `sshd: ubuntu [net] '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000003ff9d10cd54 in ?? ()
(gdb) bt
#0 0x000003ff9d10cd54 in ?? ()
#1 0x000003ff9dcf155a in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[Frank Heimes (fheimes)]

Please see also LP 1686618 aka bugzilla LTC Bug 153940 and 156865
With the new crypto features available probably more/new things to be allowed in sshd.
Check that that/if sandboxing works in sshd with hw crypto.

[Harald Freudenberger (freude-5)]

Can you recreate this with debug info installed ?
I'd like to see the callstack with symbol information to find out
where the upper stack frames are located.
I see a lot of audit messages in the syslog snipped. Is there
SELINUX enabled ? If so, try to recreate with selinux disabled.
Some more history at the syslog messages would also help.
If the sandboxed process spawned by the sshd at the login attemt
denies some system calls (for example an ioctl on the device
/dev/zcrypt) there are usually some notes to find there.
Some information about the crypto hardware you are using
would also be fine. Accelerator ? Coprocessor ? The output
of lszcrypt -V would help. Can you see the counter(s) increasing
with each attempt ?

Thanks
Harald Freudenberger

[Dimitri John Ledkov (xnox)]

This is Ubuntu, so apparmor is enabled by default, rather than selinux. Also, in stock configurations we shouldn't be disabling selinux/apparmor.

@Frank if there is a crash file, you can do $ ubuntu-bug /var/crash/path-to.crash

For them to be generated, you may need to install apport whoopsie (the non-interactive ui one maybe) cause then launchpad can retrace crashes with debug symbols on.

[Dimitri John Ledkov (xnox)]

Oooh, i see crashes anyway, let's see if i can force launchpad to retrace them for us.

[Frank Heimes (fheimes)]

Two crash files were already uploaded - see above, but manually.

I just created another nice(er) 'apport-retrace --gdb' output locally on that system (w/ dbgsym):

$ sudo apport-retrace --gdb --rebuild-package-info /var/crash/_usr_sbin_sshd.0.crash
GNU gdb (Ubuntu 8.2.90-0ubuntu1) 8.2.90
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "s390x-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from /usr/sbin/sshd...
Reading symbols from /usr/lib/debug/.build-id/18/2cb9bda0be98e50d87a4104872aead696f1998.debug...
[New LWP 4508]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/sshd -T'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __pthread_rwlock_wrlock_full (abstime=0x0, rwlock=0x0)
    at pthread_rwlock_common.c:581
581 pthread_rwlock_common.c: No such file or directory.
(gdb) bt
#0 __pthread_rwlock_wrlock_full (abstime=0x0, rwlock=0x0)
    at pthread_rwlock_common.c:581
#1 __GI___pthread_rwlock_wrlock (rwlock=0x0) at pthread_rwlock_wrlock.c:27
#2 0x000003ffab27238a in CRYPTO_THREAD_write_lock ()
   from /lib/s390x-linux-gnu/libcrypto.so.1.1
#3 0x000003ffab208b32 in ERR_unload_strings ()
   from /lib/s390x-linux-gnu/libcrypto.so.1.1
#4 0x000003ffab6868b4 in ERR_unload_IBMCA_strings () at e_ibmca_err.c:122
#5 0x000003ffab684dd2 in ibmca_destructor () at e_ibmca.c:754
#6 0x000003ffab7100ea in ?? () from /lib/ld64.so.1
#7 0x000003ffaabc79e8 in __run_exit_handlers (status=<optimized out>,
    listp=0x3ffaad189a8 <__exit_funcs>,
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
    at exit.c:108
#8 0x000003ffaabc7ad8 in __GI_exit (status=<optimized out>) at exit.c:139
#9 0x000002aa192909c0 in main (ac=<optimized out>, av=<optimized out>)
    at ../../sshd.c:2257
(gdb) quit

(if you wanna see it colored, see attachment)

[Frank Heimes (fheimes)]

apport information

[Frank Heimes (fheimes)]

apport information

[Frank Heimes (fheimes)]

apport information

[Frank Heimes (fheimes)]

ubuntu-bug upload see: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1820056

[Harald Freudenberger (freude-5)]

enough ffdc data
Thanks

This happens at ibmca destruction time.
Looks like there is some kind of double free or so in the
destructor for the ibmca engine. We had some issues like
this in the past already. I'll have a look onto the code ...

Thanks Frank and Xnox

regards Harald Freudenberger

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-15 08:55 EDT-------
Description will be mirrored via reverse mirroring

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-18 10:31 EDT-------
I am unable to recreate this on Ubuntu 18.10. However, I reviewed the destructor code in ibmca and found some code which could be the reason.
So I'll give you a patch for checking - just a shot in the dark...

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2019-03-18 10:34 EDT-------

Here is an attempt to maybe fix the issue.
Can you please apply this on top of the ibmca code, build and test ?

Btw. some info about your crypto config would also help me to try to recreate this. Just an lszcrypt -V output would be fine.

Thanks
Harald Freudenberger

[Frank Heimes (fheimes)]

$ sudo lszcrypt -b
ap_domain=0xc
ap_max_domain_id=0x54
ap_interrupts are enabled
config_time=30 (seconds)
poll_thread is disabled
poll_timeout=250000 (nanoseconds)

$ sudo lszcrypt -V
CARD.DOMAIN TYPE MODE STATUS REQUESTS PENDING HWTYPE QDEPTH FUNCTIONS DRIVER
--------------------------------------------------------------------------------------------
00 CEX5C CCA-Coproc online 100016 0 11 08 S--D--N-- cex4card
00.000c CEX5C CCA-Coproc online 100016 0 11 08 S--D--N-- cex4queue

[Dimitri John Ledkov (xnox)]

This cannot be reproduced on 18.10, as on 18.10 openssh uses libssl1.0.0 package, yet openssl-ibmca is provided for libssl1.1 only.

In disco (19.04), openssh has been upgraded to use libssl1.1 with a matching openssl-ibmca for libssl1.1.

If you want to reproduce this on 18.10, do add 'disco' repository in /etc/apt/sources.list and upgrade to disco's version of openssh-server.

(note that 19.04 uses newer openssl-ibmca too 2.0.2 vs 2.0.0 which was in 18.10).

[Dimitri John Ledkov (xnox)]

This PPA:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3681

Will soon have openssl-ibmca with above mentioned patch, built and published.

[Frank Heimes (fheimes)]

I just verified the updated package:
$ apt-cache policy openssl-ibmca
openssl-ibmca:
  Installed: 2.0.2-0ubuntu2
  Candidate: 2.0.2-0ubuntu2
  Version table:
 *** 2.0.2-0ubuntu2 100
        100 /var/lib/dpkg/status
     2.0.2-0ubuntu1 500
        500 http://us.ports.ubuntu.com/ubuntu-ports disco/universe s390x Packages
and it looks like it fixes the problem.
With this package installed and hw crypto enabled,
I can again remotely (and locally) login again into that disco system w/o a sshd crash.
Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl-ibmca - 2.0.2-0ubuntu2

---------------
openssl-ibmca (2.0.2-0ubuntu2) disco; urgency=medium

  * Rework error string init and exit. LP: #1819487

 -- Dimitri John Ledkov <email address hidden> Mon, 18 Mar 2019 15:03:08 +0000

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-04-02 07:38 EDT-------
IBM Bugzilla status -> closed, Fix Released with Disco