Ubuntu

ssh crashed with SIGSEGV

Reported by Scott Betts on 2012-03-30
126
This bug affects 14 people
Affects Status Importance Assigned to Milestone
openssh (Ubuntu)
High
Unassigned
Precise
High
Unassigned
openssl (Ubuntu)
High
Unassigned
Precise
High
Unassigned

Bug Description

trying to login to certain hosts (Cisco, Solaris) using ssh causes segfault immediately following the authentication stage.

ApportVersion: 1.95-0ubuntu1
Dependencies:
 adduser 3.113ubuntu2
 coreutils 8.13-3ubuntu2
 debconf 1.5.42ubuntu1
 debianutils 4.2.1ubuntu2
 dpkg 1.16.1.2ubuntu5
 gcc-4.6-base 4.6.3-1ubuntu3
 libacl1 2.2.51-5ubuntu1
 libattr1 1:2.4.46-5ubuntu1
 libbsd0 0.3.0-1build1
 libbz2-1.0 1.0.6-1
 libc-bin 2.15-0ubuntu6
 libc6 2.15-0ubuntu6
 libcomerr2 1.42-1ubuntu1
 libdb5.1 5.1.25-11build1
 libedit2 2.11-20080614-3ubuntu1
 libgcc1 1:4.6.3-1ubuntu3
 libgssapi-krb5-2 1.10+dfsg~beta1-2
 libk5crypto3 1.10+dfsg~beta1-2
 libkeyutils1 1.5.2-2
 libkrb5-3 1.10+dfsg~beta1-2
 libkrb5support0 1.10+dfsg~beta1-2
 liblzma5 5.1.1alpha+20110809-3
 libpam-modules 1.1.3-7ubuntu2
 libpam-modules-bin 1.1.3-7ubuntu2
 libpam0g 1.1.3-7ubuntu2
 libselinux1 2.1.0-4.1ubuntu1
 libssl1.0.0 1.0.1-2ubuntu2
 libtinfo5 5.9-4
 multiarch-support 2.15-0ubuntu6
 passwd 1:4.1.4.2+svn3283-3ubuntu4
 perl-base 5.14.2-6ubuntu2
 sensible-utils 0.0.6ubuntu2
 tar 1.26-4
 tzdata 2012b-1
 xz-utils 5.1.1alpha+20110809-3
 zlib1g 1:1.2.3.4.dfsg-3ubuntu4
Disassembly:
 => 0xb757944d: movdqu (%edx),%xmm0
    0xb7579451: jne 0xb7579370
    0xb7579457: movdqa 0x60(%ebx),%xmm4
    0xb757945c: pshufb %xmm2,%xmm4
    0xb7579461: pxor %xmm0,%xmm4
    0xb7579465: movdqa 0x70(%ebx),%xmm0
    0xb757946a: movdqa (%ecx),%xmm2
    0xb757946e: pshufb %xmm3,%xmm0
    0xb7579473: pxor %xmm4,%xmm0
    0xb7579477: pshufb %xmm2,%xmm0
    0xb757947c: ret
    0xb757947d: lea 0x0(%esi),%esi
    0xb7579480: add (%esp),%ebp
    0xb7579483: movdqu (%esi),%xmm0
    0xb7579487: movdqa 0x140(%ebp),%xmm2
    0xb757948f: movdqa %xmm0,%xmm3
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Beta i386 (20120328)
Package: openssh-client 1:5.9p1-4ubuntu1
PackageArchitecture: i386
ProcVersionSignature: Ubuntu 3.2.0-20.33-generic-pae 3.2.12
Registers:
 eax 0x2def2882 770648194
 ecx 0xb7578f80 -1218998400
 edx 0xb938aff8 -1187467272
 ebx 0xb75790d0 -1218998064
 esp 0xbf8ba31c 0xbf8ba31c
 ebp 0xb7578e70 0xb7578e70
 esi 0xb937ce90 -1187524976
 edi 0xffffdd90 -8816
 eip 0xb757944d 0xb757944d
 eflags 0x10206 [ PF IF RF ]
 cs 0x73 115
 ss 0x7b 123
 ds 0x7b 123
 es 0x7b 123
 fs 0x0 0
 gs 0x33 51
RelatedPackageVersions:
 ssh-askpass N/A
 libpam-ssh N/A
 keychain 2.7.1-1
 ssh-askpass-gnome 1:5.9p1-4ubuntu1
SSHClientVersion: OpenSSH_5.9p1 Debian-4ubuntu1, OpenSSL 1.0.1 14 Mar 2012
SegvAnalysis:
 Segfault happened at: 0xb757944d: movdqu (%edx),%xmm0
 PC (0xb757944d) ok
 source "(%edx)" (0xb938aff8) ok
 destination "%xmm0" ok
 SP (0xbf8ba31c) ok
 Reason could not be automatically determined.
SourcePackage: openssh
Stacktrace:
 #0 0xb757944d in ?? () from /lib/i386-linux-gnu/libcrypto.so.1.0.0
 No symbol table info available.
 Cannot access memory at address 0xb7578e74
StacktraceTop: ?? () from /lib/i386-linux-gnu/libcrypto.so.1.0.0
Tags: precise
ThreadStacktrace:
 .
 Thread 1 (Thread 0xb71fa740 (LWP 30147)):
 #0 0xb757944d in ?? () from /lib/i386-linux-gnu/libcrypto.so.1.0.0
 No symbol table info available.
 Cannot access memory at address 0xb7578e74
Title: ssh crashed with SIGSEGV
UpgradeStatus: No upgrade log present (probably fresh install)

ProblemType: Crash
DistroRelease: Ubuntu 12.04
Package: openssh-client 1:5.9p1-4ubuntu1
ProcVersionSignature: Ubuntu 3.2.0-20.33-generic-pae 3.2.12
Uname: Linux 3.2.0-20-generic-pae i686
ApportVersion: 1.95-0ubuntu1
Architecture: i386
Date: Thu Mar 29 17:23:17 2012
ExecutablePath: /usr/bin/ssh
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Beta i386 (20120328)
ProcCmdline: ssh trumpeter
ProcEnviron:
 TERM=screen
 SHELL=/bin/bash
 LANG=en_US.UTF-8
 LANGUAGE=
SSHClientVersion: OpenSSH_5.9p1 Debian-4ubuntu1, OpenSSL 1.0.1 14 Mar 2012
Signal: 11
SourcePackage: openssh
StacktraceTop: ?? () from /lib/i386-linux-gnu/libcrypto.so.1.0.0
Title: ssh crashed with SIGSEGV
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo

Related branches

Scott Betts (scottebetts) wrote :
visibility: private → public

StacktraceTop:
 _vpaes_decrypt_core () at vpaes-x86.s:221
 vpaes_cbc_encrypt () at vpaes-x86.s:641
 ?? ()

Changed in openssh (Ubuntu):
importance: Undecided → Medium
tags: removed: need-i386-retrace

Also occurs for me, trying to login to a host running CentOS-4 (yes I know rather old):

Program received signal SIGSEGV, Segmentation fault.
_vpaes_decrypt_core () at vpaes-x86.s:221
221 vpaes-x86.s: No such file or directory.
(gdb) bt
#0 _vpaes_decrypt_core () at vpaes-x86.s:221
#1 0xb7e369e5 in vpaes_cbc_encrypt () at vpaes-x86.s:641
#2 0x732d6361 in ?? ()

Many others reporting this bug. I updated the corresponding OpenSSL ticket: http://rt.openssl.org/Ticket/Display.html?id=2775

Actually I lied: that host is running CentOS-3 and it has the RSA PAM modules. Not sure which is the issue.

I was able to work with Andy from OpenSSL and come up with a fix. Since the backtrace and disassemble in this bug report is the same as I was getting I think it should work for you too.

Robie Basak (racb) wrote :

Mike,

Thank you for your involvement in this bug and helping to make Ubuntu better.

The rt.openssl.org link you posted above does not appear to be public. Is the fix you refer to in there, and if so, please could you paste a publicly available link or paste the fix here?

Thanks!

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openssh (Ubuntu):
status: New → Confirmed
Robie Basak (racb) wrote :

A possible workaround was posted in a duplicate bug by Jurjen Stellingwerff: "Regeneration of all keys on the server fixed the immediate issue".

Could others please post if this works or doesn't work for you?

LGB [Gábor Lénárt] (lgb) wrote :

This affects me too, my bugreport (which had some gdb output in the hope it can be useful) is bug 970180

However that report is marked a duplicate of this one for now.

LGB [Gábor Lénárt] (lgb) wrote :

And btw, I have this problem with trying to ssh a Debian based (though quite old) Zorp firewall box ("ZorpOS").

The rt.openssl.org can be accessed by guests with a username "guest" and password "guest". Andy provided a patch against the 1.0.1 release of openssl with these instructions (patch attached):

See if attached patch fixes the problem. In order to do that download 1.0.1 source from openssl.org and unpack. Then in source directory

patch -p0 < /some/where/vpaes-x86.diff;
./config shared;
make;
env LD_LIBRARY_PATH=`pwd` ssh failinghost;

Later on he mentions that the official solution (covering both x86 and x64) is at http://cvs.openssl.org/chngview?cn=22317.

LGB [Gábor Lénárt] (lgb) wrote :

The provided patch seems to fix the problem for me:

lgb@vega:/tmp/openssl-1.0.1$ ssh abydos
Segmentation fault (core dumped)
lgb@vega:/tmp/openssl-1.0.1$ env LD_LIBRARY_PATH=`pwd` ssh abydos
ssh: /tmp/openssl-1.0.1/libcrypto.so.1.0.0: no version information available (required by ssh)
X11 forwarding request failed on channel 0
lgb@abydos:~$

Robie Basak (racb) on 2012-04-02
Changed in openssh (Ubuntu):
status: Confirmed → In Progress
Changed in openssl (Ubuntu):
status: New → In Progress
Changed in openssh (Ubuntu):
assignee: nobody → Robie Basak (racb)
Changed in openssl (Ubuntu):
assignee: nobody → Robie Basak (racb)

If you carry the patch in Ubuntu I definitely recommend using the http://cvs.openssl.org/chngview?cn=22317 since it contains a fix for x64 as well.

Robie Basak (racb) wrote :

Thanks Mike! I've prepared new openssl packages based on the upstream fix that you've suggested.

As I can't reproduce this bug, please could someone test my packages which are based on the fix committed upstream rather than the quick workaround? My test packages are available here: https://launchpad.net/~racb/+archive/experimental

The fix is slightly different for i386 and amd64, so please state which architecture you have tested.

If I can have confirmation that this fixes the bug, I will submit this for inclusion in Ubuntu.

Andre (sonriente) wrote :

It works (i386).
Thank you for the prompt fix.

Robie Basak (racb) wrote :

Thanks Andre!

I've looked at this bug and all the duplicates and I can't find a single instance of a bug report with this issue from somebody on amd64. So although upstream's fix includes a fix for amd64, it looks like the bug isn't triggered. Nevertheless, since upstream fixes i386 and amd64 in a single commit, I think it makes sense to include it in this fix.

The upstream fix is in both their trunk and stable release branch, but they don't have a stable release that includes this fix yet. Given that we're close to release, I think it makes sense to carry this patch. It will go away as soon as Debian updates to the next upstream release and we merge it.

Debdiff attached.

Changed in openssh (Ubuntu):
status: In Progress → Triaged
assignee: Robie Basak (racb) → nobody
Changed in openssl (Ubuntu):
status: In Progress → Triaged
assignee: Robie Basak (racb) → nobody
Scott Betts (scottebetts) wrote :

Confirmed that the fix in racb's ppa works on i386 as well. Thanks Robie!

xvalentinex (xvalentinex) wrote :

Another confirmed fix for i386

Richard Johnson (nixternal) wrote :

I have this issue when attempt to connect to remote Solaris boxes. Haven't tried the patch above just yet.

Richard Johnson (nixternal) wrote :

Robie's patch works for me, x86.

One more success for the patch on x86.

Changed in openssh (Ubuntu):
importance: Medium → High
Changed in openssh (Ubuntu Precise):
milestone: none → ubuntu-12.04
Changed in openssl (Ubuntu Precise):
importance: Undecided → High
Changed in openssh (Ubuntu Precise):
status: Triaged → Invalid
Changed in openssl (Ubuntu Precise):
milestone: none → ubuntu-12.04
Colin Watson (cjwatson) wrote :

We should merge from Debian instead, since the diff from 1.0.1-2 to 1.0.1-4, discounting things we already have, is exactly this change. I agreed with Clint on #ubuntu-release that I'd take care of this.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssl - 1.0.1-4ubuntu1

---------------
openssl (1.0.1-4ubuntu1) precise; urgency=low

  * Resynchronise with Debian (LP: #968753). Remaining changes:
    - debian/libssl1.0.0.postinst:
      + Display a system restart required notification on libssl1.0.0
        upgrade on servers.
      + Use a different priority for libssl1.0.0/restart-services depending
        on whether a desktop, or server dist-upgrade is being performed.
    - debian/{libssl1.0.0-udeb.dirs, control, rules}: Create
      libssl1.0.0-udeb, for the benefit of wget-udeb (no wget-udeb package
      in Debian).
    - debian/{libcrypto1.0.0-udeb.dirs, libssl1.0.0.dirs, libssl1.0.0.files,
      rules}: Move runtime libraries to /lib, for the benefit of
      wpasupplicant.
    - debian/patches/perlpath-quilt.patch: Don't change perl #! paths under
      .pc.
    - debian/rules:
      + Don't run 'make test' when cross-building.
      + Use host compiler when cross-building. Patch from Neil Williams.
      + Don't build for processors no longer supported: i586 (on i386)
      + Fix Makefile to properly clean up libs/ dirs in clean target.
      + Replace duplicate files in the doc directory with symlinks.
    - Unapply patch c_rehash-multi and comment it out in the series as it
      breaks parsing of certificates with CRLF line endings and other cases
      (see Debian #642314 for discussion), it also changes the semantics of
      c_rehash directories by requiring applications to parse hash link
      targets as files containing potentially *multiple* certificates rather
      than exactly one.
    - Bump version passed to dh_makeshlibs to 1.0.1 for new symbols.
    - Experimental workaround to large client hello issue: if
      OPENSSL_NO_TLS1_2_CLIENT is set then TLS v1.2 is disabled for clients
      only.
    - Compile with -DOPENSSL_NO_TLS1_2_CLIENT.

openssl (1.0.1-4) unstable; urgency=low

  * Use official patch for the vpaes problem, also covering amd64.

openssl (1.0.1-3) unstable; urgency=high

  * Fix crash in vpaes (Closes: #665836)
  * use client version when deciding whether to send supported signature
    algorithms extension
 -- Colin Watson <email address hidden> Tue, 10 Apr 2012 20:50:52 +0100

Changed in openssl (Ubuntu Precise):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.