sshd core dumps while trying to connect with a password

[bnsmb]

Binary package hint: openssh-server

Running

Linux tp61p 2.6.22-14-generic #1 SMP Tue Feb 12 02:46:46 UTC 2008 x86_64 GNU/Linux

on a Thinkpad T61p; the finger print reader is configured and works.

Connecting to Ubuntu via ssh with password fails ; connecting via ssh with public key works.

In most cases the sshd silently dies; but in some cases it prints some output before dieing if executed with the -d parameter. Examples:

# first example

xtrnaw7@tp61p:~$ sudo /usr/sbin/sshd -p 1234 -d
Password or swipe finger:
debug1: sshd version OpenSSH_4.6p1 Debian-5ubuntu0.1
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-p'
debug1: rexec_argv[2]='1234'
debug1: rexec_argv[3]='-d'
debug1: Bind to port 1234 on 0.0.0.0.
Server listening on 0.0.0.0 port 1234.
socket: Address family not supported by protocol

debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
Connection from 192.168.1.164 port 50182
debug1: Client protocol version 2.0; client software version Sun_SSH_1.2
debug1: no match: Sun_SSH_1.2
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.6p1 Debian-5ubuntu0.1
debug1: permanently_set_uid: 109/65534
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user xtrnaw7 service ssh-connection method none
debug1: attempt 0 failures 0
debug1: PAM: initializing for "xtrnaw7"
debug1: PAM: setting PAM_RHOST to "pb001"
debug1: PAM: setting PAM_TTY to "ssh"
Failed none for xtrnaw7 from 192.168.1.164 port 50182 ssh2
debug1: userauth-request for user xtrnaw7 service ssh-connection method password
debug1: attempt 1 failures 1
debug1: do_cleanup
Segmentation fault (core dumped)

# second example:

xtrnaw7@tp61p:~$
xtrnaw7@tp61p:~$ sudo /usr/sbin/sshd -p 1234 -d
debug1: sshd version OpenSSH_4.6p1 Debian-5ubuntu0.1
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-p'
debug1: rexec_argv[2]='1234'
debug1: rexec_argv[3]='-d'
debug1: Bind to port 1234 on 0.0.0.0.
Server listening on 0.0.0.0 port 1234.
socket: Address family not supported by protocol
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
Connection from 192.168.1.164 port 46810
debug1: Client protocol version 2.0; client software version Sun_SSH_1.2
debug1: no match: Sun_SSH_1.2
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.6p1 Debian-5ubuntu0.1
debug1: permanently_set_uid: 109/65534
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user xtrnaw7 service ssh-connection method none
debug1: attempt 0 failures 0
debug1: PAM: initializing for "xtrnaw7"
debug1: PAM: setting PAM_RHOST to "pb001"
debug1: PAM: setting PAM_TTY to "ssh"
Failed none for xtrnaw7 from 192.168.1.164 port 46810 ssh2
debug1: userauth-request for user xtrnaw7 service ssh-connection method password
debug1: attempt 1 failures 1
Error: Bad address.
*** glibc detected *** sshd: xtrnaw7 [priv]: malloc(): memory corruption (fast): 0x00005555557cf8b0 ***
======= Backtrace: =========
/lib/libc.so.6[0x2b3f43940ecf]
/lib/libc.so.6(__libc_malloc+0x93)[0x2b3f43941e23]
/lib/libc.so.6(__nss_lookup_function+0xc0)[0x2b3f439b1d60]
/lib/libc.so.6[0x2b3f439b1ff5]
/lib/libc.so.6(getspnam_r+0x158)[0x2b3f439a68d8]
/lib/libpam.so.0(pam_modutil_getspnam+0x7a)[0x2b3f41bc624a]
/lib/security/pam_unix.so[0x2b3f452e6e57]
/lib/security/pam_unix.so(pam_sm_authenticate+0x243)[0x2b3f452e3873]
/lib/libpam.so.0[0x2b3f41bc1bd1]
/lib/libpam.so.0(pam_authenticate+0x43)[0x2b3f41bc1513]
sshd: xtrnaw7 [priv][0x55555557c23e]
sshd: xtrnaw7 [priv][0x555555562ffb]
sshd: xtrnaw7 [priv][0x555555574c40]
sshd: xtrnaw7 [priv][0x555555575276]
sshd: xtrnaw7 [priv][0x5555555754f7]
sshd: xtrnaw7 [priv][0x55555555ff20]
sshd: xtrnaw7 [priv](main+0x262d)[0x5555555626fd]
/lib/libc.so.6(__libc_start_main+0xf4)[0x2b3f438ebb44]
sshd: xtrnaw7 [priv][0x55555555efc9]
======= Memory map: ========
40000000-40001000 ---p 40000000 00:00 0
40001000-40801000 rw-p 40001000 00:00 0
40801000-40802000 ---p 40801000 00:00 0
40802000-41002000 rw-p 40802000 00:00 0
2aaaaaac0000-2aaaaaacd000 r-xp 00000000 08:03 754875 /lib/libgcc_s.so.1
2aaaaaacd000-2aaaaaccd000 ---p 0000d000 08:03 754875 /lib/libgcc_s.so.1
2aaaaaccd000-2aaaaacce000 rw-p 0000d000 08:03 754875 /lib/libgcc_s.so.1
2b3f41798000-2b3f417b5000 r-xp 00000000 08:03 754890 /lib/ld-2.6.1.so
2b3f417b5000-2b3f417b8000 rw-p 2b3f417b5000 00:00 0
2b3f417b8000-2b3f417c8000 rw-s 00000000 00:09 52678 /dev/zero (deleted)
2b3f417c8000-2b3f41908000 rw-s 00000000 00:09 52679 /dev/zero (deleted)
2b3f419b4000-2b3f419b6000 rw-p 0001c000 08:03 754890 /lib/ld-2.6.1.so
2b3f419b6000-2b3f419be000 r-xp 00000000 08:03 754894 /lib/libwrap.so.0.7.6
2b3f419be000-2b3f41bbd000 ---p 00008000 08:03 754894 /lib/libwrap.so.0.7.6
2b3f41bbd000-2b3f41bbf000 rw-p 00007000 08:03 754894 /lib/libwrap.so.0.7.6
2b3f41bbf000-2b3f41bc9000 r-xp 00000000 08:03 752270 /lib/libpam.so.0.81.6
2b3f41bc9000-2b3f41dc9000 ---p 0000a000 08:03 752270 /lib/libpam.so.0.81.6
2b3f41dc9000-2b3f41dca000 rw-p 0000a000 08:03 752270 /lib/libpam.so.0.81.6
2b3f41dca000-2b3f41dcc000 r-xp 00000000 08:03 754878 /lib/libdl-2.6.1.so
2b3f41dcc000-2b3f41fcc000 ---p 00002000 08:03 754878 /lib/libdl-2.6.1.so
2b3f41fcc000-2b3f41fce000 rw-p 00002000 08:03 754878 /lib/libdl-2.6.1.so
2b3f41fce000-2b3f41fe5000 r-xp 00000000 08:03 752267 /lib/libselinux.so.1
2b3f41fe5000-2b3f421e4000 ---p 00017000 08:03 752267 /lib/libselinux.so.1
2b3f421e4000-2b3f421e6000 rw-p 00016000 08:03 752267 /lib/libselinux.so.1
2b3f421e6000-2b3f421e8000 rw-p 2b3f421e6000 00:00 0
2b3f421e8000-2b3f42343000 r-xp 00000000 08:03 1560061 /usr/lib/libcrypto.so.0.9.8
2b3f42343000-2b3f42543000 ---p 0015b000 08:03 1560061 /usr/lib/libcrypto.so.0.9.8
2b3f42543000-2b3f42566000 rw-p 0015b000 08:03 1560061 /usr/lib/libcrypto.so.0.9.8
2b3f42566000-2b3f42569000 rw-p 2b3f42566000 00:00 0
2b3f42569000-2b3f4256b000 r-xp 00000000 08:03 754924 /lib/libutil-2.6.1.so
2b3f4256b000-2b3f4276a000 ---p 00002000 08:03 754924 /lib/libutil-2.6.1.so
2b3f4276a000-2b3f4276c000 rw-p 00001000 08:03 754924 /lib/libutil-2.6.1.so
2b3f4276c000-2b3f42782000 r-xp 00000000 08:03 1558534 /usr/lib/libz.so.1.2.3.3
2b3f42782000-2b3f42982000 ---p 00016000 08:03 1558534 /usr/lib/libz.so.1.2.3.3
2b3f42982000-2b3f42983000 rw-p 00016000 08:03 1558534 /usr/lib/libz.so.1.2.3.3
2b3f42983000-2b3f42984000 rw-p 2b3f42983000 00:00 0
2b3f42984000-2b3f4299adebug1: do_cleanup
Aborted (core dumped)
xtrnaw7@tp61p:~$

xtrnaw7@tp61p:/data/source$ grep -v "^#" /etc/ssh/sshd_config | grep -v "^$"
Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 768
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PasswordAuthentication yes
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
UsePAM yes

xtrnaw7@tp61p:/data/source$ grep -v "^#" /etc/pam.d/ssh | grep -v "^$"
auth required pam_env.so # [1]
auth required pam_env.so envfile=/etc/default/locale
@include common-auth
account required pam_nologin.so
@include common-account
@include common-session
session optional pam_motd.so # [1]
session optional pam_mail.so standard noenv # [1]
session required pam_limits.so
@include common-password

[Nicolas Valcarcel (nvalcarcel)]

Can't reproduce in hardy.

 nxvl@LePew:~$ sudo /usr/sbin/sshd -p 1234 -d
debug1: sshd version OpenSSH_4.7p1 Debian-5ubuntu1
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-p'
debug1: rexec_argv[2]='1234'
debug1: rexec_argv[3]='-d'
debug1: Bind to port 1234 on ::.
Server listening on :: port 1234.
debug1: Bind to port 1234 on 0.0.0.0.
Bind to port 1234 on 0.0.0.0 failed: Address already in use.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
Connection from 127.0.0.1 port 52375
debug1: Client protocol version 2.0; client software version OpenSSH_4.7p1 Debian-5ubuntu1
debug1: match: OpenSSH_4.7p1 Debian-5ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-5ubuntu1
debug1: permanently_set_uid: 114/65534
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user nxvl service ssh-connection method none
debug1: attempt 0 failures 0
debug1: PAM: initializing for "nxvl"
debug1: PAM: setting PAM_RHOST to "localhost"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user nxvl service ssh-connection method publickey
debug1: attempt 1 failures 1
debug1: test whether pkalg/pkblob are acceptable
Failed none for nxvl from 127.0.0.1 port 52375 ssh2
debug1: temporarily_use_uid: 1000/1000 (e=0/0)
debug1: trying public key file /home/nxvl/.ssh/authorized_keys
debug1: restore_uid: 0/0
debug1: temporarily_use_uid: 1000/1000 (e=0/0)
debug1: trying public key file /home/nxvl/.ssh/authorized_keys2
debug1: restore_uid: 0/0
Failed publickey for nxvl from 127.0.0.1 port 52375 ssh2
debug1: userauth-request for user nxvl service ssh-connection method password
debug1: attempt 2 failures 2
debug1: PAM: password authentication accepted for nxvl
debug1: do_pam_account: called
Accepted password for nxvl from 127.0.0.1 port 52375 ssh2
debug1: monitor_child_preauth: nxvl has been authenticated by privileged process
debug1: PAM: establishing credentials
debug1: permanently_set_uid: 1000/1000
debug1: SELinux support disabled
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: init
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm sess...

[bnsmb (bernd-schemmer)]

>>Can't reproduce in hardy.

And now? Do I have to wait for hardy or do you need more information?

regards

Bernd

Nicolas Valcárcel (nxvl) wrote:
> Can't reproduce in hardy.
>
> nxvl@LePew:~$ sudo /usr/sbin/sshd -p 1234 -d
> debug1: sshd version OpenSSH_4.7p1 Debian-5ubuntu1
> debug1: read PEM private key done: type RSA
> debug1: private host key: #0 type 1 RSA
> debug1: read PEM private key done: type DSA
> debug1: private host key: #1 type 2 DSA
> debug1: rexec_argv[0]='/usr/sbin/sshd'
> debug1: rexec_argv[1]='-p'
> debug1: rexec_argv[2]='1234'
> debug1: rexec_argv[3]='-d'
> debug1: Bind to port 1234 on ::.
> Server listening on :: port 1234.
> debug1: Bind to port 1234 on 0.0.0.0.
> Bind to port 1234 on 0.0.0.0 failed: Address already in use.
> debug1: Server will not fork when running in debugging mode.
> debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
> debug1: inetd sockets after dupping: 3, 3
> Connection from 127.0.0.1 port 52375
> debug1: Client protocol version 2.0; client software version OpenSSH_4.7p1 Debian-5ubuntu1
> debug1: match: OpenSSH_4.7p1 Debian-5ubuntu1 pat OpenSSH*
> debug1: Enabling compatibility mode for protocol 2.0
> debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-5ubuntu1
> debug1: permanently_set_uid: 114/65534
> debug1: list_hostkey_types: ssh-rsa,ssh-dss
> debug1: SSH2_MSG_KEXINIT sent
> debug1: SSH2_MSG_KEXINIT received
> debug1: kex: client->server aes128-cbc hmac-md5 none
> debug1: kex: server->client aes128-cbc hmac-md5 none
> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
> debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
> debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
> debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
> debug1: SSH2_MSG_NEWKEYS sent
> debug1: expecting SSH2_MSG_NEWKEYS
> debug1: SSH2_MSG_NEWKEYS received
> debug1: KEX done
> debug1: userauth-request for user nxvl service ssh-connection method none
> debug1: attempt 0 failures 0
> debug1: PAM: initializing for "nxvl"
> debug1: PAM: setting PAM_RHOST to "localhost"
> debug1: PAM: setting PAM_TTY to "ssh"
> debug1: userauth-request for user nxvl service ssh-connection method publickey
> debug1: attempt 1 failures 1
> debug1: test whether pkalg/pkblob are acceptable
> Failed none for nxvl from 127.0.0.1 port 52375 ssh2
> debug1: temporarily_use_uid: 1000/1000 (e=0/0)
> debug1: trying public key file /home/nxvl/.ssh/authorized_keys
> debug1: restore_uid: 0/0
> debug1: temporarily_use_uid: 1000/1000 (e=0/0)
> debug1: trying public key file /home/nxvl/.ssh/authorized_keys2
> debug1: restore_uid: 0/0
> Failed publickey for nxvl from 127.0.0.1 port 52375 ssh2
> debug1: userauth-request for user nxvl service ssh-connection method password
> debug1: attempt 2 failures 2
> debug1: PAM: password authentication accepted for nxvl
> debug1: do_pam_account: called
> Accepted password for nxvl from 127.0.0.1 port 52375 ssh2
> debug1: monitor_child_preauth: nxvl has been authenticated by privileged process
> debug1: PAM: establishing credentials
> debug1: permanently_set_uid: 1000/1000
> debug1: SELinux support disabled
> debug1: Entering interactive session for SSH2.
> debug1: server_init_dispatch_20
> debug1: server_input_channel_open: ctype session rchan 0 w...

[Colin Watson (cjwatson)]

This looks like a thinkfinger bug; "Error: Bad address." is (I think) being emitted by _libthinkfinger_store_fingerprint, and indicates some kind of memory corruption within that library. Perhaps allocating a 10KB array on the stack is too much for it? It's hard to be sure from here, though.

Perhaps somebody else with a thinkfinger setup can figure out what's going on here.

Could you attach /etc/pam.d/ssh and /etc/pam.d/common-auth for good measure, please?

[Chris Jones (cmsj)]

I just installed thinkfinger on a ~month old hardy install, configured it to accept my fingerprint, and installed sshd.
I can authenticate with sudo with my finger, so I believe tf is working. I can also ssh to localhost without sshd dying.

ie, I cannot reproduce this bug.

[Justin Dugger (jldugger)]

Ah, thinkfinger doesn't ship in gutsy. I do provide it in a PPA, but this should be fixed in hardy and in the long run thinkfinger will migrate to fprint. Unfortunately, Launchpad PPAs have no bug reporting facilities.

So on the one hand, this bug does exist, but on the other hand, Ubuntu doesn't ship this configuration. This bug should probably be marked invalid, unless sshd shouldn't be coredumping on broken pam modules.

[bnsmb (bernd-schemmer)]

Deinstalled the fingerprint packages

xtrnaw7@tp61p:~$ dpkg -l | grep finger
ii finger 0.17-11 user information lookup program
ii libpam-thinkfinger 0.3~ppa9 PAM module for the STMicroelectronics fingerpr
ii libthinkfinger0 0.3~ppa9 library for the STMicroelectronics fingerprint
ii thinkfinger-tools 0.3~ppa9 utilities for the STMicroelectronics fingerpri
xtrnaw7@tp61p:~$ sudo dpkg -r libpam-thinkfinger libthinkfinger0 thinkfinger-tools
Password or swipe finger:
(Reading database ... 141457 files and directories currently installed.)
Removing libpam-thinkfinger ...
dpkg - warning: while removing libpam-thinkfinger, directory `/etc/pam_thinkfinger' not empty so not removed.
Removing thinkfinger-tools ...
Removing libthinkfinger0 ...
Processing triggers for libc6 ...
ldconfig deferred processing now taking place
xtrnaw7@tp61p:~$

Now ssh connections with userid/password work again

[Justin Dugger (jldugger)]

This is a known bug upstream, with a patch to fix it. upstream was going to put up a final release to include these patches, but it hasn't happened yet. Thinkfinger in hardy is provided from upstream SVN + some extra patches not in SVN, but occasionally puts hald-input-addon (I think) into an infinite loop.

Long story short, I don't anticipate fixing this. There are work arounds, and Ubuntu has picked thinkfinger up into main for hardy, so I won't be providing it any longer anyways.

[Scott James Remnant (Canonical) (canonical-scott)]

thinkfinger in hardy definitely includes the patch to fix this