ssh hangs in initial handshaking when using IPv6

[Rhialto]

ssh hangs in initial handshaking when using IPv6 to connect with a NetBSD 4.0 ssh server. I tried NetBSD on i386 and amd64.

Here is the output from ssh -v when it fails: (I censored the IPv6 address)

rhialto@glicca:~$ ssh -v radl
OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to radl [2001:xxxx:xxxx::1] port 22.
debug1: Connection established.
debug1: identity file /home/rhialto/.ssh/identity type -1
debug1: identity file /home/rhialto/.ssh/id_rsa type -1
debug1: identity file /home/rhialto/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.4 NetBSD_Secure_Shell-20061114
debug1: match: OpenSSH_4.4 NetBSD_Secure_Shell-20061114 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1.2
debug1: SSH2_MSG_KEXINIT sent

and here it hangs, until the other side times out.
When using IPv4 it works fine.

strace shows that ssh hangs in a select() call:
....
read(3, "u", 1) = 1
read(3, "r", 1) = 1
read(3, "e", 1) = 1
read(3, "_", 1) = 1
read(3, "S", 1) = 1
read(3, "h", 1) = 1
read(3, "e", 1) = 1
read(3, "l", 1) = 1
read(3, "l", 1) = 1
read(3, "-", 1) = 1
read(3, "2", 1) = 1
read(3, "0", 1) = 1
read(3, "0", 1) = 1
read(3, "6", 1) = 1
read(3, "1", 1) = 1
read(3, "1", 1) = 1
read(3, "1", 1) = 1
read(3, "4", 1) = 1
read(3, "\n", 1) = 1
write(3, "SSH-2.0-OpenSSH_4.7p1 Debian-8ub"..., 40) = 40
fcntl64(3, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
write(3, "\0\0\3\24\10\24\264\267\3426\226\\6\254\305\267H\265QT"..., 792) = 792
select(4, [3], NULL, NULL, NULL

It is apparently an Ubuntu 8.04.1-specific problem, since some other people I polled on a #netbsd irc channel reported that they could repeat the problem on Ubuntu-based machines but the problem did not occur for other Linux distributions. I'm using eeebuntu on an Eee 901 which is based on 8.04.01.
I found a xubuntu 8.04 live cd which does not show the problem so probably the problem is quite specific.

rhialto@glicca:~$ lsb_release -rd
Description: Ubuntu 8.04.1
Release: 8.04
rhialto@glicca:~$ apt-cache policy ssh
ssh:
  Installed: (none)
  Candidate: 1:4.7p1-8ubuntu1.2
  Version table:
     1:4.7p1-8ubuntu1.2 0
        500 http://nl.archive.ubuntu.com hardy-updates/main Packages
        500 http://security.ubuntu.com hardy-security/main Packages
     1:4.7p1-8ubuntu1 0
        500 http://nl.archive.ubuntu.com hardy/main Packages

hm, does that think I don't have ssh installed? I wonder which ssh I am then supposed to be using...
It does claim to be "OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007"

I expect the ssh client to continue after SSH2_MSG_KEXINIT, like when using IPv4.

[Rhialto (rhialto-xs4all)]

Ah, it is openssh-client that is installed:

rhialto@glicca:~$ apt-cache policy openssh-client
openssh-client:
  Installed: 1:4.7p1-8ubuntu1.2
  Candidate: 1:4.7p1-8ubuntu1.2
  Version table:
 *** 1:4.7p1-8ubuntu1.2 0
        500 http://nl.archive.ubuntu.com hardy-updates/main Packages
        500 http://security.ubuntu.com hardy-security/main Packages
        100 /var/lib/dpkg/status
     1:4.7p1-8ubuntu1 0
        500 http://nl.archive.ubuntu.com hardy/main Packages

[bdot (bdot)]

Just had the same exact problem logging from an Ubuntu 8.10 laptop (OpenSSH_5.1p1 Debian-3ubuntu1, OpenSSL 0.9.8g 19 Oct 2007) *TO* and OpenBSD 4.0 box. SSH would hang at:

debug1: SSH2_MSG_KEXINIT sent

I found this: http://kerneltrap.org/mailarchive/openbsd-misc/2007/4/24/148531

and I added a "flags S/SA" to the lines which already contained a "keep state". Reloaded pf and I immediately managed to connect.

BTW, I am using a wired connection (there are reports of ssh hanging with wireless connections, but this is not the case)

[Caspar Clemens Mierau (leitmedium)]

Thanks for taking the time to report this issue. As already mentioned this seems to be an error in the handling of Ubuntu Openssh generated packages by the *bsd firewall pf. Marking therefore as invalid here. Feel free to reopen at any time.

[Rhialto (rhialto-xs4all)]

While the referenced email thread ( http://kerneltrap.org/mailarchive/openbsd-misc/2007/4/24/148531 ) is quite interesting, it doesn't seem to refer to IPv6 at all. I also don't use the pf packet filter (firewall), but ipf, and not even between my internal hosts. So the cited possible solution doesn't appear to be relevant at all.

I made a packet trace with wireshark on both ends, and it seems to indicate a TCP checksum error by the sender (Ubuntu). This is seen from both the Ubuntu side and the NetBSD side. I have attached a text version, where I have somewhat anonymised the Ethernet and IPv6 addresses.
So that would make it a kernel or an ethernet driver issue.

The Ubuntu machine is an Eee 901.

[ 51.729198] ATL1e: eth0 NIC Link is Up<100 Mbps Full Duplex>

[Scott Zawalski (cowbud)]

I have the same problem and it is quite simple to repeat even on your local computer:

 [~]$ssh -vvv localhost
OpenSSH_5.1p1 Debian-5ubuntu1, OpenSSL 0.9.8g 19 Oct 2007
debug1: Reading configuration data /home/cowbud/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [::1] port 22.

It will hang at debug1 for a good 5 minutes then continue on to IPv4.

This is really a drag when I connect to a remote ssh server and forward a lot of ssh ports to random local ports. It makes sshing to localhost painfully slow.

However if I ssh to 127.0.0.1 it goes immediately since it bypasses all ipv6 fun.

[Jonathan Marsden (jmarsden)]

I am unable to duplicate this issue on my local
Ubuntu Intrepid 8.10 (x64) desktop, even before
making any of the changes suggested in this message.

For those who are affected by this:

  ssh -4 somehost

will use IPv4 only to connect, which would appear to
be a viable workaround.

To make IPv4 only a global default for the client,
edit /etc/ssh/ssh_config and add a line

  AddressFamily inet

to the end of the default file (i.e., in the entry for "Host *).

At the server end of things, one can configure sshd to
use IPv4 only by adding a line

ListenAddress 0.0.0.0

to /etc/ssh/sshd_config and restarting sshd.

[Scott Zawalski (cowbud)]

Do you have IPV6 enabled or did you blacklist it like most people did? an ifconfig <dev> will tell you if you have an ipv6 address. Also do you have localhost defined in /etc/hosts like is now default in Jaunty?

The workaround works fine, but if we are going to enable ipv6 by default a real fix needs to be put into place.

[Rhialto (rhialto-xs4all)]

I now have eeebuntu 2.0 (based on ubuntu 8.10, kernel 2.27.8-eeepc) on my Eee, and strangely enough, I don't get an IPv6 address most of the time. Just now I tried restarting rtadvd (the routing advertising daemon) on the router and after that, I did have an IPv6 address (but that may be coincidence). Anyway, the ssh connection works fine.
So the problem fortunately seems limited to a small set of kernel versions.

[DaveAbrahams (boostpro)]

I can confirm this problem on Intrepid with kernel 2.6.27-11-server

This really bites when you try to SSH in from a Mac to an avahi name, which seems to go with ipv6 by default.

[Bryan Quigley (bryanquigley)]

"ssh -6 ip6-localhost" should trigger this, but it does not at least on 12.04.

Does anyone else still see this issue?

[Mike Bordignon (2-3ike-q)]

I'm seeing this same issue on Ubuntu 14.04 LTS using OpenSSH 1.0.1f.

mab@source:~$ ssh -v mtaxxx
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to mtaxxx [<ip address hidden>] port 22.
debug1: Connection established.
debug1: identity file /home/mab/.ssh/id_rsa type -1
debug1: identity file /home/mab/.ssh/id_rsa-cert type -1
debug1: identity file /home/mab/.ssh/id_dsa type 2
debug1: identity file /home/mab/.ssh/id_dsa-cert type -1
debug1: identity file /home/mab/.ssh/id_ecdsa type -1
debug1: identity file /home/mab/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/mab/.ssh/id_ed25519 type -1
debug1: identity file /home/mab/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr <email address hidden> none
debug1: kex: client->server aes128-ctr <email address hidden> none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by <ip6 address>

I can ping6 the host, see the traffic getting there in tcpdump, but SSH won't connect (over v6).

[Mike Bordignon (2-3ike-q)]

Update: it works if I change the cipher (to blowfish, in this instance). Doesn't work using aes128.