14.04 host can not ssh to a Cisco router

[andrew yourtchenko]

I can not connect from a freshly installed 14.04 box to a Cisco CSR1000V. 12.04 connects with no issues.

Debugging this, the symptoms are identical to this bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1026430

see the logs below (hostnames, addresses, and usernames below are edited).

Failing attempt (the default settings):

$ ssh -vvv <email address hidden>
OpenSSH_6.6, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to router.example [2001:db8::1] port 22.
debug1: Connection established.
debug1: SELinux support disabled
debug1: identity file /home/ubuntu/.ssh/id_rsa type -1
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6p1 Ubuntu-2ubuntu1
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,<email address hidden>,<email address hidden>,<email address hidden>,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,<email address hidden>
debug2: kex_parse_kexinit: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,hmac-md5,hmac-sha1,<email address hidden>,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,<email address hidden>,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,hmac-md5,hmac-sha1,<email address hidden>,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,<email address hidden>,hmac-sha1-96,hmac-md5-96debug2: kex_parse_kexinit: none,<email address hidden>,zlib
debug2: kex_parse_kexinit: none,<email address hidden>,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug2: mac_setup: setup hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<7680<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Connection closed by 2001:db8::1

Working attempt:

$ ssh -vvv -o KexAlgorithms=diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 <email address hidden>
OpenSSH_6.6, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to router.example [2001:db8::1] port 22.
debug1: Connection established.
debug1: identity file /home/ubuntu/.ssh/id_rsa type -1
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6p1 Ubuntu-2ubuntu1
debug1: Remote protocol version 1.99, remote software version Cisco-1.25
debug1: no match: Cisco-1.25
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "router.example" from file "/home/ubuntu/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/ubuntu/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: <email address hidden>,<email address hidden>,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: <email address hidden>,<email address hidden>,ssh-rsa,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,<email address hidden>,<email address hidden>,<email address hidden>,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,<email address hidden>
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,<email address hidden>,<email address hidden>,<email address hidden>,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,<email address hidden>
debug2: kex_parse_kexinit: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,hmac-md5,hmac-sha1,<email address hidden>,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,<email address hidden>,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,hmac-md5,hmac-sha1,<email address hidden>,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,<email address hidden>,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,<email address hidden>,zlib
debug2: kex_parse_kexinit: none,<email address hidden>,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/ubuntu/.ssh/id_rsa ((nil)),
debug2: key: /home/ubuntu/.ssh/id_dsa ((nil)),
debug2: key: /home/ubuntu/.ssh/id_ecdsa ((nil)),
debug2: key: /home/ubuntu/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,keyboard-interactive,password
debug3: start over, passed a different list publickey,keyboard-interactive,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/ubuntu/.ssh/id_rsa
debug3: no such identity: /home/ubuntu/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_dsa
debug3: no such identity: /home/ubuntu/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa
debug3: no such identity: /home/ubuntu/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ed25519
debug3: no such identity: /home/ubuntu/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: packet_send2: adding 32 (len 24 padlen 8 extra_pad 64)
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to router.example ([2001:db8::1]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IPV6_TCLASS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env XDG_VTNR
debug3: Ignored env LESS_TERMCAP_mb
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env LESS_TERMCAP_md
debug3: Ignored env LESS_TERMCAP_me
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env BYOBU_CONFIG_DIR
debug3: Ignored env HUSHLOGIN
debug3: Ignored env LESS_TERMCAP_ue
debug3: Ignored env BYOBU_TTY
debug3: Ignored env BYOBU_READLINK
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env GREP_COLORS
debug3: Ignored env BYOBU_RUN_DIR
debug3: Ignored env BYOBU_DISTRO
debug3: Ignored env BYOBU_DATE
debug3: Ignored env TMUX
debug3: Ignored env LESS_TERMCAP_us
debug3: Ignored env BYOBU_BACKEND
debug3: Ignored env BYOBU_SED
debug3: Ignored env PATH
debug3: Ignored env MAIL
debug3: Ignored env BYOBU_DARK
debug3: Ignored env PWD
debug3: Ignored env BYOBU_ULIMIT
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env TMUX_PANE
debug3: Ignored env BYOBU_PYTHON
debug3: Ignored env BYOBU_CHARMAP
debug3: Ignored env BYOBU_WINDOW_NAME
debug3: Ignored env HOME
debug3: Ignored env SHLVL
debug3: Ignored env XDG_SEAT
debug3: Ignored env LANGUAGE
debug3: Ignored env BYOBU_LIGHT
debug3: Ignored env LOGNAME
debug3: Ignored env LESS_TERMCAP_so
debug3: Ignored env BYOBU_PAGER
debug3: Ignored env BYOBU_ACCENT
debug3: Ignored env LESSOPEN
debug3: Ignored env BYOBU_PREFIX
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env BYOBU_HIGHLIGHT
debug3: Ignored env LESSCLOSE
debug3: Ignored env LESS_TERMCAP_se
debug3: Ignored env BYOBU_TIME
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 8192 rmax 4096
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0

router#

version information:

$ lsb_release -rd
Description: Ubuntu 14.04 LTS
Release: 14.04

$ apt-cache policy openssh-client
openssh-client:
  Installed: 1:6.6p1-2ubuntu1
  Candidate: 1:6.6p1-2ubuntu1
  Version table:
 *** 1:6.6p1-2ubuntu1 0
        500 http://be.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status

CSR1000V version:

router#sh ver
Cisco IOS XE Software, Version 03.12.00.S - Standard Support Release
Cisco IOS Software, CSR1000V Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(2)S, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Wed 26-Mar-14 21:09 by mcpre

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[andrew yourtchenko (ayourtch)]

The bug filed against the Cisco ssh client is CSCuo76464 - its fix resolves the issue.