SSH fails with connection timed out - in VPN and hangs here "expecting SSH2_MSG_KEX_ECDH_REPLY" + Ubuntu 16.04.6 LTS

[Jay]

Hello Team,

SSH timeout issue, once connect to VPN.

Environment

======
Dell XPS 9570
Ubuntu 16.04.6 Xenial Xerus)
kernel - 4.15.0-55-generic

$dpkg -l | grep -i openssh
ii openssh-client 1:7.2p2-4ubuntu2.8 -->
ii openssh-server 1:7.2p2-4ubuntu2.8
ii openssh-sftp-server 1:7.2p2-4ubuntu2.8

VPN tunnel info
====
vpn0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:IP P-t-P:xx Mask:255.255.252.0
          inet6 addr: fe80::b8e2:bea4:2e62:fe08/64 Scope:Link
          UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1406 Metric:1
          RX packets:962 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1029 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:87839 (87.8 KB) TX bytes:238740 (238.7 KB)

Issue
====
Unable to connect to any host via ssh or sftp after VPN connection

Tried
=====

Reinstalled the openssh-client package and still no luck. May I know why the default cipher is not taking/hanging? Please let me know . There were no recent changes.

Workaround
===
Able to connect to ssh / sftp $ssh -c aes128-ctr user@IP

Below is the debug ssh client logs ===
======

$ssh -vvv user@ip
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "IP" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to IP [IP] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to IP:22 as 'user'
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: <email address hidden>,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: <email address hidden>,aes128-ctr,aes192-ctr,aes256-ctr,<email address hidden>,<email address hidden>,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: <email address hidden>,aes128-ctr,aes192-ctr,aes256-ctr,<email address hidden>,<email address hidden>,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,<email address hidden>,zlib
debug2: compression stoc: none,<email address hidden>,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,<email address hidden>,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: <email address hidden>,aes128-ctr,aes192-ctr,aes256-ctr,<email address hidden>,<email address hidden>
debug2: ciphers stoc: <email address hidden>,aes128-ctr,aes192-ctr,aes256-ctr,<email address hidden>,<email address hidden>
debug2: MACs ctos: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,<email address hidden>
debug2: compression stoc: none,<email address hidden>
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: <email address hidden>
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: <email address hidden> MAC: <implicit> compression: none
debug1: kex: client->server cipher: <email address hidden> MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

<< Hangs here >>

Please shed some views

Thanks
Jay

[Jay (jayram1989)]

Tried with putty no issues.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1874257

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Jay (jayram1989)]

Hello Team,

We tried to change the MTU setting to 1200, still no luck.

Shared all information in the comment.

Please let me know

Thanks
Jay

[Jay (jayram1989)]

After changing the VPN interface MTU to 1100, fixed the problem.

May I know why this behaviour occurred?

[Jay (jayram1989)]

Hello,

This is an issue with VPN MTU configuration. May I know the root cause why this is causing the issue? After adjusted the MTU , this is fixed.

As the same time putty works without any modification.

Why ssh client not negotiating properly with the default MTU

It could be a noisy or unreliable connection to the server. Probably that data corruption is happening to the packet when it is sent, which is my assumption

https://en.wikipedia.org/wiki/Maximum_transmission_unit
https://en.wikipedia.org/wiki/Forward_error_correction

From ubuntu community point of view any suggestions>

thanks
Jay

[Lucas Kanashiro (lucaskanashiro)]

Thank you for taking the time to file a bug report.

In order to reproduce the bug you faced, could you please share your config files? OpenSSH and OpenVPN ones. Otherwise we cannot do anything.

Since there is not enough information in your report to begin triage or to
differentiate between a local configuration problem and a bug in Ubuntu, I
am marking this bug as "Incomplete". We would be grateful if you would:
provide a more complete description of the problem, explain why you
believe this is a bug in Ubuntu rather than a problem specific to your
system, and then change the bug status back to "New".

For local configuration issues, you can find assistance here:
http://www.ubuntu.com/support/community

[Jay (jayram1989)]

Hello Lucas,

Thanks for taking this up. Yes I can provide you the required information.

Why I believe this is a bug, is due to using putty there were no issues.

So ssh client is causing the issue. For fixing reducing the MTU of VPN tunnel interface fixed the issue.

So I think there is still a problem with ssh client with negotiating the SSH communication.

===
$dpkg -l | grep -i openssh
ii openssh-client 1:7.2p2-4ubuntu2.8 -->
ii openssh-server 1:7.2p2-4ubuntu2.8
ii openssh-sftp-server 1:7.2p2-4ubuntu2.8

$cat /etc/ssh/ssh_config

# This is the ssh client system-wide configuration file. See
# ssh_config(5) for more information. This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
# 1. command line options
# 2. user-specific file
# 3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options. For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
# ForwardAgent no
# ForwardX11 no
# ForwardX11Trusted yes
# RhostsRSAAuthentication no
# RSAAuthentication yes
# PasswordAuthentication yes
# HostbasedAuthentication no
# GSSAPIAuthentication no
# GSSAPIDelegateCredentials no
# GSSAPIKeyExchange no
# GSSAPITrustDNS no
# BatchMode no
# CheckHostIP yes
# AddressFamily any
# ConnectTimeout 0
# StrictHostKeyChecking ask
# IdentityFile ~/.ssh/identity
# IdentityFile ~/.ssh/id_rsa
# IdentityFile ~/.ssh/id_dsa
# IdentityFile ~/.ssh/id_ecdsa
# IdentityFile ~/.ssh/id_ed25519
# Port 22
# Protocol 2
# Cipher 3des
# Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
# MACs hmac-md5,hmac-sha1,<email address hidden>,hmac-ripemd160
# EscapeChar ~
# Tunnel no
# TunnelDevice any:any
# PermitLocalCommand no
# VisualHostKey no
# ProxyCommand ssh -q -W %h:%p gateway.example.com
# RekeyLimit 1G 1h
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no
ForwardX11Trusted yes
ForwardX11Timeout 596h

For VPN client connection we are using open-connect, so usually connect via
$sudo openconnect <domain-name>

$dpkg -l | grep openconnect
ii libopenconnect5:amd64 7.06-2build2 amd64 open client for Cisco AnyConnect VPN - shared library
ii openconnect 7.06-2build2 amd64 open client for Cisco AnyConnect VPN

I'm marking this bug status to confirmed.

Thanks
Jay

[Dan Lenski (lenski)]

It appears you are using *OpenConnect*, although both the strings OpenVPN and OpenConnect appear in prior posts. These are *completely different* VPN clients.

You are using an *ancient* old release of OpenConnect v7.06. The automatic MTU detection logic has been vastly improved in newer versions of OpenConnect: https://www.infradead.org/openconnect/changelog.html

So yes, this is indeed a bug in OpenConnect's MTU handling, but likely one which we've long since fixed upstream.

[Christian Ehrhardt  (paelzer)]

Yeah Dan, thanks for chiming in.
In particular that would be at least (but not lmited to) the changes:

8.04
Rework DTLS MTU detection. (#10)
7.08
Support automatic DTLS MTU detection with OpenSSL.
7.07
Automatic DTLS MTU detection.

Ubuntu has these newer versions.
Bionic 18.04 is on 7.08 and the most recent LTS Focal is at 8.05.
The current development release is at the latest 8.09 of openconnect.

These are new features added in 7.07 and 7.08 - IMHO they do not qualify for a SRU release into Xenial [1] - especially since you can "get away" with a config change that mitigates the issue.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates

[Jay (jayram1989)]

Thanks Dan and chris for the update,
I can understand this will not backport in Xenial openconnect version correct.
Because most of our Ubuntu desktop/laptop running on 16.04.6 LTS version and i see it has support agreement until April 2021.

If there is any chance to backport this MTU detection handling on openconnect 7.06, then it would be really great.

Thanks
Jay

[Stuart (stuart-ward)]

See this AskUbuntu item on this bug
https://askubuntu.com/questions/1229456/ssh-fails-with-connection-timed-out-in-vpn-and-hangs-here-expecting-ssh2-msg

I resolved this with the suggested setting:
> As a temporary workaround, setting the KEX algorithm manually solves this problem for me.
> Add KexAlgorithms ecdh-sha2-nistp521 to the corresponding SSH config, or add -oKexAlgorithms=ecdh-sha2-nistp521 to the command line args for one time use.

[Stuart (stuart-ward)]

This is still a current problem

$ dpkg -l | grep -i openvpn
ii network-manager-openvpn 1.8.14-1 amd64 network management framework (OpenVPN plugin core)
ii network-manager-openvpn-gnome 1.8.14-1 amd64 network management framework (OpenVPN plugin GNOME GUI)
ii openvpn 2.5.1-3ubuntu1 amd64 virtual private network daemon

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 21.10
Release: 21.10
Codename: impish

[Dan Lenski (lenski)]

@Stuart-ward wrote…

> This is still a current problem
>
> $ dpkg -l | grep -i openvpn
> ii network-manager-openvpn 1.8.14-1 amd64 network management framework (OpenVPN plugin core)
> ii network-manager-openvpn-gnome 1.8.14-1 amd64 network management framework (OpenVPN plugin GNOME GUI)
> ii openvpn 2.5.1-3ubuntu1 amd64 virtual private network daemon

Once again, OpenConnect and OpenVPN are *not* the same thing. At all.

For your VPN connection, are you using OpenConenct (the original subject of this bug report), or are you using OpenVPN (completely different and should have a separate bug files)?

[vodopad27 (family-gan)]

I faced with same issue after changed router from Zyxel Keenetic Lite to Xiaomi A4 Giga Edition.

My connection use PPPOE on router.

Ubuntu 21.10 and openconnect VPN.

[Lucas Kanashiro (lucaskanashiro)]

@family-gan are you saying this is an issue in Ubuntu Impish (21.10)? It seems to be already fixed in supported releases. Could you share any steps to reproduce it? If you consider the issue you are facing different than the one discussed in this bug please consider filing a separate bug.

[Sergio Durigan Junior (sergiodj)]

Ubuntu Xenial has reached end of standard support, so I marked its task as Won't Fix.