ssh-keyscan : bad host signature when using port option

Bug #1670745 reported by Gautier HUSSON on 2017-03-07
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
portable OpenSSH
Unknown
Unknown
openssh (Debian)
Fix Released
Unknown
openssh (Ubuntu)
High
Unassigned
Xenial
Undecided
Unassigned
Yakkety
Undecided
Unassigned

Bug Description

[Impact]

 * using ssh-keyscan while using the port (-p) option of it will create
   bad entries. They will contain the port and thereby be invalid for
   latter use under the purpose of known_hosts.

 * Fix by backporting upstream fix.

[Test Case]

 * Further evolving from the simplification Josh provided:
Testcase:
$ release=xenial
$ lxc launch ubuntu-daily:${release} ${release}-test-ssh-port-scan-client
$ lxc launch ubuntu-daily:${release} ${release}-test-ssh-port-scan-server
$ lxc exec ${release}-test-ssh-port-scan-server -- sed -i 's/Port 22/Port 2222/' /etc/ssh/sshd_config
$ lxc exec ${release}-test-ssh-port-scan-server -- service ssh restart
$ IP=$(lxc exec ${release}-test-ssh-port-scan-server -- hostname --ip-address)
$ lxc exec ${release}-test-ssh-port-scan-client -- ssh-keyscan -H -p 2222 ${IP}

# See the port in the Hash still

# Install the fixed version in *-client and see the port gone from the output

[Regression Potential]

 * Change is limited to ssh-keyscan (not any touching other parts of openssh)
 * Fix is from upstream (no "Ubuntu special" change)
 * Fix is small and "only" changing string creation (11 lines touched)
 So overall the regression potential should be low.

[Other Info]

 * n/a

---

When I use the port option with ssh-keygen, the result is not compatible with ssh known_host file format.

UBUNTU VERSION :
================
lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04

BAD :
============
:~/.ssh$ cat /etc/issue
Ubuntu 16.04.1 LTS \n \l
:~/.ssh$ ssh-keyscan -v -p [...port...] -t ecdsa -H [...snip...]
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000
# [...snip...]:[...port...] SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Enabling compatibility mode for protocol 2.0
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: <email address hidden>
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: <email address hidden> MAC: <implicit> compression: none
debug1: kex: client->server cipher: <email address hidden> MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
[|1|BEEwVcggbNPf7fUydgU4O+BDoLg=|9SmWBUxFZkpR70Hqq8uqxLAzXFU=]:[...port...] ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLEde+dZfL0TW6Z9jh+gOkW5fG/qeP9JAejKQXdmg9D7CH4NwMrWDEjXBDDo6iirIPAB6M0uUnK2mDw7uUWXYt8=

==> we see the port number because it is not hashed !

GOOD :
============
rm ~/.ssh/known_hosts
:~/$ ssh -p [...port...] [...snip...]
The authenticity of host '[[...snip...]]:[...port...] ([[...snip...]]:[...port...])' can't be established.
ECDSA key fingerprint is SHA256:b/Jx+y3fNWFqOqTzFRI3XGrz33DBtAFFLmQaYQYFRnM.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[[...snip...]]:[...port...],[[...snip...]]:[...port...]' (ECDSA) to the list of known hosts.
[...snip...]@[...snip...]'s password:

:~/$ !cat
cat ~/.ssh/known_hosts
|1|qdg91H9/DMHLO7yGOivI17+WFQI=|B+a6SrzF1GBd3XFvmAvQRnJxLWs= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLEde+dZfL0TW6Z9jh+gOkW5fG/qeP9JAejKQXdmg9D7CH4NwMrWDEjXBDDo6iirIPAB6M0uUnK2mDw7uUWXYt8=
|1|8I/vbrBV04VaUF12JXRwxvAL9So=|ToMf+kRwbSeNertVdUVuG3iLdH8= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLEde+dZfL0TW6Z9jh+gOkW5fG/qeP9JAejKQXdmg9D7CH4NwMrWDEjXBDDo6iirIPAB6M0uUnK2mDw7uUWXYt8=

==> we cannot see the port number as it is well hashed !

REMARKS :
==============
Same problem has already reported here (on macOS): https://github.com/ansible/ansible-modules-extras/issues/2651

It seems that ssh-keyscan version and open-ssh version differs :
dpkg -l | grep openssh :: ii openssh-client 1:7.2p2-4ubuntu2.1 [...]
ssh-keyscan -v [...] :: debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000

It is very annoying because I am trying to manage hand installed VMs with Ansible. For that I want to automate SSH host keys storing in known_hosts database. And because of this bug I can't. (ansible KIKIN project in development).

Thank you,
BR,
Gautier HUSSON.

Joshua Powers (powersj) wrote :

Thanks for the bug report!

Steps to reproduce:

$ lxc launch ubuntu-daily:xenial xenial
# edit /etc/ssh/sshd_config and change port to 2222
# service ssh restart
# ip a to note container IP
# exit
$ ssh-keyscan -H -p 2222 <container IP>

The port will be in the output and not hashed as described in the report.

The linked Github issue did state there is a workaround by getting the values unhashed and then hashing them in a second step. Not saying this is ideal, but it is a workaround:
$ ssh-keyscan -p 2222 <container IP>
$ ssh-keygen -H -f .ssh/authorized_keys && rm .ssh/authorized_keys.old

Because the man page for ssh-keyscan clearly states that -H will include the hostnames and addres and makes no mention of port in the hash, I have filed a bug with openssh to get clarity on the expected behavior and if this should be fixed.

Gautier HUSSON (ghusson) wrote :

Thank you for your action Joshua.

The most weird thing is that it is known to work on debian (as reported here : https://github.com/ansible/ansible-modules-extras/issues/2651) !

For future reference, your bug report URL to openssh is : https://bugzilla.mindrot.org/show_bug.cgi?id=2692

Joshua Powers (powersj) wrote :

As an update, the issue was confirmed and fixed by the maintainer. We have to decide now if and how to SRU this and to which releases.

Changed in openssh (Ubuntu):
importance: Undecided → High
assignee: nobody → Joshua Powers (powersj)
status: New → In Progress
Changed in openssh (Debian):
status: Unknown → New
Changed in openssh (Debian):
status: New → Fix Released
ChristianEhrhardt (paelzer) wrote :

FYI I'll bundle this with the fix for 1668093

ChristianEhrhardt (paelzer) wrote :

FYI - Colin has merged the fix, uploaded to Debian and synced to Zesty.
But the sync is blocked by an issue with another bundled fix (see bug 1668093).

Just checked affected Releases for the SRUs to be prepared:
- Trusty: not affected
- Xenial: affected
- Yakkety: affected

That is just the set I prepare the SRU for anyway, as discussed including the fix in my prep.
And adding a proper SRU Template here now + bug tasks ...

description: updated
ChristianEhrhardt (paelzer) wrote :

I was able to confirm the fix working on the bileto ppas I'm working on.
All further updates will be made in bug 1668093, some time will pass until all tests/verifications are complete and we can sync the package to the SRU unapproved queue.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:7.4p1-9

---------------
openssh (1:7.4p1-9) unstable; urgency=medium

  * Fix null pointer dereference in ssh-keygen; this fixes an autopkgtest
    regression introduced in 1:7.4p1-8.

 -- Colin Watson <email address hidden> Thu, 16 Mar 2017 13:43:15 +0000

Changed in openssh (Ubuntu):
status: In Progress → Fix Released
tags: added: server-next

Hello Gautier, or anyone else affected,

Accepted openssh into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:7.3p1-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in openssh (Ubuntu Yakkety):
status: New → Fix Committed
tags: added: verification-needed
Changed in openssh (Ubuntu Xenial):
status: New → Fix Committed
Łukasz Zemczak (sil2100) wrote :

Hello Gautier, or anyone else affected,

Accepted openssh into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:7.2p2-4ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

description: updated
Gautier HUSSON (ghusson) wrote :

First : thank you everybody for all your work !

At now I am not able to see the difference. But I think I may not doing the right thing in order to test :
wget "http://launchpadlibrarian.net/254059000/openssh-client_7.2p2-4_amd64.deb"
dpkg -i openssh-client_7.2p2-4_amd64.deb
ssh-keyscan -v -t dsa -H -p 22000 -T 5 test.[snip]
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000
# test.liberasys.com:22000 SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Enabling compatibility mode for protocol 2.0
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: <email address hidden>
debug1: kex: host key algorithm: ssh-dss
debug1: kex: server->client cipher: <email address hidden> MAC: <implicit> compression: none
debug1: kex: client->server cipher: <email address hidden> MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
[|1|xkG3IqIo2wV4LVdHUIdFxpV+cN0=|qPnIs826Ny8lK2r9MYnN94rt+3w=]:22000 ssh-dss AAAAB3NzaC1kc3MAAACBAOWWBwm[snip]

==> OpenSSH_6.7p1 is not the right version... although :

ssh -V
OpenSSH_7.2p2 Ubuntu-4, OpenSSL 1.0.2g 1 Mar 2016

root@ghu-thinkpad:~# dpkg -l | grep ssh
ii libssh-4:amd64 0.6.3-4.3 amd64 tiny C SSH library (OpenSSL flavor)
ii libssh-gcrypt-4:amd64 0.6.3-4.3 amd64 tiny C SSH library (gcrypt flavor)
ii libssh2-1:amd64 1.5.0-2ubuntu0.1 amd64 SSH2 client-side library
ii openssh-client 1:7.2p2-4 amd64 secure shell (SSH) client, for secure access to remote machines
ii python-paramiko 1.16.0-1 all Make ssh v2 connections with Python (Python 2)
ii sshpass 1.05-1 amd64 Non-interactive ssh password authentication

It may be due to the libssh that I have not updated ?
RQ : I preafer to install updated packages directly rather than playing with dpgk database (I may be wrong on this point).

Thank you for your help.
B.R.
Gautier.

ChristianEhrhardt (paelzer) wrote :

@Gautier - thanks for trying to verify - I'm currently doing the same.
The known_hosts file you have still has the broken content.
The fix cannot change your existing known_hosts file to be good.
You'd have to remove the broken lines and add them again with ssh-keyscan.
But the fix in keyscan is in a way that now it should create valid entries that work.

So when you rescan your target afterwards and appand that to your known hosts it will be good then.

ChristianEhrhardt (paelzer) wrote :

Also after enabling proposed you can install "just the package and its dependencies" by instead of running "apt upgrade" just running in this case "apt install openssh-client".

tags: added: verification-done
removed: verification-needed
ChristianEhrhardt (paelzer) wrote :

Prior to updating when scanning from a target I still have ports in the output:

ssh-keyscan -H -p 2222 10.0.4.153
# 10.0.4.153:2222 SSH-2.0-OpenSSH_7.4p1 Ubuntu-10
[|1|nTWrNxylhhwkyicxuuS/5p2Tlts=|ziw5AkY9sg/oRSpkYIuxswDgIf8=]:2222 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID4/W/XQvgco3vziMDLT5PyHwuqY9pupyB2NNvpF+9bR

Upgrading to proposed version.

Afterwards the entry looks good like:
# 10.0.4.153:2222 SSH-2.0-OpenSSH_7.4p1 Ubuntu-10
|1|A5Uker2NsxV0OgNrMNT+aaUDrJM=|bsWW57Lhko5h+t6BtjU9u/wkAkw= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID4/W/XQvgco3vziMDLT5PyHwuqY9pupyB2NNvpF+9bR

Verified on Xenial and Yakkety.

@Gautier - please go on with your own verification and in the worst case if you can prove it not working change it back to verification-needed or even -failed. I'm away the next few days, but I'm sure cjwatson or powerj who are on this bug can help you in case it is needed.

Gautier, as well as Christian's comments:

(1) In your output, 6.7p1 is the version of the server you're connecting
    to, not the client version.
(2) libssh is irrelevant here.
(3) 7.2p2-4 is not the fixed version. See comment #9 on this bug.

Gautier HUSSON (ghusson) wrote :

Hello,
Thank you Colin for your clarification.
I don't understand why the package I downloaded was not the good one.
Anyway, after having followed selective proposed update documentation, I have :

#ssh -V
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g 1 Mar 2016
#dpkg -l | grep openssh
ii openssh-client 1:7.2p2-4ubuntu2.2 amd64 secure shell (SSH) client, for secure access to remote machines

==> and now, the problem is fixed. I can confirm that the port is well hashed.

For information, what will be the releasing process of this proposed patch ? Will it be included main stream ?

Thank you for your work,
Sincerely,
Gautier.

Joshua Powers (powersj) on 2017-04-19
Changed in openssh (Ubuntu):
assignee: Joshua Powers (powersj) → nobody
Joshua Powers (powersj) wrote :

@ghusson,
This will be available soon in Xenial and Yakkety. As of today it is still in -proposed.

@sru-team,
Based on ghusson's comment and my own testing of Xenial (7.2p2-4ubuntu2.2) and Yakkety (7.3p1-1ubuntu0.1) I am marking this as verification-done. I followed the test case I originally developed and is documented in the SRU template and confirmed that the port is no longer in the ssh-keyscan output.

tags: added: verification-done-xenial verification-done-yakkety

The verification of the Stable Release Update for openssh has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:7.3p1-1ubuntu0.1

---------------
openssh (1:7.3p1-1ubuntu0.1) yakkety; urgency=medium

  * Fix ssh-keygen -H accidentally corrupting known_hosts that contained
    already-hashed entries (LP: #1668093).
  * Fix ssh-keyscan to correctly hash hosts with a port number (LP: #1670745).

 -- Christian Ehrhardt <email address hidden> Wed, 15 Mar 2017 14:25:22 +0100

Changed in openssh (Ubuntu Yakkety):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:7.2p2-4ubuntu2.2

---------------
openssh (1:7.2p2-4ubuntu2.2) xenial; urgency=medium

  * Fix ssh-keygen -H accidentally corrupting known_hosts that contained
    already-hashed entries (LP: #1668093).
  * Fix ssh-keyscan to correctly hash hosts with a port number (LP: #1670745).

 -- Christian Ehrhardt <email address hidden> Wed, 15 Mar 2017 13:16:56 +0100

Changed in openssh (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.