[cosmic regression] fails to parse known_hosts, resulting in SSH_SERVER_FOUND_OTHER error for hostkey verification

Bug #1799665 reported by Martin Pitt on 2018-10-24
This bug affects 1 person
Affects Status Importance Assigned to Milestone
libssh (Ubuntu)
Martin Pitt
Martin Pitt

Bug Description

Ubuntu 18.10's libssh 0.8.1 regresses parsing of known_hosts. This happens (sometimes) if there are multiple known_host key types (e. g. ssh-rsa and ssh-ed25519), then it can happen that ssh_session_is_known_server() fails with SSH_SERVER_FOUND_OTHER [1].

I noticed this with testing Cockpit on Ubuntu 18.10 [2], which has a few test cases exercising cockpit-ssh (which uses libssh), e. g. [3]. The scenario is a FreeIPA centrally managed known_hosts file with these entries:

x0.cockpit.lan ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCv5sLKfLDuEAbTcHC3eOgJM+Ot7F077KewD4e1lGzfw300Jo4xnuPsoJEVSCR7OjsYQCnuVGlqtlavMCLFzIBNk06iTBg/nl+W+xa3CFNITbAjiBif7SeY0XL6Xeqzb1VYXNVfwKQKpcGIbDne6jyou4wRZV1eay03FHTSkd2+XKM6GOUGlkEUoPyAwYPHqoKUYiiyBxJs20l/peXVx6jsGgs2Sc6gl3KJP0TB2E7ncD1pWHGRtiNshFFVarw/YKr+Rs+KhiVS3CAAfYDhpBNWXOwTKyx2euJjAhsRF10bx6pnuadSEpT8Ufo5/YFIVAD1GHptULSzVjUoJm6ktoHB
x0.cockpit.lan ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkJ6CaqhzUhrbpbVmZ8BmZZgM3u6BukZ6HFB2a4NLQBdgpHlHbxoJ47ocTImctyFMiDi0y6vCb4tFuZgp6Krmk= root@(none)
x0.cockpit.lan ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINK6gcOyH4OhiKPcNr33Kl6e+wFAUy9tGFBU/o4yWkxh root@(none)

Connecting to that host with the standard ssh client works:

$ ssh -vv x0.cockpit.lan
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:MgfkN6HEl+pdz0X7+6q08IVkUZOtEDzfA6V18Wm9DgA
debug1: Host 'x0.cockpit.lan' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/sss/pubconf/known_hosts:3
<email address hidden>@x0:~$

But not with cockpit-ssh. This shows the JSON protocol (note that you need to copy&paste the correct cookie value from the response):

$ G_MESSAGES_DEBUG=cockpit-ssh cockpit-bridge --interact=---

{ "command": "open", "channel": "c", "payload": "echo", "host": "x0.cockpit.lan", "user": "<email address hidden>" }


{"command":"authorize", "response": "password foobarfoo", "cookie": "session107271540364829"}
(cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.049: cockpit-ssh x0.cockpit.lan: host not known in any local file, asking sssd
(cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.472: cockpit-ssh x0.cockpit.lan: using known hosts file /tmp/known-hosts.IDKNRZ
(cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.542: cockpit-ssh x0.cockpit.lan: connected
cockpit-ssh-Message: 03:07:30.828: cockpit-ssh x0.cockpit.lan: host key for this server changed key type: ssh-ed25519

{"command":"close","host-key":"x0.cockpit.lan ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINK6gcOyH4OhiKPcNr33Kl6e+wFAUy9tGFBU/o4yWkxh\n","host-fingerprint":"a0:27:1e:80:de:fd:4b:8a:0d:9d:a9:b6:42:7d:5c:b9","problem":"invalid-hostkey","error":"invalid-hostkey","auth-method-results":{},"channel":"c"}

The "host key for this server changed key type" is the effect of this bug.


[IMPACT]: libssh connections that worked in previous Ubuntu releases now may fail on host key verification

[TEST CASE]: See reproducer below. This isn't too easy to reproduce for someone else, so I'm happy to do the validation myself. This can also be verified with the Cockpit integration tests:

    bots/image-prepare ubuntu-stable
    TEST_OS=ubuntu-stable test/verify/check-realms TestRealms.testIpa

[REGRESSION POTENTIAL]: In principle these patches could break known_hosts validation further. However, these fixes have been in Debian testing for a while and validated through e. g. Cockpit's tests (which exercise cockpit-ssh quite heavily). There are also upstream unit tests, and while they didn't pick up that particular regression, they at least make sure that known_hosts verification still works for common cases.
Also, libssh-4 does not have that many reverse dependencies. So overall, I think this is bearable for an SRU, especially as the impact is quite high.

[1] http://api.libssh.org/master/group__libssh__session.html#gac%20bc5d04fe66beee863a0c61a93fdf765
[2] https://github.com/cockpit-project/cockpit/pull/10357
[3] https://fedorapeople.org/groups/cockpit/logs/pull-10357-20181022-204242-8672df31-verify-ubuntu-stable/log.html#186

Martin Pitt (pitti) wrote :

This works fine with the latest libssh 0.8.4, when building the Debian unstable package for 18.10, it works fine:

$ G_MESSAGES_DEBUG=cockpit-ssh cockpit-bridge --interact=---

{ "command": "open", "channel": "c", "payload": "echo", "host": "x0.cockpit.lan", "user": "<email address hidden>" }


{"command":"authorize", "response": "password foobarfoo", "cookie":"session109311540371777"}
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:06.880: cockpit-ssh x0.cockpit.lan: host not known in any local file, asking sssd
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.165: cockpit-ssh x0.cockpit.lan: using known hosts file /tmp/known-hosts.KIBHRZ
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.248: cockpit-ssh x0.cockpit.lan: connected
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.249: cockpit-ssh x0.cockpit.lan: verified host key
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.278: cockpit-ssh x0.cockpit.lan: agent auth failed
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.282: cockpit-ssh x0.cockpit.lan: Got prompt Password: prompt
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.995: cockpit-ssh x0.cockpit.lan: Couldn't set COCKPIT_REMOTE_PEER: Channel request env failed
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.995: cockpit-ssh x0.cockpit.lan: opened channel
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:08.071: cockpit-ssh x0.cockpit.lan: queued 162 bytes
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:08.071: cockpit-ssh x0.cockpit.lan: wrote 162 bytes


I bisected this to this upstream fix: https://git.libssh.org/projects/libssh.git/commit/?id=45058285fca549876449afef2c32833b24817e77 . I prepare an SRU.

There are also a few other known_hosts fixes which should get included:


Changed in libssh (Ubuntu Cosmic):
status: New → In Progress
Martin Pitt (pitti) wrote :

Added SRU information and uploaded SRU to unapproved queue.

description: updated

Hello Martin, or anyone else affected,

Accepted libssh into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libssh/0.8.1-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in libssh (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Martin Pitt (pitti) wrote :

I installed libssh-4 0.8.1-1ubuntu0.2 from cosmic-proposed, and confirm that the manual ssh connection with "cockpit-ssh" as well as all the integration tests that involve talking to remote machines through ssh now work.

tags: added: verification-done verification-done-cosmic
removed: verification-needed verification-needed-cosmic

The verification of the Stable Release Update for libssh has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in libssh (Ubuntu):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libssh - 0.8.1-1ubuntu0.2

libssh (0.8.1-1ubuntu0.2) cosmic-proposed; urgency=medium

  * Fix regressions with known_host parsing: ssh_session_is_known_server()
    sometimes fails with SSH_SERVER_FOUND_OTHER if known_hosts contains
    multiple key types for the target host. (LP: #1799665)
  * Also backport fixes for some related bugs:
    - Use all supported hostkey algorithms for negotiation
    - Honor more host key algorithms than the first one (ssh-ed25519)
    - Use the correct name for ECDSA keys for host key negotiation

 -- Martin Pitt <email address hidden> Wed, 24 Oct 2018 08:20:23 +0000

Changed in libssh (Ubuntu Cosmic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers