[cosmic regression] fails to parse known_hosts, resulting in SSH_SERVER_FOUND_OTHER error for hostkey verification

Bug #1799665 reported by Martin Pitt
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
libssh (Ubuntu)
Fix Released
Undecided
Martin Pitt
Cosmic
Fix Released
Undecided
Martin Pitt

Bug Description

Ubuntu 18.10's libssh 0.8.1 regresses parsing of known_hosts. This happens (sometimes) if there are multiple known_host key types (e. g. ssh-rsa and ssh-ed25519), then it can happen that ssh_session_is_known_server() fails with SSH_SERVER_FOUND_OTHER [1].

I noticed this with testing Cockpit on Ubuntu 18.10 [2], which has a few test cases exercising cockpit-ssh (which uses libssh), e. g. [3]. The scenario is a FreeIPA centrally managed known_hosts file with these entries:

x0.cockpit.lan ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCv5sLKfLDuEAbTcHC3eOgJM+Ot7F077KewD4e1lGzfw300Jo4xnuPsoJEVSCR7OjsYQCnuVGlqtlavMCLFzIBNk06iTBg/nl+W+xa3CFNITbAjiBif7SeY0XL6Xeqzb1VYXNVfwKQKpcGIbDne6jyou4wRZV1eay03FHTSkd2+XKM6GOUGlkEUoPyAwYPHqoKUYiiyBxJs20l/peXVx6jsGgs2Sc6gl3KJP0TB2E7ncD1pWHGRtiNshFFVarw/YKr+Rs+KhiVS3CAAfYDhpBNWXOwTKyx2euJjAhsRF10bx6pnuadSEpT8Ufo5/YFIVAD1GHptULSzVjUoJm6ktoHB
x0.cockpit.lan ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkJ6CaqhzUhrbpbVmZ8BmZZgM3u6BukZ6HFB2a4NLQBdgpHlHbxoJ47ocTImctyFMiDi0y6vCb4tFuZgp6Krmk= root@(none)
x0.cockpit.lan ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINK6gcOyH4OhiKPcNr33Kl6e+wFAUy9tGFBU/o4yWkxh root@(none)

Connecting to that host with the standard ssh client works:

$ ssh -vv x0.cockpit.lan
[...]
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:MgfkN6HEl+pdz0X7+6q08IVkUZOtEDzfA6V18Wm9DgA
debug1: Host 'x0.cockpit.lan' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/sss/pubconf/known_hosts:3
[...]
<email address hidden>@x0:~$

But not with cockpit-ssh. This shows the JSON protocol (note that you need to copy&paste the correct cookie value from the response):

$ G_MESSAGES_DEBUG=cockpit-ssh cockpit-bridge --interact=---

{ "command": "open", "channel": "c", "payload": "echo", "host": "x0.cockpit.lan", "user": "<email address hidden>" }
---

{"command":"authorize","challenge":"*","cookie":"session107271540364829"}
---

{"command":"authorize", "response": "password foobarfoo", "cookie": "session107271540364829"}
---
(cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.049: cockpit-ssh x0.cockpit.lan: host not known in any local file, asking sssd
(cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.472: cockpit-ssh x0.cockpit.lan: using known hosts file /tmp/known-hosts.IDKNRZ
(cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.542: cockpit-ssh x0.cockpit.lan: connected
cockpit-ssh-Message: 03:07:30.828: cockpit-ssh x0.cockpit.lan: host key for this server changed key type: ssh-ed25519

{"command":"close","host-key":"x0.cockpit.lan ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINK6gcOyH4OhiKPcNr33Kl6e+wFAUy9tGFBU/o4yWkxh\n","host-fingerprint":"a0:27:1e:80:de:fd:4b:8a:0d:9d:a9:b6:42:7d:5c:b9","problem":"invalid-hostkey","error":"invalid-hostkey","auth-method-results":{},"channel":"c"}
---

The "host key for this server changed key type" is the effect of this bug.

SRU INFORMATION:

[IMPACT]: libssh connections that worked in previous Ubuntu releases now may fail on host key verification

[TEST CASE]: See reproducer below. This isn't too easy to reproduce for someone else, so I'm happy to do the validation myself. This can also be verified with the Cockpit integration tests:

    bots/image-prepare ubuntu-stable
    TEST_OS=ubuntu-stable test/verify/check-realms TestRealms.testIpa

[REGRESSION POTENTIAL]: In principle these patches could break known_hosts validation further. However, these fixes have been in Debian testing for a while and validated through e. g. Cockpit's tests (which exercise cockpit-ssh quite heavily). There are also upstream unit tests, and while they didn't pick up that particular regression, they at least make sure that known_hosts verification still works for common cases.
Also, libssh-4 does not have that many reverse dependencies. So overall, I think this is bearable for an SRU, especially as the impact is quite high.

[1] http://api.libssh.org/master/group__libssh__session.html#gac%20bc5d04fe66beee863a0c61a93fdf765
[2] https://github.com/cockpit-project/cockpit/pull/10357
[3] https://fedorapeople.org/groups/cockpit/logs/pull-10357-20181022-204242-8672df31-verify-ubuntu-stable/log.html#186

Revision history for this message
Martin Pitt (pitti) wrote :

This works fine with the latest libssh 0.8.4, when building the Debian unstable package for 18.10, it works fine:

$ G_MESSAGES_DEBUG=cockpit-ssh cockpit-bridge --interact=---

{ "command": "open", "channel": "c", "payload": "echo", "host": "x0.cockpit.lan", "user": "<email address hidden>" }
---

{"command":"authorize","challenge":"*","cookie":"session109311540371777"}
---

{"command":"authorize", "response": "password foobarfoo", "cookie":"session109311540371777"}
---
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:06.880: cockpit-ssh x0.cockpit.lan: host not known in any local file, asking sssd
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.165: cockpit-ssh x0.cockpit.lan: using known hosts file /tmp/known-hosts.KIBHRZ
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.248: cockpit-ssh x0.cockpit.lan: connected
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.249: cockpit-ssh x0.cockpit.lan: verified host key
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.278: cockpit-ssh x0.cockpit.lan: agent auth failed
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.282: cockpit-ssh x0.cockpit.lan: Got prompt Password: prompt
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.995: cockpit-ssh x0.cockpit.lan: Couldn't set COCKPIT_REMOTE_PEER: Channel request env failed
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:07.995: cockpit-ssh x0.cockpit.lan: opened channel
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:08.071: cockpit-ssh x0.cockpit.lan: queued 162 bytes
(cockpit-ssh:10931): cockpit-ssh-DEBUG: 05:03:08.071: cockpit-ssh x0.cockpit.lan: wrote 162 bytes

{"command":"ready","channel":"c"}
---

I bisected this to this upstream fix: https://git.libssh.org/projects/libssh.git/commit/?id=45058285fca549876449afef2c32833b24817e77 . I prepare an SRU.

There are also a few other known_hosts fixes which should get included:

    https://git.libssh.org/projects/libssh.git/commit/?id=35a64554899f142a2b8b68c79007ad9c3ce00cb1
    https://git.libssh.org/projects/libssh.git/commit/?id=c1a8c41c5daf79e37aa5fde67dd94c8596e81102
    https://git.libssh.org/projects/libssh.git/commit/?id=893b69d82b4435973ec4d15aaecdf352f5f827e2

Changed in libssh (Ubuntu Cosmic):
status: New → In Progress
Revision history for this message
Martin Pitt (pitti) wrote :

Added SRU information and uploaded SRU to unapproved queue.

description: updated
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Martin, or anyone else affected,

Accepted libssh into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libssh/0.8.1-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in libssh (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Revision history for this message
Martin Pitt (pitti) wrote :

I installed libssh-4 0.8.1-1ubuntu0.2 from cosmic-proposed, and confirm that the manual ssh connection with "cockpit-ssh" as well as all the integration tests that involve talking to remote machines through ssh now work.

tags: added: verification-done verification-done-cosmic
removed: verification-needed verification-needed-cosmic
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for libssh has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in libssh (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libssh - 0.8.1-1ubuntu0.2

---------------
libssh (0.8.1-1ubuntu0.2) cosmic-proposed; urgency=medium

  * Fix regressions with known_host parsing: ssh_session_is_known_server()
    sometimes fails with SSH_SERVER_FOUND_OTHER if known_hosts contains
    multiple key types for the target host. (LP: #1799665)
  * Also backport fixes for some related bugs:
    - Use all supported hostkey algorithms for negotiation
    - Honor more host key algorithms than the first one (ssh-ed25519)
    - Use the correct name for ECDSA keys for host key negotiation

 -- Martin Pitt <email address hidden> Wed, 24 Oct 2018 08:20:23 +0000

Changed in libssh (Ubuntu Cosmic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.