Activity log for bug #1799665

Date Who What changed Old value New value Message
2018-10-24 09:19:59 Martin Pitt bug added bug
2018-10-24 09:22:18 Martin Pitt nominated for series Ubuntu Cosmic
2018-10-24 09:22:18 Martin Pitt bug task added libssh (Ubuntu Cosmic)
2018-10-24 09:22:29 Martin Pitt libssh (Ubuntu Cosmic): status New In Progress
2018-10-24 09:30:01 Martin Pitt description Ubuntu 18.10's libssh 0.8.1 regresses parsing of known_hosts. This happens (sometimes) if there are multiple known_host key types (e. g. ssh-rsa and ssh-ed25519), then it can happen that ssh_session_is_known_server() fails with SSH_SERVER_FOUND_OTHER [1]. I noticed this with testing Cockpit on Ubuntu 18.10 [2], which has a few test cases exercising cockpit-ssh (which uses libssh). The scenario is a FreeIPA centrally managed known_hosts file with these entries: x0.cockpit.lan ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCv5sLKfLDuEAbTcHC3eOgJM+Ot7F077KewD4e1lGzfw300Jo4xnuPsoJEVSCR7OjsYQCnuVGlqtlavMCLFzIBNk06iTBg/nl+W+xa3CFNITbAjiBif7SeY0XL6Xeqzb1VYXNVfwKQKpcGIbDne6jyou4wRZV1eay03FHTSkd2+XKM6GOUGlkEUoPyAwYPHqoKUYiiyBxJs20l/peXVx6jsGgs2Sc6gl3KJP0TB2E7ncD1pWHGRtiNshFFVarw/YKr+Rs+KhiVS3CAAfYDhpBNWXOwTKyx2euJjAhsRF10bx6pnuadSEpT8Ufo5/YFIVAD1GHptULSzVjUoJm6ktoHB x0.cockpit.lan ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkJ6CaqhzUhrbpbVmZ8BmZZgM3u6BukZ6HFB2a4NLQBdgpHlHbxoJ47ocTImctyFMiDi0y6vCb4tFuZgp6Krmk= root@(none) x0.cockpit.lan ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINK6gcOyH4OhiKPcNr33Kl6e+wFAUy9tGFBU/o4yWkxh root@(none) Connecting to that host with the standard ssh client works: $ ssh -vv x0.cockpit.lan [...] debug1: Server host key: ecdsa-sha2-nistp256 SHA256:MgfkN6HEl+pdz0X7+6q08IVkUZOtEDzfA6V18Wm9DgA debug1: Host 'x0.cockpit.lan' is known and matches the ECDSA host key. debug1: Found key in /var/lib/sss/pubconf/known_hosts:3 [...] admin@cockpit.lan@x0:~$ But not with cockpit-ssh. This shows the JSON protocol (note that you need to copy&paste the correct cookie value from the response): $ G_MESSAGES_DEBUG=cockpit-ssh cockpit-bridge --interact=--- { "command": "open", "channel": "c", "payload": "echo", "host": "x0.cockpit.lan", "user": "admin@cockpit.lan" } --- {"command":"authorize","challenge":"*","cookie":"session107271540364829"} --- {"command":"authorize", "response": "password foobarfoo", "cookie": "session107271540364829"} --- (cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.049: cockpit-ssh x0.cockpit.lan: host not known in any local file, asking sssd (cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.472: cockpit-ssh x0.cockpit.lan: using known hosts file /tmp/known-hosts.IDKNRZ (cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.542: cockpit-ssh x0.cockpit.lan: connected cockpit-ssh-Message: 03:07:30.828: cockpit-ssh x0.cockpit.lan: host key for this server changed key type: ssh-ed25519 {"command":"close","host-key":"x0.cockpit.lan ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINK6gcOyH4OhiKPcNr33Kl6e+wFAUy9tGFBU/o4yWkxh\n","host-fingerprint":"a0:27:1e:80:de:fd:4b:8a:0d:9d:a9:b6:42:7d:5c:b9","problem":"invalid-hostkey","error":"invalid-hostkey","auth-method-results":{},"channel":"c"} --- The "host key for this server changed key type" is the effect of this bug. [1] http://api.libssh.org/master/group__libssh__session.html#gac%20bc5d04fe66beee863a0c61a93fdf765 [2] https://github.com/cockpit-project/cockpit/pull/10357 Ubuntu 18.10's libssh 0.8.1 regresses parsing of known_hosts. This happens (sometimes) if there are multiple known_host key types (e. g. ssh-rsa and ssh-ed25519), then it can happen that ssh_session_is_known_server() fails with SSH_SERVER_FOUND_OTHER [1]. I noticed this with testing Cockpit on Ubuntu 18.10 [2], which has a few test cases exercising cockpit-ssh (which uses libssh), e. g. [3]. The scenario is a FreeIPA centrally managed known_hosts file with these entries: x0.cockpit.lan ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCv5sLKfLDuEAbTcHC3eOgJM+Ot7F077KewD4e1lGzfw300Jo4xnuPsoJEVSCR7OjsYQCnuVGlqtlavMCLFzIBNk06iTBg/nl+W+xa3CFNITbAjiBif7SeY0XL6Xeqzb1VYXNVfwKQKpcGIbDne6jyou4wRZV1eay03FHTSkd2+XKM6GOUGlkEUoPyAwYPHqoKUYiiyBxJs20l/peXVx6jsGgs2Sc6gl3KJP0TB2E7ncD1pWHGRtiNshFFVarw/YKr+Rs+KhiVS3CAAfYDhpBNWXOwTKyx2euJjAhsRF10bx6pnuadSEpT8Ufo5/YFIVAD1GHptULSzVjUoJm6ktoHB x0.cockpit.lan ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkJ6CaqhzUhrbpbVmZ8BmZZgM3u6BukZ6HFB2a4NLQBdgpHlHbxoJ47ocTImctyFMiDi0y6vCb4tFuZgp6Krmk= root@(none) x0.cockpit.lan ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINK6gcOyH4OhiKPcNr33Kl6e+wFAUy9tGFBU/o4yWkxh root@(none) Connecting to that host with the standard ssh client works: $ ssh -vv x0.cockpit.lan [...] debug1: Server host key: ecdsa-sha2-nistp256 SHA256:MgfkN6HEl+pdz0X7+6q08IVkUZOtEDzfA6V18Wm9DgA debug1: Host 'x0.cockpit.lan' is known and matches the ECDSA host key. debug1: Found key in /var/lib/sss/pubconf/known_hosts:3 [...] admin@cockpit.lan@x0:~$ But not with cockpit-ssh. This shows the JSON protocol (note that you need to copy&paste the correct cookie value from the response): $ G_MESSAGES_DEBUG=cockpit-ssh cockpit-bridge --interact=--- { "command": "open", "channel": "c", "payload": "echo", "host": "x0.cockpit.lan", "user": "admin@cockpit.lan" } --- {"command":"authorize","challenge":"*","cookie":"session107271540364829"} --- {"command":"authorize", "response": "password foobarfoo", "cookie": "session107271540364829"} --- (cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.049: cockpit-ssh x0.cockpit.lan: host not known in any local file, asking sssd (cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.472: cockpit-ssh x0.cockpit.lan: using known hosts file /tmp/known-hosts.IDKNRZ (cockpit-ssh:10814): cockpit-ssh-DEBUG: 03:11:51.542: cockpit-ssh x0.cockpit.lan: connected cockpit-ssh-Message: 03:07:30.828: cockpit-ssh x0.cockpit.lan: host key for this server changed key type: ssh-ed25519 {"command":"close","host-key":"x0.cockpit.lan ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINK6gcOyH4OhiKPcNr33Kl6e+wFAUy9tGFBU/o4yWkxh\n","host-fingerprint":"a0:27:1e:80:de:fd:4b:8a:0d:9d:a9:b6:42:7d:5c:b9","problem":"invalid-hostkey","error":"invalid-hostkey","auth-method-results":{},"channel":"c"} --- The "host key for this server changed key type" is the effect of this bug. SRU INFORMATION: [IMPACT]: libssh connections that worked in previous Ubuntu releases now may fail on host key verification [TEST CASE]: See reproducer below. This isn't too easy to reproduce for someone else, so I'm happy to do the validation myself. This can also be verified with the Cockpit integration tests: bots/image-prepare ubuntu-stable TEST_OS=ubuntu-stable test/verify/check-realms TestRealms.testIpa [REGRESSION POTENTIAL]: In principle these patches could break known_hosts validation further. However, these fixes have been in Debian testing for a while and validated through e. g. Cockpit's tests (which exercise cockpit-ssh quite heavily). There are also upstream unit tests, and while they didn't pick up that particular regression, they at least make sure that known_hosts verification still works for common cases. Also, libssh-4 does not have that many reverse dependencies. So overall, I think this is bearable for an SRU, especially as the impact is quite high. [1] http://api.libssh.org/master/group__libssh__session.html#gac%20bc5d04fe66beee863a0c61a93fdf765 [2] https://github.com/cockpit-project/cockpit/pull/10357 [3] https://fedorapeople.org/groups/cockpit/logs/pull-10357-20181022-204242-8672df31-verify-ubuntu-stable/log.html#186
2018-11-03 00:20:18 Steve Langasek libssh (Ubuntu Cosmic): status In Progress Fix Committed
2018-11-03 00:20:22 Steve Langasek bug added subscriber Ubuntu Stable Release Updates Team
2018-11-03 00:20:25 Steve Langasek bug added subscriber SRU Verification
2018-11-03 00:20:27 Steve Langasek tags cosmic regression-release cosmic regression-release verification-needed verification-needed-cosmic
2018-11-08 21:16:58 Martin Pitt tags cosmic regression-release verification-needed verification-needed-cosmic cosmic regression-release verification-done verification-done-cosmic
2018-11-12 18:27:22 Brian Murray libssh (Ubuntu): status In Progress Fix Released
2018-11-12 18:27:40 Brian Murray removed subscriber Ubuntu Stable Release Updates Team
2018-11-12 18:28:13 Launchpad Janitor libssh (Ubuntu Cosmic): status Fix Committed Fix Released