tripleo

live migration over ssh failing due to invalid ssh_known_hosts

Bug #1781027 reported by Oliver Walsh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Oliver Walsh
tripleo rocky-3

Bug Description

Live migration is failing because the SSH config is broken:

[root@compute-1 heat-admin]# ssh -o BatchMode=yes -v compute-0 echo hello
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to compute-0 [172.17.1.20] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to compute-0:22 as 'root'
debug1: /etc/ssh/ssh_known_hosts:3: parse error in hostkeys file
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: <email address hidden> MAC: <implicit> compression: none
debug1: kex: client->server cipher: <email address hidden> MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:OfoBYdzyWNRYLDUWgtsRaBV+BvJKYhgMlxepaf6O09Q
debug1: /etc/ssh/ssh_known_hosts:3: parse error in hostkeys file
debug1: /etc/ssh/ssh_known_hosts:3: parse error in hostkeys file
Host key verification failed.

Known hosts entries are missing the key type. Works when I inject it:

[root@compute-1 heat-admin]# sed -i -e 's/ / ecdsa-sha2-nistp256 /' /etc/ssh/ssh_known_hosts

[root@compute-1 heat-admin]# ssh -o BatchMode=yes -v compute-0 echo hello
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to compute-0 [172.17.1.20] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to compute-0:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: <email address hidden> MAC: <implicit> compression: none
debug1: kex: client->server cipher: <email address hidden> MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:OfoBYdzyWNRYLDUWgtsRaBV+BvJKYhgMlxepaf6O09Q
debug1: Host 'compute-0' is known and matches the ECDSA host key.
debug1: Found key in /etc/ssh/ssh_known_hosts:3
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)

debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)

debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

(Permission denied is expected, I'm not using the correct private key).

Looks like this was introduced in the changes for https://bugs.launchpad.net/tripleo/+bug/1746336

Oliver Walsh (owalsh)
Changed in tripleo:
status: New → Triaged
importance: Undecided → High
milestone: none → rocky-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/581442

Changed in tripleo:
assignee: nobody → Oliver Walsh (owalsh)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/581443

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Oliver Walsh (<email address hidden>) on branch: master
Review: https://review.openstack.org/581443

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/581442
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=82b4a0cee107b5054114db71d73a4a41e07f5f03
Submitter: Zuul
Branch: master

commit 82b4a0cee107b5054114db71d73a4a41e07f5f03
Author: Oliver Walsh <email address hidden>
Date: Tue Jul 10 19:22:48 2018 +0100

    Fix ssh_known_hosts format

    The template is missing the key type.

    Change-Id: I9cd43e933d6e5963f8da52f1821f11841920ec21
    Closes-bug: #1781027

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/582421

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/queens)

Reviewed: https://review.openstack.org/582421
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=121e6b054a124e32e8f82bb8983afeead26873c8
Submitter: Zuul
Branch: stable/queens

commit 121e6b054a124e32e8f82bb8983afeead26873c8
Author: Oliver Walsh <email address hidden>
Date: Tue Jul 10 19:22:48 2018 +0100

    Fix ssh_known_hosts format

    The template is missing the key type.

    Change-Id: I9cd43e933d6e5963f8da52f1821f11841920ec21
    Closes-bug: #1781027
    (cherry picked from commit 82b4a0cee107b5054114db71d73a4a41e07f5f03)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 9.2.0

This issue was fixed in the openstack/tripleo-common 9.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 8.6.4

This issue was fixed in the openstack/tripleo-common 8.6.4 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.