live migration over ssh failing due to missing port in ssh_known_hosts

[Oliver Walsh]

It appears the behaviour of ssh client has changed. Previously a port was not required in /etc/ssh/ssh_known_hosts when running on a non-default port, now it is:

[root@compute-1 heat-admin]# ssh compute-0.localdomain
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

[root@compute-1 heat-admin]# ssh -p 2022 compute-0.localdomain
The authenticity of host '[compute-0.localdomain]:2022 ([172.17.1.19]:2022)' can't be established.
ECDSA key fingerprint is SHA256:1kTWyAQTD3K1vGsiQQitfRChSk1drrtewcZ82VhGx1c.
ECDSA key fingerprint is MD5:61:53:4a:e2:69:a6:b9:8b:70:39:8f:a6:a2:20:18:bd.
Are you sure you want to continue connecting (yes/no)? ^C

[Oliver Walsh (owalsh)]

NB live/cold migration are configured to use a non-default port when containers are used

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/597165

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/597165
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=876683f317640aa6877f991f2d4d0b098de3b7b3
Submitter: Zuul
Branch: master

commit 876683f317640aa6877f991f2d4d0b098de3b7b3
Author: Oliver Walsh <email address hidden>
Date: Tue Aug 28 16:34:21 2018 +0100

    Include ssh known_hosts entries for non-default port

    The ssh client no longer appears to accept the regular known hosts entry when
    the target is running on a non-default port.
    Adding '[host]:*' should fix this, regardless of the port.
    However this does not work for the default port so we must include both.

    Change-Id: I519ff6053676870dff1bdff60fb1f6b2aa5ee8c9
    Closes-bug: #1789452

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/598137

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/598137
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c70d197d36d48ce29486e2744a8bc9fda630dd11
Submitter: Zuul
Branch: master

commit c70d197d36d48ce29486e2744a8bc9fda630dd11
Author: Oliver Walsh <email address hidden>
Date: Thu Aug 30 12:23:45 2018 +0100

    Simplify ssh known_hosts entries for non-default port

    '[host]*' matches both default port and non-default port.

    Change-Id: Id83bed36f3ab7f8d0fbdbd03f3960307af62fc84
    Related-bug: #1789452

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/601817

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/601818

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.0.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/601817
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3d5d275c6dfb11ea9410ec9aa926df500cfd22c1
Submitter: Zuul
Branch: stable/rocky

commit 3d5d275c6dfb11ea9410ec9aa926df500cfd22c1
Author: Oliver Walsh <email address hidden>
Date: Tue Aug 28 16:34:21 2018 +0100

    Include ssh known_hosts entries for non-default port

    The ssh client no longer appears to accept the regular known hosts entry when
    the target is running on a non-default port.
    Adding '[host]:*' should fix this, regardless of the port.
    However this does not work for the default port so we must include both.

    Change-Id: I519ff6053676870dff1bdff60fb1f6b2aa5ee8c9
    Closes-bug: #1789452
    (cherry picked from commit 876683f317640aa6877f991f2d4d0b098de3b7b3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/601818
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7a05790d51ebba48cf7e9bf230a4cb87acac7315
Submitter: Zuul
Branch: stable/rocky

commit 7a05790d51ebba48cf7e9bf230a4cb87acac7315
Author: Oliver Walsh <email address hidden>
Date: Thu Aug 30 12:23:45 2018 +0100

    Simplify ssh known_hosts entries for non-default port

    '[host]*' matches both default port and non-default port.

    Change-Id: Id83bed36f3ab7f8d0fbdbd03f3960307af62fc84
    Related-bug: #1789452
    (cherry picked from commit c70d197d36d48ce29486e2744a8bc9fda630dd11)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.2.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/649021

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/649022

[Martin Schuppert (mschuppert)]

also an issue with queens when you use config-download where we deploy the ssh-rsa host keys and not the ecdsa like before.

()[root@compute-0 /]$ ssh -p2022 -vvv -i /etc/nova/migration/identity <email address hidden>
...
debug1: Authenticating to compute-1.internalapi.localdomain:2022 as 'nova_migration'
debug3: put_host_port: [compute-1.internalapi.localdomain]:2022
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
...
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256

-> ecdsa gets selected but we only have rsa keys:

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:94Hm16YJquI0Q+GR0/F3E0PFDg1hrIa4egqmQCaJQUY
debug3: put_host_port: [172.17.1.19]:2022
debug3: put_host_port: [compute-1.internalapi.localdomain]:2022
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug1: checking without port identifier
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:1
debug3: load_hostkeys: loaded 1 keys from compute-1.internalapi.localdomain
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 172.17.1.19
The authenticity of host '[compute-1.internalapi.localdomain]:2022 ([172.17.1.19]:2022)' can't be established.
ECDSA key fingerprint is SHA256:94Hm16YJquI0Q+GR0/F3E0PFDg1hrIa4egqmQCaJQUY.
ECDSA key fingerprint is MD5:c1:d9:44:e2:fe:47:63:e1:fc:7c:1e:7c:5c:87:c7:74.
Are you sure you want to continue connecting (yes/no)?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649021
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=65b285ffad09df242a989eb294d9ce5e9cfd951d
Submitter: Zuul
Branch: stable/queens

commit 65b285ffad09df242a989eb294d9ce5e9cfd951d
Author: Oliver Walsh <email address hidden>
Date: Tue Aug 28 16:34:21 2018 +0100

    Include ssh known_hosts entries for non-default port

    The ssh client no longer appears to accept the regular known hosts entry when
    the target is running on a non-default port.
    Adding '[host]:*' should fix this, regardless of the port.
    However this does not work for the default port so we must include both.

    Change-Id: I519ff6053676870dff1bdff60fb1f6b2aa5ee8c9
    Closes-bug: #1789452
    (cherry picked from commit 876683f317640aa6877f991f2d4d0b098de3b7b3)
    (cherry picked from commit 3d5d275c6dfb11ea9410ec9aa926df500cfd22c1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649022
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ff5a5bf5a0654b7927ea4b759c4f1770c8e9e8c1
Submitter: Zuul
Branch: stable/queens

commit ff5a5bf5a0654b7927ea4b759c4f1770c8e9e8c1
Author: Oliver Walsh <email address hidden>
Date: Thu Aug 30 12:23:45 2018 +0100

    Simplify ssh known_hosts entries for non-default port

    '[host]*' matches both default port and non-default port.

    Change-Id: Id83bed36f3ab7f8d0fbdbd03f3960307af62fc84
    Related-bug: #1789452
    (cherry picked from commit c70d197d36d48ce29486e2744a8bc9fda630dd11)
    (cherry picked from commit 7a05790d51ebba48cf7e9bf230a4cb87acac7315)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.