SSH connection to a newly upgraded subcloud failed due to Host key verification

[Yuxing]

Brief Description
-----------------
The host key of the subcloud was changed after upgrade, ~/.ssh/known_hosts on System Controller contains the old host key fingerprint. This results the ssh attempt failure from the system controller.

Severity
--------
Minor

Steps to Reproduce
------------------
1. Upgrade the system controller
2. Deploy a subcloud
3. Upgrade a single subcloud using dcmanager upgrade orchestrator
4. SSH to the subcloud

Expected Behavior
------------------
SSH to the subcloud and accept the fingerprint.

Actual Behavior
----------------
SSH Host key verification failed.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Distributed cloud

Branch/Pull Time/Commit
-----------------------
2021-05-24_20-00-07

Last Pass
---------
an

Timestamp/Logs
--------------
na

Test Activity
-------------
Developer Testing

Workaround
----------
Delete the key for the host in ~/.ssh/known_hosts

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/798909

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/798918

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/798918
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/cfe9421fc4001e9a8c54d5e47ce2cd97d3e4f428
Submitter: "Zuul (22348)"
Branch: master

commit cfe9421fc4001e9a8c54d5e47ce2cd97d3e4f428
Author: Yuxing Jiang <email address hidden>
Date: Wed Jun 30 09:58:24 2021 -0500

    Restore ssh data from the backup data

    This commit restores the ssh configurations and keys in /etc/ssh from
    the backup tarball. After this commit, host keys from the backed-up
    host will be restored to the newly installed host, so the other hosts
    can ssh to the new host as a known host with the existing keys stored
    in there ~/.ssh/known_hosts.

    This commit is based on:
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/798909.
    With that commit, the ansible ssh connection will not be rejected due
    to the remote host key change.

    Test steps:

    Test for backup/restore:
    1. Build a fresh ISO with these two commits.
    2. Install a DC system with two system controllers and one aiosx
    subcloud controller.
    3. SSH this subcloud as sysadmin and su with its oam address, mgmt
    address and subcloud name to add its key to ~/.ssh/known_hosts. Repeat
    this step(except the subcloud name) from an external host for the same
    purpose.
    4. Backup the subcloud data and reinstall the subcloud afterwards.
    5. Restore the subcloud successfully, and unlock the subcloud
    afterwards.
    6. After the restore, ssh to the subcloud with its oam address, mgmt
    address and subcloud name will be accepted from the system controllers
    or the external host.

    Test for upgrade orchestration:
    1. Install a N load in DC system
    2. Ssh to a subcloud with its name from a central cloud as the
    sysadmin user.
    3. Build a N+1 ISO with these two commits
    4. Upgrade the system controllers as well as the subcloud
    5. After the upgrading the subcloud, the sysadmin and still ssh to the
    subcloud with its name without deleting the previous host key in
    /home/sysadmin/.ssh/known_hosts

    Depends-On:
    https: //review.opendev.org/c/starlingx/ansible-playbooks/+/798909
    Closes-Bug: 1934154
    Signed-off-by: Yuxing Jiang <email address hidden>
    Change-Id: I3dd2820645fadae0903bc8784a0841ac5cca8a30

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/798909
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/7374b81251a3b5b60e38238e4c679fa5215ffd0c
Submitter: "Zuul (22348)"
Branch: master

commit 7374b81251a3b5b60e38238e4c679fa5215ffd0c
Author: Yuxing Jiang <email address hidden>
Date: Wed Jun 30 09:32:52 2021 -0500

    Check and add remote host key into known_hosts

    The previous method of updating SSH known hosts removes the existing
    host key of the ansible remote host. This commit changes this method
    to scan and add the new ssh key of the ansible remote host into
    ~/.ssh/known_hosts. After this change, the ssh connection to the
    ansible remote host will be accepted if the ansible remote host has
    either the former key or the latter key.

    Test see in:
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/798918

    Partial-Bug: 1934154
    Signed-off-by: Yuxing Jiang <email address hidden>
    Change-Id: I7c56e130ad8b75f5c8ffead0b2041a5dedac6343