Prestage playbook failed due to subcloud access denied when host key changes

[Li Zhu]

Brief Description
-----------------
Prestage playbook failed due to subcloud access denied when host key changes.

Severity
--------
Major

Steps to Reproduce
--------
1 Upgrade system controller
2 Upgrade subcloud
3 prestage subcloud for reinstallation

working scenario:
1 upgrade system controller
2 prestage subcloud for upgrade
3 upgrade subcloud
4 prestage subcloud for reinstall

Expected Behavior
------------------
The prestage for reinstall should work without any previous step.

Actual Behavior
----------------
The prestage for reinstall fails to access the subcloud due to host key changes as the the subcloud was upgraded.

Reproducibility
---------------
100% reproducible.

System Configuration
--------------------
DC

Branch/Pull Time/Commit
-----------------------
21.12 + patch001

Last Pass
---------
new test scenario.

Timestamp/Logs
--------------
TASK [prestage/prepare-env : Retrieve current software version of the host] ****
Tuesday 22 March 2022 13:29:26 +0000 (0:00:00.080) 0:00:00.252 *********
fatal: [subcloud1]: UNREACHABLE! => changed=false
  msg: |-
    Failed to connect to the host via ssh: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that a host key has just been changed.
    The fingerprint for the ECDSA key sent by the remote host is
    SHA256:qTAPchM9ZHsVDhoDiuS4ImGnkqNOzN6LcFA2wAR3AM4.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending ECDSA key in /root/.ssh/known_hosts:13
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Release 21.12
    ------------------------------------------------------------------------
    W A R N I N G *** W A R N I N G *** W A R N I N G *** W A R N I N G ***
    ------------------------------------------------------------------------
    THIS IS A PRIVATE COMPUTER SYSTEM.
    This computer system including all related equipment, network devices
    (specifically including Internet access), are provided only for authorized use.
    All computer systems may be monitored for all lawful purposes, including to
    ensure that their use is authorized, for management of the system, to
    facilitate protection against unauthorized access, and to verify security
    procedures, survivability and operational security. Monitoring includes active
    attacks by authorized personnel and their entities to test or verify the
    security of the system. During monitoring, information may be examined,
    recorded, copied and used for authorized purposes. All information including
    personal information, placed on or sent over this system may be monitored. Uses
    of this system, authorized or unauthorized, constitutes consent to monitoring
    of this system. Unauthorized use may subject you to criminal prosecution.
    Evidence of any such unauthorized use collected during monitoring may be used
    for administrative, criminal or other adverse action. Use of this system
    constitutes consent to monitoring for these purposes.

    Permission denied (publickey,password).
  unreachable: true

PLAY RECAP *********************************************************************
subcloud1 : ok=2 changed=0 unreachable=1 failed=0

Tuesday 22 March 2022 13:32:23 +0000 (0:02:56.607) 0:02:56.860 *********
===============================================================================
prestage/prepare-env : Retrieve current software version of the host – 176.61s
prestage/prepare-env : Set prestage dir and subdirectory facts ---------- 0.08s
prestage/prepare-env : Set prestage software version and cleanup directores fact — 0.08s
prestage/prepare-env : Fail if software version for images prestaging is not specified — 0.02s

Test Activity
-------------
Feature Testing

Workaround
----------
Clean up the host key

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/distcloud/+/836400

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/836400
Committed: https://opendev.org/starlingx/distcloud/commit/d9af1bf0ec17955d9a76800448189b37443fd6b6
Submitter: "Zuul (22348)"
Branch: master

commit d9af1bf0ec17955d9a76800448189b37443fd6b6
Author: Li Zhu <email address hidden>
Date: Mon Apr 4 15:42:02 2022 -0400

    Fix prestage playbook access to subcloud denied when host key changes

    Steps to Reproduce:
    1 Upgrade system controller
    2 Upgrade subcloud
    3 prestage subcloud for reinstallation

    The issue came from “Upgrade Subcloud” rather than prestage. At the
    end of upgrade, it restored the old ssh configurations including
    public keys on the subclouds which cause the ssh authentication
    failed in the next step.

    This is a generic fix for all of the playbooks (eg. reinstall,
    upgrade, prestage, etc.). This UserKnownHostsFile=/dev/null ssh
    option will make sure not save the host key as trusted for added
    security no matter the host key changed or not. It combined with
    the existing StrictHostKeyChecking=no option will disable
    the public key checking for ssh connections.

    Test Plan:

    PASS:
    - Verify installation without failures
    - Verify upgrade subclouds without any access failures
    - Verify prestage without subclouds access failures
    - Verify no subclouds access failures according to the given
      100% reproducible steps.

    Closes-Bug: 1967798
    Signed-off-by: Li Zhu <email address hidden>
    Change-Id: I100a65845875d93666f7e6b60b92f8e9add653e4