tripleo-ci-centos-8-containers-multinode randomly failing tempest tests with paramiko.ssh_exception.BadHostKeyException

[Pooja Jadhav]

wallaby : tripleo-ci-centos-8-containers-multinode-wallaby randomly failing tempest test with below traceback :

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 113, in _get_ssh_connection
        sock=proxy_chan)
      File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 423, in connect
        raise BadHostKeyException(hostname, server_key, our_key)
    paramiko.ssh_exception.BadHostKeyException: Host key for server '192.168.24.102' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZfUVUdvppIyu5pzZtZNK86vKeJ2rA0SiBDNuIRp4DJNv+V1FvWMUMCd9roJ+ZZKLrSaOzm9JR7brZYa6iqRYY=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIKsMDVTFcuwVHAtDCpLceTDjDwUEoUrpGPJclmzmYngi'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
        return f(*func_args, **func_kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_basic_ops.py", line 619, in test_subnet_details
        self._check_public_network_connectivity(should_connect=True)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_basic_ops.py", line 214, in _check_public_network_connectivity
        message, server, mtu=mtu)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/manager.py", line 956, in check_vm_connectivity
        server=server)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/manager.py", line 726, in get_remote_client
        linux_client.validate_authentication()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 59, in wrapper
        six.reraise(*original_exception)
      File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
        raise value
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 32, in wrapper
        return function(self, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 115, in validate_authentication
        self.ssh_client.test_connection_auth()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 217, in test_connection_auth
        connection = self._get_ssh_connection()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 129, in _get_ssh_connection
        password=self.password)
    tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.102 via SSH timed out.
    User: cirros, Password: None

References :

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3e/786619/7/gate/tripleo-ci-centos-8-containers-multinode-wallaby/f3e7f65/logs/undercloud/var/log/tempest/tempest_run.log

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e03/777106/30/check/tripleo-ci-centos-8-containers-multinode-wallaby/e03ba03/logs/undercloud/var/log/tempest/tempest_run.log

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3e/786619/7/gate/tripleo-ci-centos-8-containers-multinode-wallaby/f3e7f65/logs/undercloud/var/log/tempest/tempest_run.log

Randomly tests are failing with the same issue reported.

[Pooja Jadhav (poojajadhav22)]

same issue is hitting on master branch patch : https://review.opendev.org/c/openstack/tripleo-heat-templates/+/790495/

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_305/790495/2/gate/tripleo-ci-centos-8-containers-multinode/30581b1/logs/undercloud/var/log/tempest/stestr_results.html ..

[Sandeep Yadav (sandeepyadav93)]

This is still affecting check jobs randomly:-

https://7c2c97e2a5a64c927d32-11c736403b210e03aa568c45890a1263.ssl.cf2.rackcdn.com/793145/2/check/tripleo-ci-centos-8-containers-multinode/ac6402d/logs/undercloud/var/log/tempest/stestr_results.html

~~~
paramiko.ssh_exception.BadHostKeyException: Host key for server '192.168.24.102' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMDTkRI81rgO4dscUnS6QCHo/cr075Fz/rH1rn1u//za61TH5xpIi6Ra1sY2WMR29gR1me722nxi1geu+Li4+bc=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIPgp6MuT68LtjbmRqt+Yegruz1s5OXLt7ufzlTXxpm51'
~~~

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/794164

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/794164
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/52e5da0b8f2dbf812a21011cf619bc8df2413ef9
Submitter: "Zuul (22348)"
Branch: master

commit 52e5da0b8f2dbf812a21011cf619bc8df2413ef9
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jun 2 10:41:18 2021 +0200

    Update public subnet allocation pool

    Previously allocation pool was starting with tempest_cidr.100
    and ended with tempest_cidr.150 IP addresses.
    In some jobs it overlapped with the IPs used for
    RedisVirtualFixedIPs (192.168.24.101) and
    OVNDBsVirtualFixedIPs (192.168.24.102)

    In such case, when e.g. 192.168.24.102 was used in tempest test as FIP,
    test was failing due to paramiko.ssh_exception.BadHostKeyException
    raised as on the ~/.ssh/known_hosts in the undercloud there was already
    key for host with IP 192.168.24.102.

    Those RedisVirtualFixedIPs and OVNDBsVirtualFixedIPs were added in
    commit [1].
    This patch changes allocation pool used in tempest to be
    tempest_cidr.150 - tempest_cidr.200 to avoid that overlap.

    [1] https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/777106

    Closes-Bug: #1928933
    Change-Id: Ia515a1523a38a2d634d9520a9d9453218b9c38e1