All dynamic workloads that depends on ssh from undercloud should use default configured key

[jaison]

By default workloads that spawn instances in dynamic_workloads use temporarily created rally keypair.
Although some of the workloads use ssh to the instances.
Such operations will always fail for centos vm as it requires ssh public/private key based authentication.

2021-12-21 12:31:04.342 347891 ERROR paramiko.transport [-] Exception: Error reading SSH protocol banner: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.342 347891 ERROR paramiko.transport [-] Traceback (most recent call last):: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.342 347891 ERROR paramiko.transport [-] File "/home/stack/browbeat/.rally-venv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.342 347891 ERROR paramiko.transport [-] buf = self.packetizer.readline(timeout): paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.342 347891 ERROR paramiko.transport [-] File "/home/stack/browbeat/.rally-venv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.342 347891 ERROR paramiko.transport [-] buf += self._read_timeout(timeout): paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.342 347891 ERROR paramiko.transport [-] File "/home/stack/browbeat/.rally-venv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] raise EOFError(): paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] EOFError: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] : paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] During handling of the above exception, another exception occurred:: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] : paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] Traceback (most recent call last):: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] File "/home/stack/browbeat/.rally-venv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] self._check_banner(): paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] File "/home/stack/browbeat/.rally-venv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner: paramiko.ssh_exception.SSHException: Error reading SSH p
rotocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] "Error reading SSH protocol banner" + str(e): paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.343 347891 ERROR paramiko.transport [-] : paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2021-12-21 12:31:04.344 347891 RALLYDEBUG rally.utils.sshutils [-] Ssh is still unavailable: SSHError("Exception <class 'paramiko.ssh_exception.SSHException'> was raised during connect to centos@172.31.12.191:22. Exception value is: SSHException('Error reading SSH protocol banner',)",)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/x/browbeat/+/823474

[Sanjay Chari (sanjaychari)]

After doing some debugging, this bug appears in trunk dynamic workloads when ssh to jumphost is required. This bug seems to appear because of the behaviour of the paramiko module when installed inside a venv. I tried calling client.connect(<host_ip>, username=<username>, pkey=<private key>) both inside rally venv and outside rally venv. Inside rally venv, ssh fails multiple times with message
"Ssh is still unavailable: SSHError("Exception <class 'paramiko.ssh_exception.Au
thenticationException'> was raised during connect to cirros@172.31.9.177:22. Exception value is: AuthenticationException('Authentication failed.',
)",)",
and finally the cirros host stops accepting ssh connections, causing the error in the original issue.
When the same function call is made after deactivating rally venv, it works on the first attempt.
A stackoverflow[1] answer explains this issue.

[1] https://stackoverflow.com/a/70794909

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Sanjay Chari <email address hidden>" on branch: master
Review: https://review.opendev.org/c/x/browbeat/+/823474
Reason: The patch to pin paramiko to 2.7.2 was merged, and that fixes this issue, so I am abandoning this patch.