The testcase 'test_connectivity_between_vms_on_different_networks' was broken that got AuthenticationException

[Eric Xie]

With paramiko==2.9.2, the testcase was broken.

tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks[compute,id-1546850e-fbaa-42f5-8b5f-03d8a6a95f15,network,slow]
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/usr/local/lib/python3.8/dist-packages/tempest/lib/common/ssh.py", line 107, in _get_ssh_connection
    ssh.connect(self.host, port=self.port, username=self.username,

      File "/usr/local/lib/python3.8/dist-packages/paramiko/client.py", line 435, in connect
    self._auth(

      File "/usr/local/lib/python3.8/dist-packages/paramiko/client.py", line 766, in _auth
    raise saved_exception

      File "/usr/local/lib/python3.8/dist-packages/paramiko/client.py", line 666, in _auth
    self._transport.auth_publickey(username, pkey)

      File "/usr/local/lib/python3.8/dist-packages/paramiko/transport.py", line 1634, in auth_publickey
    return self.auth_handler.wait_for_response(my_event)

      File "/usr/local/lib/python3.8/dist-packages/paramiko/auth_handler.py", line 258, in wait_for_response
    raise e

    paramiko.ssh_exception.AuthenticationException: Authentication failed.

root@openstack-tempest-run-tests-b8b4r:/# pip3 list | grep paramiko
paramiko 2.9.2

[Lukáš Piwowarski (piwowarl)]

I can confirm this bug. It seems that the bug was introduced with paramiko==2.9.1.

I have tried to run the test both with paramiko==2.9.1 and paramiko==2.8.1. With paramiko==2.8.1 the test finished successfully but with paramiko==2.9.1. the test failed with the same error mentioned in the bug description.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/834646

[Martin Kopec (mkopec)]

Great findings Lukáš! .. what about newer paramiko versions? Is it reproducible only with 2.9.1?
https://pypi.org/project/paramiko/#history

[Martin Kopec (mkopec)]

Maybe this is related?
https://github.com/paramiko/paramiko/issues/1955

[Martin Kopec (mkopec)]

the issue I linked above points to this one https://github.com/paramiko/paramiko/issues/1961

those 2 must be connected to our issue somehow

[Lukáš Piwowarski (piwowarl)]

Yes, it fails also with the newest version of paramiko (2.10.3).

Here is patch (https://review.opendev.org/c/openstack/tempest/+/834686) that uses fix from the link you provided (https://github.com/paramiko/paramiko/issues/1961).

[Lukáš Piwowarski (piwowarl)]

The test fails because paramiko expects to receive server-sig-algs [1] from the server. When it does not receive the server-sig-algs it defaults to rsa-sha2 [2].

Cirros does not send the server-sig-algs [3] => paramiko tries to use rsa-sha2 => paramiko fails to authenticate with the cirros server because it does not support rsa-sha2 (Dropbear<2020.79 is used by cirros [4]) => and because of that the test fails.

Possible solutions:
-------------------
1) Disable rsa-sha2-256 and rsa-sha2-512 for paramiko and force it this way to use rsa-sha.
2) Fix cirros [5] so it sends server-sig-algs during ssh authentication or supports rsa-sha2 (update Dropbear).

[1] https://datatracker.ietf.org/doc/html/rfc8308#section-3.1
[2] https://www.paramiko.org/changelog.html#2.9.0 (see Warning)
[3] (ssh -vvv cirros@[cirros-hostname] | grep "server-sig-list") => nothing
[4] Release notes of Dropbear: https://mirror.dropbear.nl/mirror/CHANGES (look for rsa-sha2)
[5] https://github.com/cirros-dev/cirros/issues/74

[Lukáš Piwowarski (piwowarl)]

rsa-sha2 maybe will be supported in the next release of cirros:
https://github.com/cirros-dev/cirros/issues/77

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/838753

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Lukas Piwowarski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/834646
Reason: No longer necessary. Paramiko issue fixed here: https://review.opendev.org/c/openstack/tempest/+/838753

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/838753
Committed: https://opendev.org/openstack/tempest/commit/75ca0b87c6957aa21d4945cc698a2a39e544185d
Submitter: "Zuul (22348)"
Branch: master

commit 75ca0b87c6957aa21d4945cc698a2a39e544185d
Author: Martin Kopec <email address hidden>
Date: Wed Apr 20 17:57:45 2022 +0200

    Switch to ecdsa ssh key type by default

    As the version of cirros used in OpenStack CI does not support SHA-2
    signatures for ssh, any connection from a FIPS enabled machine will fail
    in case validation.ssh_key_type is set to rsa (the default until now).
    Using ecdsa keys helps us avoid the mentioned issue.

    From now on, the validation.ssh_key_type option will be set to ecdsa
    by default for testing simplicity.

    This change shouldn't have any drastic effect on any tempest consumer,
    in case rsa ssh type is required in a consumer's scenario,
    validation.ssh_key_type can be overridden to rsa.

    Relevant-Bug: #1960692
    Change-Id: If9becae119e2a5dc51d4911a0ac9759fbcf24998

[Martin Kopec (mkopec)]

This is a bug in cirros image - it should be fixed soon, see the cirros bug tracker above. We have addressed this in Tempest by switching to ecdsa keys by default, see https://review.opendev.org/c/openstack/tempest/+/838753

This is considered fixed from Tempest point of view. Feel free to reopen if you feel otherwise.