Multiple neutron tempest tests failure with Authentication failed error

[Ananya Banerjee]

Tempest tests in periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master and periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master are failing with [1]:

```
{0} neutron_tempest_plugin.scenario.test_dhcp.DHCPTest.test_extra_dhcp_opts [411.917754s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 131, in _get_ssh_connection
    ssh.connect(self.host, port=self.port, username=self.username,

      File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 435, in connect
    self._auth(

      File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 764, in _auth
    raise saved_exception

      File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 664, in _auth
    self._transport.auth_publickey(username, pkey)

      File "/usr/lib/python3.9/site-packages/paramiko/transport.py", line 1580, in auth_publickey
    return self.auth_handler.wait_for_response(my_event)

      File "/usr/lib/python3.9/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response
    raise e

    paramiko.ssh_exception.AuthenticationException: Authentication failed.

During handling of the above exception, another exception occurred:

    Traceback (most recent call last):

      File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
    return func(*func_args, **func_kwargs)

      File "/usr/lib/python3.9/site-packages/neutron_tempest_plugin/scenario/test_dhcp.py", line 89, in test_extra_dhcp_opts
    vm_resolv_conf = ssh_client.exec_command(

      File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 333, in wrapped_f
    return self(f, *args, **kw)

      File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
    do = self.iter(retry_state=retry_state)

      File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter
    return fut.result()

      File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()

      File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception

      File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)

      File "/usr/lib/python3.9/site-packages/neutron_tempest_plugin/common/ssh.py", line 171, in exec_command
    return super(Client, self).exec_command(cmd=cmd, encoding=encoding)

      File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 182, in exec_command
    ssh = self._get_ssh_connection()

      File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 150, in _get_ssh_connection
    raise exceptions.SSHTimeout(host=self.host,

    tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.152 via SSH timed out.

```

While taking a look at error.txt file [2]
```

2022-03-08 03:13:10.758 ERROR /var/log/containers/cinder/cinder-api.log: 6 ERROR oslo.messaging._drivers.impl_rabbit [-] [5c38f35a-de38-4bc9-9eb8-453526059529] AMQP server on standalone.ctlplane.localdomain:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>

2022-03-08 03:13:09.387 ERROR /var/log/containers/neutron/server.log: 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [97a86f43-64b6-4459-9bc6-bbe8a095717b] AMQP server on standalone.ctlplane.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer

2022-03-08 03:13:09.175 ERROR /var/log/containers/nova/nova-conductor.log: 2 ERROR oslo.messaging._drivers.impl_rabbit [-] [e4479979-4796-4dcc-90fb-9a5495bb377d] AMQP server on standalone.ctlplane.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer
```
It might be linked with the above error.

```
AMQP server on standalone.ctlplane.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer
```

I also compared with the passing log file [3]. There is no such error.

Logs:

[1]. https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master/af5c09c/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

[2]. https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master/af5c09c/logs/undercloud/var/log/extra/errors.txt.gz

[3]. https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master/4c6e8f5/logs/undercloud/var/log/extra/errors.txt.gz

[chandan kumar (chkumar246)]

BY taking a look at F020 error logs on compute nodes:
```
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db File "/usr/lib/python3.9/site-packages/nova/servicegroup/drivers/db.py", line 92, in _report_state
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db service.service_ref.save()
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db File "/usr/lib/python3.9/site-packages/oslo_versionedobjects/base.py", line 209, in wrapper
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db updates, result = self.indirection_api.object_action(
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db File "/usr/lib/python3.9/site-packages/nova/conductor/rpcapi.py", line 247, in object_action
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db return cctxt.call(context, 'object_action', objinst=objinst,
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db result = self.transport._send(
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db return self._driver.send(target, ctxt, message,
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db return self._send(target, ctxt, message, wait_for_reply, timeout,
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db raise result
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db oslo_messaging.rpc.client.RemoteError: Remote error: DBConnectionError (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
2022-03-08 11:05:19.332 2 ERROR nova.servicegroup.drivers.db (Background on this error at: https://sqlalche.me/e/14/e3q8)

```

Logs: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/3b6a109/logs/overcloud-novacompute-1/var/log/containers/nova/nova-compute.log.txt.gz

and the same tempest tests are failing here also with above reason: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/3b6a109/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz

[chandan kumar (chkumar246)]

Below is the list of all failing tempest tests:
```
neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin.test_two_vms_fips[id-6bba729b-3fb6-494b-9e1e-82bbd89a1045]
neutron_tempest_plugin.scenario.test_basic.NetworkBasicTest.test_basic_instance[id-de07fe0a-e955-449e-b48b-8641c14cd52e]
tearDownClass (neutron_tempest_plugin.scenario.test_basic.NetworkBasicTest)
tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern[compute,id-10fd234a-515c-41e5-b092-8323060598c5,image,network,slow,volume]
neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_router_east_west_traffic[id-b72c3b77-3396-4144-b05d-9cd3c0099893]
neutron_tempest_plugin.scenario.test_dhcp.DHCPTest.test_extra_dhcp_opts[id-58f7c094-1980-4e03-b0d3-6c4dd27217b1]
neutron_tempest_plugin.scenario.test_port_forwardings.PortForwardingTestJSON.test_port_forwarding_editing_and_deleting_tcp_rule[id-aa19d46c-a4a6-11ea-bb37-0242ac130002]
neutron_tempest_plugin.scenario.test_floatingip.FloatingIpMultipleRoutersTest.test_reuse_ip_address_with_other_fip_on_other_router[id-b0382ab3-3c86-4415-84e3-649a8b040dab]
neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers[id-8944b90d-1766-4669-bd8a-672b5d106bb7]
neutron_tempest_plugin.scenario.test_port_forwardings.PortForwardingTestJSON.test_port_forwarding_editing_and_deleting_udp_rule[id-6d05b1b2-6109-4c30-b402-1503f4634acb]
neutron_tempest_plugin.scenario.test_floatingip.DefaultSnatToExternal.test_snat_external_ip[id-3d73ea1a-27c6-45a9-b0f8-04a283d9d764]
neutron_tempest_plugin.scenario.test_port_forwardings.PortForwardingTestJSON.test_port_forwarding_to_2_fixed_ips[id-5971881d-06a0-459e-b636-ce5d1929e2d4]
neutron_tempest_plugin.scenario.test_ports.PortsTest.test_port_with_fixed_ip[id-62e32802-1d21-11eb-b322-74e5f9e2a801]
neutron_tempest_plugin.scenario.test_ports.PortsTest.test_previously_used_port[id-5500797e-b8c2-4e07-a5e0-89fa4e814965]
neutron_tempest_plugin.scenario.test_port_forwardings.PortForwardingTestJSON.test_port_forwarding_to_2_servers[id-ab40fc48-ca8d-41a0-b2a3-f6679c847bfe]
neutron_tempest_plugin.scenario.test_portsecurity.PortSecurityTest.test_port_security_removed_added[id-61ab176e-d48b-42b7-b38a-1ba571ecc033]
neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest.test_default_sec_grp_scenarios[id-3d73ec1a-2ec6-45a9-b0f8-04a283d9d764]
neutron_tempest_plugin.scenario.test_floatingip.TestFloatingIPUpdate.test_floating_ip_update[id-1bdd849b-03dd-4b8f-994f-457cf8a36f93]
neutron_tempest_plugin.scenario.test_internal_dns.InternalDNSTest.test_create_and_update_port_with_dns_name[id-db5e612f-f17f-4974-b5f1-9fe89f4a6fc9]
neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest.test_established_tcp_session_after_re_attachinging_sg[id-01f0ddca-b049-47eb-befd-82acb502c9ec]
neutron_tempest_plugin.scenario.test_qos.QoSTest.test_attach_previously_used_port_to_new_instance[id-66e5673e-0522-11ea-8d71-362b9e155667]
neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest.test_intra_sg_isolation[id-f07d0159-8f9e-4faa-87f5-a869ab0ad490]
neutron_tempest_plugin.scenario.test_securi...

[Marios Andreou (marios-b)]

I am digging at the wallaby line and have filed a duplicate (I am marking it so now) there FYI https://bugs.launchpad.net/tripleo/+bug/1964131 hitting multiple jobs

[yatin (yatinkarel)]

openssl new version in CentOS 9-stream dropped support for 'rsa' keys, so need to use the supported keys.

New keys are supported in Tempest via config option and tempest tests are running fine as config already set[1], currently neutron-tempest-plugin tests are impacted and needs a fix there, pushed https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/832640, testing it in https://review.rdoproject.org/r/c/testproject/+/40212.

[1] https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/831362

[chandan kumar (chkumar246)]

So with above https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/832640 fix, lot of neutron tempest plugin failure is gone.

Below are the current failing tests:
```
neutron_tempest_plugin.scenario.test_port_forwardings.PortForwardingTestJSON.test_port_forwarding_to_2_servers[id-ab40fc48-ca8d-41a0-b2a3-f6679c847bfe]
-------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/usr/lib/python3.9/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 126, in test_port_forwarding_to_2_servers
    servers = self._prepare_resources(num_servers=2,

      File "/usr/lib/python3.9/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 67, in _prepare_resources
    server = self.create_server(

      File "/usr/lib/python3.9/site-packages/neutron_tempest_plugin/scenario/base.py", line 147, in create_server
    self.wait_for_guest_os_ready(server['server'], client=client)

      File "/usr/lib/python3.9/site-packages/neutron_tempest_plugin/scenario/base.py", line 542, in wait_for_guest_os_ready
    utils.wait_until_true(system_booted, timeout=90, sleep=5)

      File "/usr/lib/python3.9/site-packages/neutron_tempest_plugin/common/utils.py", line 82, in wait_until_true
    while not predicate():

      File "/usr/lib/python3.9/site-packages/neutron_tempest_plugin/scenario/base.py", line 535, in system_booted
    console_output = client.get_console_output(server['id'])['output']

      File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/servers_client.py", line 649, in get_console_output
    return self.action(server_id, 'os-getConsoleOutput',

      File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/servers_client.py", line 221, in action
    resp, body = self.post('servers/%s/action' % server_id,

      File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)

      File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/base_compute_client.py", line 47, in request
    resp, resp_body = super(BaseComputeClient, self).request(

      File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)

      File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,

    tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.DBConnectionError'>

```

and

```
tempest.scenario.test_network_v6.TestGettingAddress.test_slaac_from_os[compute,id-2c92df61-29f0-4eaa-bee3-7c65bef62a43,network,slow]
------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~...

[chandan kumar (chkumar246)]

On CS9 wallaby, we are seeing similar failure
```
https://logserver.rdoproject.org/15/151734d416f03f3b5c9c93b1cb999f490c49589c/openstack-periodic-integration-stable1-cs9/periodic-tripleo-ci-centos-9-standalone-wallaby/43072dc/logs/undercloud/var/log/tempest/stestr_results.html.gz

Failure seems to be different
```
{0} tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesUnderV243Test.test_add_remove_fixed_ip [349.895008s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 107, in _get_ssh_connection
        ssh.connect(self.host, port=self.port, username=self.username,
      File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 435, in connect
        self._auth(
      File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 764, in _auth
        raise saved_exception
      File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 751, in _auth
        self._transport.auth_password(username, password)
      File "/usr/lib/python3.9/site-packages/paramiko/transport.py", line 1509, in auth_password
        return self.auth_handler.wait_for_response(my_event)
      File "/usr/lib/python3.9/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response
        raise e
    paramiko.ssh_exception.AuthenticationException: Authentication failed.
```

[chandan kumar (chkumar246)]

In order to fix the above issues, We need to pin neutron-tempest-plugin to this https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/832640 and tempest to https://github.com/openstack/tempest/commit/6ded070b511263df2b6c06905615a951b29e2035

[chandan kumar (chkumar246)]

40228: Pin tempest and neutron-tempest-plugin for Wallaby and Xena | https://review.rdoproject.org/r/c/rdoinfo/+/40228

[Marios Andreou (marios-b)]

every job in the latest wallaby 9 buildset is failing on this

        * https://review.rdoproject.org/zuul/buildset/3ae5ce4e79784fab81260584860766a2

It will run again today in about an hour - we are hoping https://review.rdoproject.org/r/c/rdoinfo/+/40228/ will help but may not get it before today's run let's see

[chandan kumar (chkumar246)]

While testing the neutron tempest plugin from master. https://review.rdoproject.org/r/c/testproject/+/40260

on wallaby job: https://logserver.rdoproject.org/73/40273/1/check/distgit-tripleo-ci-centos-9-standalone-full-tempest-scenario-wallaby/ae50963/job-output.txt

we are seeing this
```
TASK [os_tempest : List tempest tests] *****************************************
2022-03-09 22:37:06.175551 | primary | Wednesday 09 March 2022 22:37:06 -0500 (0:00:02.226) 0:31:37.348 *******
2022-03-09 22:37:10.619152 | primary | fatal: [undercloud]: FAILED! => {"changed": false, "cmd": "set -e\nif [ -d /openstack/venvs/tempest-untagged/bin ];\nthen\n . /openstack/venvs/tempest-untagged/bin/activate\nfi\ntempest run -l\n", "delta": "0:00:03.014071", "end": "2022-03-10 03:37:10.395057", "msg": "non-zero return code", "rc": 100, "start": "2022-03-10 03:37:07.380986", "stderr": "", "stderr_lines": [], "stdout": "\n=========================\nFailures during discovery\n=========================\n--- import errors ---\nFailed to import test module: neutron_tempest_plugin.api.test_qos\nTraceback (most recent call last):\n File \"/usr/lib64/python3.9/unittest/loader.py\", line 436, in _find_test_path\n module = self._get_module_from_name(name)\n File \"/usr/lib64/python3.9/unittest/loader.py\", line 377, in _get_module_from_name\n __import__(name)\n File \"/usr/lib/python3.9/site-packages/neutron_tempest_plugin/api/test_qos.py\", line 1401, in <module>\n class QosMinimumPpsRuleTestJSON(base.BaseAdminNetworkTest):\n File \"/usr/lib/python3.9/site-packages/neutron_tempest_plugin/api/test_qos.py\", line 1402, in QosMinimumPpsRuleTestJSON\n RULE_NAME = qos_consts.RULE_TYPE_MINIMUM_PACKET_RATE + \"_rule\"\nAttributeError: module 'neutron_lib.services.qos.constants' has no attribute 'RULE_TYPE_MINIMUM_PACKET_RATE'\n\n================================================================================\nThe above traceback was encountered during test discovery which imports all the found test modules in the specified test_path.", "stdout_lines": ["", "=========================", "Failures during discovery", "=========================", "--- import errors ---", "Failed to import test module: neutron_tempest_plugin.api.test_qos", "Traceback (most recent call last):", " File \"/usr/lib64/python3.9/unittest/loader.py\", line 436, in _find_test_path", " module = self._get_module_from_name(name)", " File \"/usr/lib64/python3.9/unittest/loader.py\", line 377, in _get_module_from_name", " __import__(name)", " File \"/usr/lib/python3.9/site-packages/neutron_tempest_plugin/api/test_qos.py\", line 1401, in <module>", " class QosMinimumPpsRuleTestJSON(base.BaseAdminNetworkTest):", " File \"/usr/lib/python3.9/site-packages/neutron_tempest_plugin/api/test_qos.py\", line 1402, in QosMinimumPpsRuleTestJSON", " RULE_NAME = qos_consts.RULE_TYPE_MINIMUM_PACKET_RATE + \"_rule\"", "AttributeError: module 'neutron_lib.services.qos.constants' has no attribute 'RULE_TYPE_MINIMUM_PACKET_RATE'", "", "================================================================================", "The above traceback was encountered during test discovery which imports all the...

[Ronelle Landy (rlandy)]

promoted both master and wallaby - closing out

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin 1.9.0 release.