Test tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario triggers to an Exception SSH Timeout

[Sergii Turivnyi]

Detailed bug description:
Sometimes Tempest-Test Test tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario triggers to an Exception SSH Timeout
ISO:
    #107
    http://paste.openstack.org/show/492040/
ISO configuration: [9.0][MOSQA] Tempest 9.0 (VLAN_CEPH_GLANCE_CEPH-TEMPEST, Ceph(scenario))

Steps to reproduce:
Deploy ISO in current configuration
Run Tesmpest-Test

Expected results:
Tesmpest-Test is passed

Actual result:
Test triggers to an Exception:
Traceback (most recent call last):
  File "tempest/test.py", line 113, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "tempest/scenario/test_minimum_basic.py", line 134, in test_minimum_basic_scenario
    floating_ip['ip'], private_key=keypair['private_key'])
  File "tempest/scenario/manager.py", line 378, in get_remote_client
    linux_client.validate_authentication()
  File "tempest/common/utils/linux/remote_client.py", line 51, in validate_authentication
    self.ssh_client.test_connection_auth()
  File "tempest/lib/common/ssh.py", line 173, in test_connection_auth
    connection = self._get_ssh_connection()
  File "tempest/lib/common/ssh.py", line 88, in _get_ssh_connection
    password=self.password)
tempest.lib.exceptions.SSHTimeout: Connection to the 10.109.4.134 via SSH timed out.
User: cirros, Password: None

Reproducibility:
See attachment

Workaround:
Increase time-out in the test

[Timur Nurlygayanov (tnurlygayanov)]

The issue reproduced not in 100% of cases and it looks like the reason of the issue can be in test lab, but not in the product.
Let's increase the timeout and check does it help or not.

[Sergii Turivnyi (sturivnyi)]

It also can be reproduced on:
[9.0][MOSQA] Tempest 9.0 (VLAN_CEPH_GLANCE_CEPH-TEMPEST, Ceph(scenario))

tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_instance
Stacktrace

Traceback (most recent call last):
  File "tempest/test.py", line 113, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "tempest/scenario/test_shelve_instance.py", line 98, in test_shelve_instance
    self._create_server_then_shelve_and_unshelve()
  File "tempest/scenario/test_shelve_instance.py", line 82, in _create_server_then_shelve_and_unshelve
    private_key=keypair['private_key'])
  File "tempest/scenario/manager.py", line 623, in create_timestamp
    private_key=private_key)
  File "tempest/scenario/manager.py", line 378, in get_remote_client
    linux_client.validate_authentication()
  File "tempest/common/utils/linux/remote_client.py", line 51, in validate_authentication
    self.ssh_client.test_connection_auth()
  File "tempest/lib/common/ssh.py", line 173, in test_connection_auth
    connection = self._get_ssh_connection()
  File "tempest/lib/common/ssh.py", line 88, in _get_ssh_connection
    password=self.password)
tempest.lib.exceptions.SSHTimeout: Connection to the 10.109.4.205 via SSH timed out.
User: cirros, Password: None

[Sergii Turivnyi (sturivnyi)]

Status to `medium` was changed because it didn't reproduced on test-lab

[Oleksiy Butenko (obutenko)]

fix was added to ci-jobs

[Sofiia Andriichenko (sandriichenko)]

reproduced for: tempest.api.compute.servers.test_create_server.ServersWithSpecificFlavorTestJSON.test_verify_created_server_ephemeral_disk
Iso: 270
Tempest (LVM,Cinder,DVR,Sahara,Ceilometer,Ironic))

Traceback (most recent call last):
  File "/home/rally/.rally/tempest/for-deployment-dd2e1be8-8cc8-471d-9772-4990f3eda110/tempest/api/compute/servers/test_create_server.py", line 344, in test_verify_created_server_ephemeral_disk
    partition_num_emph = len(linux_client.get_partitions().split('\n'))
  File "/home/rally/.rally/tempest/for-deployment-dd2e1be8-8cc8-471d-9772-4990f3eda110/tempest/common/utils/linux/remote_client.py", line 70, in get_partitions
    output = self.exec_command(command)
  File "/home/rally/.rally/tempest/for-deployment-dd2e1be8-8cc8-471d-9772-4990f3eda110/tempest/common/utils/linux/remote_client.py", line 44, in exec_command
    return self.ssh_client.exec_command(cmd)
  File "/home/rally/.rally/tempest/for-deployment-dd2e1be8-8cc8-471d-9772-4990f3eda110/tempest/lib/common/ssh.py", line 118, in exec_command
    ssh = self._get_ssh_connection()
  File "/home/rally/.rally/tempest/for-deployment-dd2e1be8-8cc8-471d-9772-4990f3eda110/tempest/lib/common/ssh.py", line 88, in _get_ssh_connection
    password=self.password)
tempest.lib.exceptions.SSHTimeout: Connection to the 10.109.4.159 via SSH timed out.
User: cirros, Password: password

[Roman Podoliaka (rpodolyaka)]

Sophia, it works for me on the latest ISO (#284):

{0} tempest.api.compute.servers.test_create_server.ServersWithSpecificFlavorTestJSON.test_verify_created_server_ephemeral_disk [120.625643s] ... ok

======
Totals
======
Ran: 1 tests in 169.0000 sec.
 - Passed: 1
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 120.6256 sec.

Please make sure your environment is not overloaded. If you continue to see this issue on your environment, please ping devs in Slack and give them SSH access, so that they can jump in for live debugging.

[Oleksiy Butenko (obutenko)]

Change to Invalid
Can't reproduce this bug manual and on ci

[Sergey Nechaev (spnechaev)]

Seems to be reproduced in Jenkins here
http://logs.openstack.org/47/301747/11/check/gate-tempest-dsvm-neutron-full/c5e6f54/console.html

[Sergii Turivnyi (sturivnyi)]

@spnechaev: looks like link is broken:

> File Not Found

[Sergey Nechaev (spnechaev)]

sorry, here is right one:
http://logs.openstack.org/47/301747/11/check/gate-tempest-dsvm-neutron-full/c5e6f54/console.html.gz