Job "Tempest-9.0_LVM_Cinder_DVR_Sahara_Ceilometer_Ironic" didn't upload result

[Sofiia Andriichenko]

Job "Tempest-9.0_LVM_Cinder_DVR_Sahara_Ceilometer_Ironic" didn't upload result to testrail 16-17.08.2016 for 9.1 snapshot #140 and 9.1 snapshot #145

[Alexander Gubanov (ogubanov)]

I have been digging into this issue
(http://cz7776.bud.mirantis.net:8080/jenkins/view/Tempest_9.%D0%A5/job/9.x_Tempest_LVM_no_ssl/58/console)

and found that root cause of problem is deadlock in paramiko.
Here is processes state and logs from docker container with rally/tempest http://paste.openstack.org/raw/560778/

and gdb trace http://paste.openstack.org/raw/560777/

last tempest test "tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops"
calls remote ssh exec_command which uses paramiko recv_exit_status method that hangs and keep waiting event from another thread (which is died and never happened) forever.
Seems, it should have some timeout for receive.

Also, I found warnings in paramiko docs about method "recv_exit_status"
http://docs.paramiko.org/en/2.0/api/channel.html#paramiko.channel.Channel.recv_exit_status
http://docs.paramiko.org/en/2.0/api/channel.html#paramiko.channel.Channel.exit_status_ready

and unresolved issues with the
https://github.com/paramiko/paramiko/search?q=recv_exit_status&type=Issues&utf8=%E2%9C%93

Similar problem in tempest https://bugs.launchpad.net/tempest/+bug/1577632

Question is still open, I'm going to get to know how it works in upstream tempest.

[Alexander Gubanov (ogubanov)]

Filed new tempest upstream bug
https://bugs.launchpad.net/tempest/+bug/1615659
and fix https://review.openstack.org/#/c/358610/

waiting for tempest community feedback.

[Alexander Gubanov (ogubanov)]

Patch-set was merged https://review.openstack.org/#/c/358610/
Move to "Fix release"