Paramiko error <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error' occured after executing system tests

[Alexander Ignatov]

Strange behavior observed in this job https://packaging-ci.infra.mirantis.net/job/8.0-pkg-systest-ubuntu/1541/console

Job shows that OSTF tests are passed successfully but got an exception during shutdown/cleanup.

2015-12-21 09:17:22,885 - INFO fuel_web_client.py:253 -- OSTF test statuses are :

 Create volume and boot instance from it success
 Launch instance, create snapshot, launch instance from snapshot success
 Create instance flavor success
 Request volume list success
 Request user list success
 Request image list using Glance v2 success
 Create security group success
 Launch instance success
 Check if amount of tables in databases is the same on each node success
 Request active services list success
 RabbitMQ replication success
 Check pacemaker status success
 Check state of haproxy backends on controllers success
 Check that required services are running success
 Create user and authenticate with it. success
 Request image list using Glance v1 success
 Request snapshot list success
 Check network connectivity from instance via floating IP success
 RabbitMQ availability success
 Create keypair success
 Request flavor list success
 Check create, update and delete image actions using Glance v1 success
 Check create, update and delete image actions using Glance v2 success
 Request image list using Nova success
 Create volume and attach it to instance success
 Request instance list success
 Request stack list success
 Check galera environment state success
 Check network parameters success
 Request list of networks success
 Check data replication over mysql success
 Request absolute limits list success

2015-12-21 09:17:24,121 - INFO environment.py:400 -- Admin node login name: 'root' , password: 'r00tme'
2015-12-21 09:17:24,335 - INFO fuel_web_client.py:1056 -- Wait for task 600 seconds:
 status running
 name dump
 cluster None
 result {}
 progress 0
 message None
 id 14
 uuid fd40b250-b0c3-4964-af21-47795f34a4e2
2015-12-21 09:20:09,852 - INFO fuel_web_client.py:1073 -- Task finished. Took 165.509234905 seconds.
 status ready
 name dump
 cluster None
 result {}
 progress 100
 message /api/dump/fuel-snapshot-2015-12-21_09-17-24.tar.xz
 id 14
 uuid fd40b250-b0c3-4964-af21-47795f34a4e2
2015-12-21 09:20:09,858 - INFO decorators.py:47 -- Saving logs to "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/logs/pass_ceph_rados_gw-fuel-snapshot-2015-12-21_09-17-24.tar.xz" file
ok

----------------------------------------------------------------------
Ran 3 tests in 8671.091s

OK
Exception in thread Thread-521 (most likely raised during interpreter shutdown):
 Exception in thread Thread-293 (most likely raised during interpreter shutdown):
 Exception in thread Thread-229 (most likely raised during interpreter shutdown):Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
Exception in thread Thread-397 (most likely raised during interpreter shutdown):Traceback (most recent call last):Exception in thread Thread-545 (most likely raised during interpreter shutdown): File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/transport.py", line 1777, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/transport.py", line 1777, in run

  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/transport.py", line 1777, in runTraceback (most recent call last):
Segmentation fault
+ ec=139
+ [ != no ]
+ find /home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/logs -name fail*.tar.xz -type f
+ [ != yes ]
+ [ no = yes ]
+ dos.py destroy 8.0-pkg-systest.1541.1541
+ exit 139
Build step 'Execute shell' marked build as failure

[Artem Panchenko (apanchenko-8)]

Looks like issue with paramiko came again, related issue bug #1455357

[Igor Marnat (imarnat)]

As per out agreement with Dmitry Borodaenko, bugs which cause tests in CI to fail (as this one) should be at least of High priority. Bumping to high.

[Tatyanka (tatyana-leontovich)]

Looks like issue is not reproduced now , If you see this behaviour against on the latest qa code, please feel free confirm the issue and ping me in our usual channels

[Igor Marnat (imarnat)]

Might be fixed by fix for https://bugs.launchpad.net/fuel/+bug/1503210

[Tatyanka (tatyana-leontovich)]

looks like fix from https://bugs.launchpad.net/fuel/+bug/1503210 helps to resolve the problem, so move to fix committed status, If you see this issue again, please reopen this report

[Sergey Kreys (skreys)]

We still observe this issue.
Sometimes it's just a bunch of errors, but sometimes it fails with segfault which leads to test failure.
Note: more SSH connections used in the test -> more probability to reproduce the issue.
Thus, more nodes in the environment - more chance to hit this bug as well.

This bug is related to known issues:
paramiko: https://github.com/paramiko/paramiko/issues/17
python: http://bugs.python.org/issue1856

We believe it's possible to avoid this bug if we will close SSH connections opened by fuel-qa.
There is method clean_all_connections(), but it is called in resume_environment() only.
Maybe calling this method on tests tear-down/exit will fix the issue.