Sometimes libvirt service closes current connection during test execution

[Dennis Dmitriev]

This is a floating bug related to the connection from fuel-devops to libvirt-bin service.

Sometimes the libvirt-bin service closes the connection to the libvirt client, so the connection should be re-established.

But in fuel-devops3.0.3 the connection is cached, so after the first failure, fuel-devops tries to re-use already closed connection:

------------------------------------------
2016-11-04 11:51:16 - DEBUG - libvirt_driver.py:70 -- Cannot recv data: Connection reset by peer
2016-11-04 11:51:16 - DEBUG - decorators.py:113 -- Exception libvirtError('Cannot recv data: Connection reset by peer',) while running 'exists'. Waiting 1 sec
onds.
Traceback (most recent call last):
  File "/home/dis/devops-venv-30/local/lib/python2.7/site-packages/devops/helpers/decorators.py", line 104, in wrapper
    return func(*args, **kwargs)
  File "/home/dis/devops-venv-30/local/lib/python2.7/site-packages/devops/driver/libvirt/libvirt_driver.py", line 970, in exists
    self.driver.conn.storageVolLookupByKey(self.uuid)
  File "/home/dis/devops-venv-30/local/lib/python2.7/site-packages/libvirt.py", line 4495, in storageVolLookupByKey
    if ret is None:raise libvirtError('virStorageVolLookupByKey() failed', conn=self)
libvirtError: Cannot recv data: Connection reset by peer

2016-11-04 11:51:18 - DEBUG - decorators.py:120 -- Retrying devops.driver.libvirt.libvirt_driver.exists(<LibvirtVolume: LibvirtVolume object>, )
2016-11-04 11:51:18 - DEBUG - libvirt_driver.py:70 -- internal error: client socket is closed
2016-11-04 11:51:18 - DEBUG - decorators.py:113 -- Exception libvirtError('internal error: client socket is closed',) while running 'exists'. Waiting 1 second
s.
Traceback (most recent call last):
  File "/home/dis/devops-venv-30/local/lib/python2.7/site-packages/devops/helpers/decorators.py", line 104, in wrapper
    return func(*args, **kwargs)
  File "/home/dis/devops-venv-30/local/lib/python2.7/site-packages/devops/driver/libvirt/libvirt_driver.py", line 970, in exists
    self.driver.conn.storageVolLookupByKey(self.uuid)
  File "/home/dis/devops-venv-30/local/lib/python2.7/site-packages/libvirt.py", line 4495, in storageVolLookupByKey
    if ret is None:raise libvirtError('virStorageVolLookupByKey() failed', conn=self)
libvirtError: internal error: client socket is closed
...

2016-11-04 11:51:20 - DEBUG - decorators.py:120 -- Retrying devops.driver.libvirt.libvirt_driver.exists(<LibvirtVolume: LibvirtVolume object>, )
...
libvirtError: internal error: client socket is closed

...
------------------------------------------

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/393776
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=b5a5886c37d1aeda37e7de5553708fc76639bfe2
Submitter: Jenkins
Branch: master

commit b5a5886c37d1aeda37e7de5553708fc76639bfe2
Author: Dennis Dmitriev <email address hidden>
Date: Fri Nov 4 15:41:59 2016 +0200

    Reconnect to libvirt service if previous connection is lost

    re-trying a requiest doesn't help in cases when chached connection
    was closed.

    - do not cache the connection in a property
    - check conn.isAlive() each time when already existing connection
      is picked

    Closes-Bug:#1639788
    Change-Id: I23c65162e32c97a03fc2950c41d13b83f350fee2