unstable unit tests

[Valeriy Ponomaryov]

I get following unit tests errors running them as part of 'cover' tox job:

${PYTHON:-python} -m subunit.run discover -t ./ ./manila/tests --load-list /tmp/tmpjgWnXE
======================================================================
FAIL: manila.tests.scheduler.test_host_manager.HostManagerTestCase.test_get_all_host_states_share
tags: worker-0
----------------------------------------------------------------------
Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
WARNING [manila.scheduler.host_manager] Share service is down. (host: host1@AAA)
 WARNING [manila.scheduler.host_manager] Share service is down. (host: host2@BBB)
 WARNING [manila.scheduler.host_manager] Share service is down. (host: host3@CCC)
 WARNING [manila.scheduler.host_manager] Share service is down. (host: host4@DDD)
 WARNING [manila.scheduler.host_manager] Share service is down. (host: host5@EEE)
}}}

Traceback (most recent call last):
  File "manila/tests/scheduler/test_host_manager.py", line 157, in test_get_all_host_states_share
    self.assertEqual(4, len(host_state_map))
  File "/opt/stack/manila/.tox/cover/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/manila/.tox/cover/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 4 != 0
======================================================================
FAIL: manila.tests.test_wsgi.TestWSGIServer.test_app_using_ssl
tags: worker-1
----------------------------------------------------------------------
Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{INFO [manila.wsgi] Started test_app on 0.0.0.0:43225}}}

Traceback (most recent call last):
  File "manila/tests/test_wsgi.py", line 154, in test_app_using_ssl
    response = urllib2.urlopen('https://127.0.0.1:%d/' % server.port)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error <Mock name='mock().getresponse().status' id='140179632986768'>: <Mock name='mock().getresponse().reason' id='140179632986704'>

Also, last one was observed yesterday on running of py27 tox job:

http://logs.openstack.org/91/202991/4/gate/gate-manila-python27/bcc7328/testr_results.html.gz

[Valeriy Ponomaryov (vponomaryov)]

+1:

======================================================================
FAIL: manila.tests.test_wsgi.TestWSGIServer.test_app_using_ipv6_and_ssl
tags: worker-0
----------------------------------------------------------------------
Empty attachments:
  pythonlogging:''
  stderr
  stdout

Traceback (most recent call last):
  File "manila/tests/test_wsgi.py", line 187, in test_app_using_ipv6_and_ssl
    response = urllib2.urlopen('https://[::1]:%d/' % server.port)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error <Mock name='mock().getresponse().status' id='140671740408464'>: <Mock name='mock().getresponse().reason' id='140671740408592'>

[Valeriy Ponomaryov (vponomaryov)]

It occurs often running unit tests in two threads:

$ tox -e py27 -- --concurrency=2

[Valeriy Ponomaryov (vponomaryov)]

It became a blocker:

https://jenkins07.openstack.org/job/gate-manila-python27/829/console

https://jenkins06.openstack.org/job/gate-manila-python27/754/console

[Valeriy Ponomaryov (vponomaryov)]

Found root cause. Problem was in not released mocked objects.

See https://github.com/openstack/manila/commit/dabe8c7a24a8cf81796d2b0f38886dae99b75775

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/205456

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/205456
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=694044e9fcc14dc026a623b51333dc86e5c60d3a
Submitter: Jenkins
Branch: master

commit 694044e9fcc14dc026a623b51333dc86e5c60d3a
Author: Valeriy Ponomaryov <email address hidden>
Date: Fri Jul 24 12:16:11 2015 +0300

    Fix unit tests for quobyte

    Unit tests in module 'manila.tests.share.drivers.quobyte.test_jsonrpc' use
    permanent mocking that leads to broken unit tests that run after this one
    and use mocked stuff expecting it be not mocked.

    So, remove permanent mocks and reuse per-test mocking 'self.mock_object'.

    Change-Id: Id575302545a5a9c09560c42bafc121dab82fc0c0
    Closes-Bug: #1477852