Quobyte unit test fails : manila.tests.share.drivers.quobyte.test_jsonrpc.QuobyteJsonRpcTestCase.test_jsonrpc_call_ssl_disable

[li,chen]

======================================================================
FAIL: manila.tests.share.drivers.quobyte.test_jsonrpc.QuobyteJsonRpcTestCase.test_jsonrpc_call_ssl_disable
----------------------------------------------------------------------
Traceback (most recent call last):
testtools.testresult.real._StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
DEBUG [manila.share.drivers.quobyte.jsonrpc] Posting to Quobyte backend: {"params": {"retry": "INFINITELY", "param": "value"}, "jsonrpc": "2.0", "method": "method", "id": "1"}
   DEBUG [manila.share.drivers.quobyte.jsonrpc] Posting to Quobyte backend: {"params": {"retry": "INFINITELY", "param": "value"}, "jsonrpc": "2.0", "method": "method", "id": "2"}
}}}

Traceback (most recent call last):
  File "/opt/stack/manila/manila/tests/share/drivers/quobyte/test_jsonrpc.py", line 182, in test_jsonrpc_call_ssl_disable
    'method', {'param': 'value'})
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 422, in assertRaises
    self.assertThat(our_callable, matcher)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 433, in assertThat
    mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 483, in _matchHelper
    mismatch = matcher.match(matchee)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
    mismatch = self.exception_matcher.match(exc_info)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
    mismatch = matcher.match(matchee)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 414, in match
    reraise(*matchee)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
    result = matchee()
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 969, in __call__
    return self._callable_object(*self._args, **self._kwargs)
  File "/opt/stack/manila/manila/share/drivers/quobyte/jsonrpc.py", line 130, in call
    dict(Authorization=(self._credentials.
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 1172, in connect
    self.timeout, self.source_address)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/eventlet/green/socket.py", line 53, in create_connection
    sock.connect(sa)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 237, in connect
    self._trampoline(fd, write=True)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 201, in _trampoline
    mark_as_closed=self._mark_as_closed)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 162, in trampoline
    return hub.switch()
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run
    self.wait(sleep_time)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 85, in wait
    presult = self.do_poll(seconds)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll
    return self.poll.poll(seconds)
  File "/opt/stack/manila/.venv/local/lib/python2.7/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
    raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException

[Valeriy Ponomaryov (vponomaryov)]

What branch is used? How tests were run? I can not reproduce it. Unit tests work for me, as well as for CI with master branch.

[Silvan Kaiser (2-silvan)]

Same over here, just ran a few updates & tests but could not reproduce this issue.

[Silvan Kaiser (2-silvan)]

After some staring at the error log and the code:
Please try running " tox -r " to ensure your virtual environment is updated.

[li,chen (chen-li)]

Thanks! "tox -r" works!

[Clinton Knight (clintonk)]

This is a real issue, and rebuilding the virtual environment doesn't solve it.

I can't explain why it doesn't always happen, but when it does, it takes 60 seconds to fail.

Root cause is that the HTTPSConnection constructor isn't mocked but a new connection is instantiated in the method being tested. I have a fix that consistently works.

[Silvan Kaiser (2-silvan)]

I'll take a look at this again today, if you want to drop me a paste of your fix i can take a look at it, too.

[Silvan Kaiser (2-silvan)]

Hmm, still cannot reproduce. Any hints on how to reproduce this for me?
Environment, python version, etc.

Currently i have no way of testing if something i do fixes this.

[Clinton Knight (clintonk)]

Thanks for trying, Silvan. I'll push something up for your review.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/218978
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=9f747a75cd013a3a517cabc79e843919cf4f3566
Submitter: Jenkins
Branch: master

commit 9f747a75cd013a3a517cabc79e843919cf4f3566
Author: Clinton Knight <email address hidden>
Date: Fri Aug 28 00:52:09 2015 -0400

    Fix failing Quobyte unit test

    There is one Quobyte unit test, test_jsonrpc_call_ssl_disable, that
    consistently fails on my dev system. This commit adds a mock to
    resolve the issue.

    Closes-Bug: 1456938
    Change-Id: I36918f2497ce6be1e00483358a77e043b288a6fc