test_create_backing_from_stream_optimized_file fails with "AttributeError: '_SentinelObject' object has no attribute 'readlines'"

[Alex O'Rourke]

There are multiple patches failing cinder.tests.unit.test_vmware_vmdk.VMwareVcVmdkDriverTestCase.test_create_backing_from_stream_optimized_file with the same error: "AttributeError: '_SentinelObject' object has no attribute 'readlines'"

Please see the following patches for examples:
https://jenkins01.openstack.org/job/gate-cinder-python27/1172/console
https://jenkins01.openstack.org/job/gate-cinder-python27/1176/console
https://jenkins04.openstack.org/job/gate-cinder-python27/1348/console
https://jenkins06.openstack.org/job/gate-cinder-python27/4256/console

2015-08-25 22:37:15.391 | ==============================
2015-08-25 22:37:15.391 | Failed 1 tests - output below:
2015-08-25 22:37:15.391 | ==============================
2015-08-25 22:37:15.391 |
2015-08-25 22:37:15.391 | cinder.tests.unit.test_vmware_vmdk.VMwareVcVmdkDriverTestCase.test_create_backing_from_stream_optimized_file
2015-08-25 22:37:15.391 | ------------------------------------------------------------------------------------------------------------
2015-08-25 22:37:15.391 |
2015-08-25 22:37:15.391 | Captured traceback:
2015-08-25 22:37:15.392 | ~~~~~~~~~~~~~~~~~~~
2015-08-25 22:37:15.392 | Traceback (most recent call last):
2015-08-25 22:37:15.392 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
2015-08-25 22:37:15.392 | return func(*args, **keywargs)
2015-08-25 22:37:15.392 | File "cinder/tests/unit/test_vmware_vmdk.py", line 2568, in test_create_backing_from_stream_optimized_file
2015-08-25 22:37:15.392 | file_open, download_data, delete_temp_backing)
2015-08-25 22:37:15.392 | File "cinder/tests/unit/test_vmware_vmdk.py", line 1723, in _test_create_backing_from_stream_optimized_file
2015-08-25 22:37:15.392 | context, name, volume, tmp_file_path, file_size_bytes)
2015-08-25 22:37:15.392 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 422, in assertRaises
2015-08-25 22:37:15.392 | self.assertThat(our_callable, matcher)
2015-08-25 22:37:15.392 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 433, in assertThat
2015-08-25 22:37:15.393 | mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
2015-08-25 22:37:15.393 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 483, in _matchHelper
2015-08-25 22:37:15.393 | mismatch = matcher.match(matchee)
2015-08-25 22:37:15.393 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
2015-08-25 22:37:15.393 | mismatch = self.exception_matcher.match(exc_info)
2015-08-25 22:37:15.393 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
2015-08-25 22:37:15.393 | mismatch = matcher.match(matchee)
2015-08-25 22:37:15.393 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 414, in match
2015-08-25 22:37:15.393 | reraise(*matchee)
2015-08-25 22:37:15.393 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
2015-08-25 22:37:15.393 | result = matchee()
2015-08-25 22:37:15.393 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 969, in __call__
2015-08-25 22:37:15.394 | return self._callable_object(*self._args, **self._kwargs)
2015-08-25 22:37:15.394 | File "cinder/volume/drivers/vmware/vmdk.py", line 1682, in _create_backing_from_stream_optimized_file
2015-08-25 22:37:15.394 | self._delete_temp_backing(backing)
2015-08-25 22:37:15.394 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 192, in __exit__
2015-08-25 22:37:15.394 | self.tb))
2015-08-25 22:37:15.394 | File "/usr/lib/python2.7/traceback.py", line 141, in format_exception
2015-08-25 22:37:15.394 | list = list + format_tb(tb, limit)
2015-08-25 22:37:15.394 | File "/usr/lib/python2.7/traceback.py", line 76, in format_tb
2015-08-25 22:37:15.394 | return format_list(extract_tb(tb, limit))
2015-08-25 22:37:15.394 | File "/usr/lib/python2.7/traceback.py", line 101, in extract_tb
2015-08-25 22:37:15.394 | line = linecache.getline(filename, lineno, f.f_globals)
2015-08-25 22:37:15.395 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/lib/python2.7/linecache.py", line 14, in getline
2015-08-25 22:37:15.395 | lines = getlines(filename, module_globals)
2015-08-25 22:37:15.395 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/lib/python2.7/linecache.py", line 40, in getlines
2015-08-25 22:37:15.395 | return updatecache(filename, module_globals)
2015-08-25 22:37:15.395 | File "/home/jenkins/workspace/gate-cinder-python27/.tox/py27/lib/python2.7/linecache.py", line 133, in updatecache
2015-08-25 22:37:15.395 | lines = fp.readlines()
2015-08-25 22:37:15.395 | AttributeError: '_SentinelObject' object has no attribute 'readlines'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/217882

[Jordan Pittier (jordan-pittier)]

Logstash query:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQXR0cmlidXRlRXJyb3I6ICdfU2VudGluZWxPYmplY3QnIG9iamVjdCBoYXMgbm8gYXR0cmlidXRlICdyZWFkbGluZXMnXCIgQU5EIHRhZ3M6Y29uc29sZSIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MDc2NjY0MDY5OX0=

[Nicolas Trangez (eikke)]

Some analysis:

- test_create_backing_from_stream_optimized_file mocks cinder.volume.drivers.vmware.vmdk.open (https://github.com/openstack/cinder/blob/a9d071ce70af754698b7a87a1ffbf7812118e034/cinder/tests/unit/test_vmware_vmdk.py#L1653)
- cinder.volume.drivers.vmware.vmdk.open is actually the builtin 'open' (after https://github.com/openstack/cinder/commit/212aff327a36f925be3be69b9b54b946dbdd5c2a#diff-286878cdf01623c40709b1de7600a970)
- For some reason, in _create_backing_from_stream_optimized_file (https://github.com/openstack/cinder/blob/a9d071ce70af754698b7a87a1ffbf7812118e034/cinder/volume/drivers/vmware/vmdk.py#L1629) an exception is raised, presumably in download_stream_optimized_data
- This causes an excutils.save_and_reraise_exception block to be entered in an `except Exception` handler
- excutils.save_and_reraise_exception uses traceback.format_exception, which uses linecache, which uses a `with open(...)` block, then the readlines method on the value provided by the context manager (which is now the mocked open), and things go wrong

This implies the original exception is masked by an exception in the exception handler and not properly re-raised, and as such currently unknown (to me at least).

To prevent things like this happening, I'd strongly advise against mocking globals outside the library being tested whenever possible, especially builtins: you never know who relies on them.

[Tom Barron (tpb)]

"For some reason ... an exceptions is raised, presumably in download_stream_optimized_data".

Note that this test method really is really two test cases, a positive test and a negative test combined. The negative test raises this exception as a side effect:

https://github.com/openstack/cinder/blob/a9d071ce70af754698b7a87a1ffbf7812118e034/cinder/tests/unit/test_vmware_vmdk.py#L1717L1724

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/217882
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=0cda089b341bbf0fda03b1c27edefec1771a6c04
Submitter: Jenkins
Branch: master

commit 0cda089b341bbf0fda03b1c27edefec1771a6c04
Author: Jon Bernard <email address hidden>
Date: Thu Aug 27 16:03:57 2015 -0400

    Skip intermittent VMDK tests

    The test 'test_create_backing_from_stream_optimized_file' is failing
    intermittently in the gate, which causes delays in the review process of
    other patches. Given the proximity to feature freeze, I think it's
    appropriate to temporarily disable these tests until a proper solution
    can be found.

    Change-Id: I056b64ad09bb4eb921638e2a664eda2c97daa42d
    Partial-Bug: #1488690

[Vipin Balachandran (vbala)]

Based on comment #3, posted this fix:
https://review.openstack.org/#/c/220438/

Nicholas,

There are two versions of test_create_backing_from_stream_optimized_file, one for VMwareVcVmdkDriver and the other for deprecated VMwareEsxVmdkDriver .

>cinder.volume.drivers.vmware.vmdk.open is actually the builtin 'open' (after https://github.com/openstack/cinder/commit/212aff327a36f925be3be69b9b54b946dbdd5c2a#diff-286878cdf01623c40709b1de7600a970)

This is for VMwareEsxVmdkDriver .test_create_backing_from_stream_optimized_file

Change introduced by comit 212aff327a36f925be3be69b9b54b946dbdd5c2a in VMwareEsxVmdkDriver .test_create_backing_from_stream_optimized_file:

- @mock.patch('cinder.openstack.common.fileutils.file_open')
+ @mock.patch('cinder.volume.drivers.vmware.vmdk.open')

Change in VMwareVcVmdkDriver .test_create_backing_from_stream_optimized_file (the one which is failing):

- @mock.patch('cinder.openstack.common.fileutils.file_open')
+ @mock.patch('six.moves.builtins.open')

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/220438
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=05c2ba2ebc6041c2e3d3481c1898a90517d01e5b
Submitter: Jenkins
Branch: master

commit 05c2ba2ebc6041c2e3d3481c1898a90517d01e5b
Author: Vipin Balachandran <email address hidden>
Date: Fri Sep 4 13:47:34 2015 +0530

    VMware: Remove global patching of open

    Commit 212aff327a36f925be3be69b9b54b946dbdd5c2a replaced
    'cinder.openstack.common.fileutils.file_open' with 'open'
    from builtins. It also updated unit tests to patch 'six.
    moves.builtins.open' instead of 'cinder.openstack.common.
    fileutils.file_open'. This resulted in intermittent failure
    of test 'test_create_backing_from_stream_optimized_file'
    in VMwareVcVmdkDriverTestCase. The failure happens because
    the test patches 'open' from builtins and tests code which
    throws an exception in a 'with open(...)' block. This
    exception is handled in excutils.save_and_reraise_exception
    which uses 'with open(...)' and the exception handling
    fails due to the global patching of 'open'.

    This patch replaces the global patching of builtins 'open'
    with a patch decorator which binds 'open' to the namespace
    of the code under testing.

    Closes-bug: #1488690
    Change-Id: Ib0f47b8fcbc54a9111c1bdff6d4b6fa7568518ca