oslo.serialization 2.21.2 breaks glance

[Dirk Mueller]

glance unittests break with 2.21.2, work fine with 2.21.1:

======================================================================
FAIL: glance.tests.unit.v1.test_registry_api.TestRegistryAPI.test_update_all_image_members
----------------------------------------------------------------------
Traceback (most recent call last):
  File "glance/tests/unit/v1/test_registry_api.py", line 1629, in test_update_all_image_members
    content_type='json')
  File "glance/tests/utils.py", line 564, in get_api_response_ext
    self.assertEqual(res.status_int, http_resp)
  File "/data/dmueller/src/Cloud/glance/.tox/venv/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/data/dmueller/src/Cloud/glance/.tox/venv/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 200 != 204

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/534736

[John L. Villalovos (happycamp)]

So in the unit test run using: $ tox -vvv -e py27 -- -r glance.tests.unit.v1.test_registry_api.TestRegistryAPI.test_update_all_image_members

I see that it is trying to serialize an HTTPNoContent exception and I also see a warning that in oslo.serialization 3.0 this will raise a ValueError() instead of printing a warning.

Also see it here: http://logs.openstack.org/72/533872/6/check/openstack-tox-py27/4709e32/job-output.txt.gz#_2018-01-16_13_10_38_931593

<HTTPNoContent at 0x7fe744f7fe60 204 No Content>

  File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/subunit/run.py", line 149, in <module>
    main()
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/subunit/run.py", line 145, in main
    stdout=stdout, exit=False)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/testtools/run.py", line 187, in __init__
    self.runTests()
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/testtools/run.py", line 221, in runTests
    self.result = testRunner.run(self.test)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/subunit/run.py", line 70, in run
    test(result)
  File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/unittest2/suite.py", line 87, in __call__
    return self.run(*args, **kwds)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/unittest2/suite.py", line 126, in run
    test(result)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/unittest2/suite.py", line 87, in __call__
    return self.run(*args, **kwds)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/unittest2/suite.py", line 126, in run
    test(result)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/unittest2/suite.py", line 87, in __call__
    return self.run(*args, **kwds)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/unittest2/suite.py", line 126, in run
    test(result)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 673, in __call__
    return self.run(*args, **kwds)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 671, in run
    return run_test.run(result)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/testtools/runtest.py", line 80, in run
    return self._run_one(actual_result)
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-packages/testtools/runtest.py", line 94, in _run_one
    return self._run_prepared_result(ExtendedToOriginalDecorator(result))
  File "/home/devstack/openstack/glance/.tox/py27/lib/python2.7/site-pa...

[John L. Villalovos (happycamp)]

This might fix the issue:

diff --git a/glance/common/wsgi.py b/glance/common/wsgi.py
index 6c8853da..1d763890 100644
--- a/glance/common/wsgi.py
+++ b/glance/common/wsgi.py
@@ -1142,6 +1142,8 @@ class JSONResponseSerializer(object):
             return obj.to_dict()
         if isinstance(obj, multidict.MultiDict):
             return obj.mixed()
+ if isinstance(obj, Exception):
+ return obj
         return jsonutils.to_primitive(obj)

     def to_json(self, data):

[John L. Villalovos (happycamp)]

So there is a test patch:

https://review.openstack.org/#/c/535166/

And the unit tests pass with it.

[John L. Villalovos (happycamp)]

This might be the real reason for the error, but I'm not sure how to fix it

https://github.com/openstack/glance/blob/08e28cd921920cf8f5401c839376b65c19a97880/glance/common/wsgi.py#L1250-L1252

[John L. Villalovos (happycamp)]

The reason for the previous comment is this:

https://review.openstack.org/#/c/535166/4/glance/common/wsgi.py@1149

So I was digging into this more, but going to stop as 11:30PM my time.

When this returns the obj, it actually causes a Circular reference detected exception to occur. And even without this change and the old oslo.serialization the same exception happens.

So far some reason a CircularReference exception works and returning a string representation of the HTTPNoContent fails the unit tests.

Weird!

[OpenStack Infra (hudson-openstack)]

Change abandoned by ChangBo Guo(gcb) (<email address hidden>) on branch: master
Review: https://review.openstack.org/534736

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/535166
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=6e397345b1b4f5dc9e67cebde817155693f5cf51
Submitter: Zuul
Branch: master

commit 6e397345b1b4f5dc9e67cebde817155693f5cf51
Author: John L. Villalovos <email address hidden>
Date: Wed Jan 17 21:01:29 2018 -0800

    Resolve unit test failures with going to oslo.serialization 2.3.0

    openstack/oslo.serialization made a change in commit
    c1a7079c26d27a2e46cca26963d3d9aa040bdbe8 which changed how
    jsonutils.to_primitive() handles Exception objects. Previously the
    actual Exception object would be returned but now it returns a 'repr'
    of the object. This was done because CircularReferences were occuring
    in some use cases with Exceptions and also in version 3.0 of
    oslo.serialization they were going to raise a ValueError() for
    unhandled types and this change caused Exceptions to now be handled.

    The code previously would call the serialization function and if any
    unknown Exception occurred it would just return action_result. With
    the change to oslo.serialization an exception was no longer being
    raised and thus no Exception would be raised and caught.

    Now check if action_result is an Exception and if so return it.

    Closes-Bug: #1728368
    Change-Id: Ic374ec891a65f603433091cdae27b0d4aac8362f

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 16.0.0.0rc1 release candidate.