glance_store

glance_store exceptions trigger child exceptions (which break Glance's error handling)

Bug #1501443 reported by Stuart McLaren
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
glance_store
Fix Released
Critical
Unassigned
Nominated for Kilo by Erno Kuvaja
Nominated for Liberty by Erno Kuvaja

Bug Description

 ----------------------
 import glance_store

 def fn1():
  print('goodbye')
  raise IOError()

 def fn2():
  print('hello')
  raise glance_store.NotFound()

 try:
  fn1()
 except IOError:
  print('Caught IOError exception')
  pass

 try:
  fn2()
 except glance_store.NotFound:
  print('Caught store NotFound exception')
  pass
 -----------------

Running the above gives:

 $ python /tmp/gs.py
 goodbye
 Caught IOError exception
 hello
 Traceback (most recent call last):
  File "/tmp/gs.py", line 19, in <module>
    fn2()
  File "/tmp/gs.py", line 10, in fn2
    raise glance_store.NotFound()
  File "/usr/local/lib/python2.7/dist-packages/glance_store/exceptions.py", line 49, in __init__
    self.msg = kwargs.pop('message', None) or self.message % kwargs
 KeyError: u'image'

Revision history for this message
Ian Cordasco (icordasc) wrote :

It's been this way since January of 2014. I'm confused about what's "weird".

Changed in glance-store:
status: New → Opinion
Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :

Running this script:

-----------------------------------------------
 import glance_store

 def fn1():
  print('goodbye')
  raise IOError('bye')

 def fn2():
  print('hello')
  raise glance_store.NotFound('hi')

 try:
  fn1()
 except IOError:
  print('Caught IOError exception')
  pass

 try:
  fn2()
 except glance_store.NotFound:
  print('Caught store NotFound exception')
  pass
--------------------------------------------------

Gives:

goodbye
Caught IOError exception
hello
Traceback (most recent call last):
  File "/tmp/gs2.py", line 19, in <module>
    fn2()
  File "/tmp/gs2.py", line 10, in fn2
    raise glance_store.NotFound('hi')
TypeError: __init__() takes exactly 1 argument (2 given)

The reason this is glance store exception is relevant is it seems to break NotFound error handling. Rather than a NotFound being raised, a different exception is raised, eg:

TypeError: 'ImageProxy' object is not callable

It may be that other glance store exceptions behave the same way.

Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :
Download full text (5.7 KiB)

For example, if you inject an error with:

 git diff glance/location.py
 diff --git a/glance/location.py b/glance/location.py
 index 20f8f76..1f77e16 100644
 --- a/glance/location.py
 +++ b/glance/location.py
 @@ -396,8 +396,9 @@ class ImageProxy(glance.domain.proxy.Image):
         self.image.status = 'active'

     def get_data(self, offset=0, chunk_size=None):
 - if not self.image.locations:
 - raise store.NotFound(_("No image data could be found"))
 + raise store.NotFound(_("No image data could be found"))
 + #if not self.image.locations:
 + # raise store.NotFound(_("No image data could be found"))
         err = None
         for loc in self.image.locations:
             try:

You get:

 $ glance --os-image-api-version 2 image-download 3fe391a9-26bb-46bc-a905-a3c3b101d4a8
 HTTPInternalServerError (HTTP 500)

  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 454, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/oslo_middleware/base.py", line 106, in __call__
    response = req.get_response(self.application)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/request.py", line 1317, in send
    application, catch_exc_info=False)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
    app_iter = application(self.environ, start_response)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/mnt/sam/git/glance/glance/common/wsgi.py", line 588, in __call__
    response = req.get_response(self.application)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/request.py", line 1317, in send
    application, catch_exc_info=False)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
    app_iter = application(self.environ, start_response)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/osprofiler/web.py", line 99, in __call__
    return request.get_response(self.application)
  File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site-packages/webob/request....

Read more...

Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :

In the above example it breaks error handling because though there is code to handle the NotFound exception a TypeError exception get's triggered by NotFound. The calling code doesn't expect TypeError.

summary: - glance_store exceptions acting weird
+ glance_store exceptions trigger child exceptions (which break Glance's
+ error handling)
Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :

Injecting this in the glance code:

        try:
            raise store.NotFound(_("No image data could be found"))
        except store.NotFound as e:
            pass

raises

 TypeError: 'ImageProxy' object is not callable

Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :
Download full text (3.8 KiB)

Looking through the glance_store code

$ grep -r exceptions.NotFound . |grep -v test |grep -v :raises |grep -v Bin
./_drivers/http.py: except exceptions.NotFound:
./_drivers/http.py: raise exceptions.NotFound(message=reason)
./_drivers/filesystem.py: raise exceptions.NotFound(image=filepath)
./_drivers/filesystem.py: raise exceptions.NotFound(image=fn)
./_drivers/sheepdog.py: raise exceptions.NotFound(_("Sheepdog image %s does not exist")
./_drivers/sheepdog.py: raise exceptions.NotFound(_("Sheepdog image %s does not exist")
./_drivers/sheepdog.py: raise exceptions.NotFound(_("Sheepdog image %s does not exist") %
./_drivers/gridfs.py: raise exceptions.NotFound(msg)
./_drivers/s3.py: raise exceptions.NotFound(msg)
./_drivers/s3.py: raise exceptions.NotFound(message=msg)
./_drivers/rbd.py: raise exceptions.NotFound(
./_drivers/rbd.py: raise exceptions.NotFound(msg)
./_drivers/rbd.py: raise exceptions.NotFound(message=msg)
./_drivers/rbd.py: except exceptions.NotFound:
./_drivers/swift/store.py: raise exceptions.NotFound(message=msg)
./_drivers/swift/store.py: raise exceptions.NotFound(message=msg)
./_drivers/swift/store.py: raise exceptions.NotFound(message=msg)
./_drivers/cinder.py: raise exceptions.NotFound(reason)
./_drivers/vmware_datastore.py: raise exceptions.NotFound(message=msg)
./_drivers/vmware_datastore.py: raise exceptions.NotFound(message=reason)

It seems that most of these will trigger a secondary exception, eg:

If you modify ./_drivers/swift/store.py

to add
        try:
            raise exceptions.NotFound('xxx')
        except exceptions.NotFound:
            pass

in

 def _get_object

then try to download, you get:

 2015-10-01 12:36:52.519 TRACE glance.common.wsgi Traceback (most recent call last):
 2015-10-01 12:36:52.519 TRACE glance.common.wsgi File "/mnt/sam/git/glance/glance/common/wsgi.py", line 879, in __call__
 2015-10-01 12:36:52.519 TRACE glance.common.wsgi request, **action_args)
 2015-10-01 12:36:52.519 TRACE glance.common.wsgi File "/mnt/sam/git/glance/glance/common/wsgi.py", line 907, in dispatch
 2015-10-01 12:36:52.519 TRACE glance.common.wsgi return method(*args, **kwargs)
 2015-10-01 12:36:52.519 TRACE glance.common.wsgi File "/mnt/sam/git/glance/glance/api/v1/images.py", line 516, in show
 2015-10-01 12:36:52.519 TRACE glance.common.wsgi image_meta['location'])
 2015-10-01 12:36:52.519 TRACE glance.common.wsgi File "/mnt/sam/git/glance/glance/api/v1/images.py", line 472, in _get_from_store
 2015-10-01 12:36:52.519 TRACE glance.common.wsgi image_data, image_size = src_store.get(loc, context=context)
 2015-10-01 12:36:52.519 TRACE glance.common.wsgi File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site- packages/glance_store/capabilities.py", line 226, in op_checker
 2015-10-01 12:36:52.519 TRACE glance.common.wsgi return store_op_fun(store, *args, **kwargs)
 2015-10-01 12:36:52.519 TRACE glance.common....

Read more...

Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :

If you add:

        try:
            msg = _("xxx")
            raise exceptions.NotFound(msg=msg)
        except exceptions.NotFound:
            pass

you get:

 2015-10-01 12:39:44.191 TRACE glance.common.wsgi unicode_mod = self._safe_translate(six.text_type(self), params)
 2015-10-01 12:39:44.191 TRACE glance.common.wsgi File "/mnt/sam/git/glance/.tox/py27/local/lib/python2.7/site- packages/oslo_i18n/_message.py", line 159, in _safe_translate
 2015-10-01 12:39:44.191 TRACE glance.common.wsgi translated_message = self.msgid % translated_params
 2015-10-01 12:39:44.191 TRACE glance.common.wsgi KeyError: 'image'
 2015-10-01 12:39:44.191 TRACE glance.common.wsgi

which is the same as my other reproducer.

This is problematic, because that function (like the others) actually tries to raise a glance_store NotFound exception:

                msg = _("Swift could not find object %s.") % location.obj
                LOG.warn(msg)
                raise exceptions.NotFound(message=msg)

Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :

Note that this is probably not specific to the NotFound exception, but probably applies to most glance store exceptions.

Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :

We probably need to look at how we initialise these exceptions, eg in NotFound's case ensure that 'image' defaults to something if we're going to try to print it out.

Erno Kuvaja (jokke)
Changed in glance-store:
importance: Undecided → Critical
Revision history for this message
Kairat Kushaev (kkushaev) wrote :

https://review.openstack.org/#/c/202083/ - proposed patch

Revision history for this message
Kairat Kushaev (kkushaev) wrote :

The fix has been merged 4 months ago.

Changed in glance-store:
status: Opinion → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 0.9.2

This issue was fixed in the openstack/glance_store 0.9.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.