Glance GET /images/detail DetachedIntanceError on 'tags' attribute

[Raymond Pekowski]

---------- Forwarded message ----------
From: Ray Pekowski <email address hidden>
Date: Wed, May 15, 2013 at 3:51 PM
Subject: Glance GET /images/detail DetachedIntanceError on 'tags' attribute
To: OpenStack Development Mailing List <email address hidden>

In my testing of Folsom in an HA environment including the use of haproxy fronting three "contoller" nodes and mysql using wsrep on each of those nodes I have been seeing an occasional HTTP 500 error on Glance image list operations. I don't know if this is due to my HA environment. I have only done a little troubleshooting and web searching on the problem. I'll be a little lazy and post what I am seeing here in case it is a well known problem. Here is a wireshark dump of the failing API flying across the wire:

GET /images/detail?limit=20 HTTP/1.1
Host: 10.49.56.250:9191
Accept-Encoding: identity
x-openstack-request-id: req-b1d626b9-fb91-45d6-8e7b-029caf143fd6
x-auth-token: 3600d26b1a6348859141ac5cf4e80b9f

HTTP/1.1 500 Internal Server Error
Content-Type: text/plain
Content-Length: 2839
Date: Wed, 15 May 2013 17:39:57 GMT
Connection: close

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 336, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 279, in __call__
    return self.app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 326, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 533, in __call__
    request, **action_args)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 550, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/registry/api/v1/images.py", line 117, in detail
    image_dicts = [make_image_dict(i) for i in images]
  File "/usr/lib/python2.7/dist-packages/glance/registry/api/v1/images.py", line 447, in make_image_dict
    image_tags = image['tags']
  File "/usr/lib/python2.7/dist-packages/glance/db/sqlalchemy/models.py", line 77, in __getitem__
    return getattr(self, key)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 168, in __get__
    return self.impl.get(instance_state(instance),dict_)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 453, in get
    value = self.callable_(state, passive)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 481, in _load_for_state
    (mapperutil.state_str(state), self.key)
DetachedInstanceError: Parent instance <Image at 0x352f810> is not bound to a Session; lazy load operation of attribute 'tags' cannot proceed

Any clue?

Ray

---------- Forwarded message ----------
From: Flavio Percoco <email address hidden>
Date: Thu, May 16, 2013 at 3:27 AM
Subject: Re: [openstack-dev] Glance GET /images/detail DetachedIntanceError on 'tags' attribute
To: OpenStack Development Mailing List <email address hidden>

On 15/05/13 15:51 -0500, Ray Pekowski wrote:

      In my testing of Folsom in an HA environment including the use of
      haproxy fronting three "contoller" nodes and mysql using wsrep on each
      of those nodes I have been seeing an occasional HTTP 500 error on
      Glance image list operations. I don't know if this is due to my HA
      environment. I have only done a little troubleshooting and web
      searching on the problem. I'll be a little lazy and post what I am
      seeing here in case it is a well known problem. Here is a wireshark
      dump of the failing API flying across the wire:

      Any clue?

Yep, that looks like a bug.
TL;DR:

The session attribute is removed from images dictionaries in the
to_dict method[0] and it looks like that's happening before tags are
loaded, which means that it won't be possible to load them since
there's no session and sqlalchemy gets really mad. :D

Please, file a bug for this.

Thanks a lot.
FF

[0]
https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L606

[Flavio Percoco (flaper87)]

mhh, I was going to work on this but there's something wrong with your report.

What version are you using? I checked master, grizzly and folsom and none of them have:

File "/usr/lib/python2.7/dist-packages/glance/registry/api/v1/images.py", line 447, in make_image_dict
    image_tags = image['tags']

And the line bellow is not on line N 117:

image_dicts = [make_image_dict(i) for i in images]

Did you modify that file?

https://github.com/openstack/glance/blob/master/glance/registry/api/v1/images.py#L447

Cheers

[Kashyap Chamarthy (kashyapc)]

Ray,

Thanks for the report, but also please try to include any special information like if you've altered source on your local system. It'd help narrow down the issue at hand.

Thank you

Also: https://wiki.openstack.org/wiki/BugFilingRecommendations

[Raymond Pekowski (pekowski)]

Sorry for the false alarm on this bug. I didn't realize that my code base had diverged from the OpenStack trunk. Apparently, we have some code changes to support image tagging from Mirantis. I found the problem, but I don't think it is applicable to how image tagging has been officially implemented in OpenStack.

As for future bug reports, I will try to following the guidelines more carefully. Thanks for you patience.