glance-api fails on image delivery: AttributeError: context

[Adam Gandelman]

On ubuntu natty, with a functional swift backend (can publish images into swift from the glance server or via nova), I can not deliver images to compute nodes.

I originally was blocked at an error in httplib: InvalidURL("nonnumeric port: '%s'" % host[i+1:]). I found bug #771849 and have installed glance from trunk. As of today, attempting to run an instance results in:

  File "/usr/lib/pymodules/python2.7/glance/api/v1/images.py", line 526, in get_image_meta_or_404
    request.context, id)
  File "/usr/lib/pymodules/python2.7/webob/request.py", line 1065, in __getattr__
    raise AttributeError(attr)

Full trace is attached from glance-api log.

I haven't attempted to get trunk working on any of the nova nodes, but will hopefully in the coming days.

glance version: 2011.3~d3~20110723.158-0ubuntu0ppa1~natty1
nova: 2011.3~d2-0ubuntu0~ppa1~natty1
swift: 1.4.1-0ubuntu0~ppa1~natty1

Easily reproducible on my end, can gather more information as needed.

[Jay Pipes (jaypipes)]

Hi Adam,

Looks like you've run into an issue with the packaging not upgrading the conf files properly...

Could you pastebin your glance-api.conf please? Remember to remove any security creds, of course.

Cheers,
jay

[Adam Gandelman (gandelman-a)]

Thanks for the prompt reply. Attached is the glance-api.conf I've been using.

[Jay Pipes (jaypipes)]

Hi again, Adam.

OK, yep, this is a bug in our packaging. For a quick fix, you need to replace the bottom part of your glance-api.conf with this:

http://paste.openstack.org/show/1939/

And replace the bottom part of your glance-registry.conf with this:

http://paste.openstack.org/show/1940/

Cheers!
jay

[Jay Pipes (jaypipes)]

Related: https://blueprints.launchpad.net/openstack-ci/+spec/glance-upgrade

[Jay Pipes (jaypipes)]

Some thoughts on solutions to this bug:

<clayg> jaypipes: I ran into that error as well, work around was of course adding the new context middleware
* dolphm has quit (Remote host closed the connection)
<jaypipes> clayg: yup... actually, dprince had suggested separating config files from paste.deploy to make things like this easier to upgrade properly... not sure I completely agree with that, though.
<clayg> but I wonder why not just if not hasattr(req, 'context'): req.context = DefaultAdminContext
<jaypipes> clayg: I added a blueprint to the openstack-ci project for this general problem... https://blueprints.launchpad.net/openstack-ci/+spec/glance-upgrade
<clayg> i don't know splitting your paste config from the app config, paste deploy .ini configs are pretty standard these days
<clayg> i don't really see it as an upgrade problem as much as a "required middleware" problem
<clayg> why should the app assume that something upstream added a non-standard attribute to the request?
<jaypipes> clayg: I gather you are suggesting the best way would to have not created required middleware :)
<clayg> I just mean the app could make a default decision if context attribute isn't there
<jaypipes> clayg: ya, I hear you.
<clayg> ok
<jaypipes> clayg: or even add it to wsgi.environ['context'] instead, making it environ.get('context')'able...
<jaypipes> clayg: which I think may have been how nova implemented it... anyways, point taken :)
<clayg> idk, namespacing problems apply either way, when you add an attribute to a request webob's just putting it in environ['addhoc-attrs']['attr_name']
<creiht> wsgi.environ['glance.context']
<creiht> :)
<creiht> is typical for that type of stuff
<jaypipes> creiht: ah, yes, thx
<clayg> I presonally always liked adding a attribute with the name of the app - which itself is a dict i.e. req.glance.get('context', default_context)

[Adam Gandelman (gandelman-a)]

Finally had a chance to get this going again and correct the config problems, however, I now see this. If this warrants a new bug, I will gladly file. I'll post here first in case it is related to configuration upgrade issues.

2011-07-26 21:31:47 DEBUG [routes.middleware] Route path: '/images/:(id)', defaults: {'action': u'show', 'controller': <glance.common.wsgi.Resource object at 0x91bbf4c>}
2011-07-26 21:31:47 DEBUG [routes.middleware] Match dict: {'action': u'show', 'controller': <glance.common.wsgi.Resource object at 0x91bbf4c>, 'id': u'1'}
2011-07-26 21:31:47 DEBUG [eventlet.wsgi.server] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 347, in handle_one_response
    for data in result:
  File "/usr/lib/pymodules/python2.7/glance/api/v1/images.py", line 194, in image_iterator
    options=self.options)
  File "/usr/lib/pymodules/python2.7/glance/store/__init__.py", line 79, in get_from_backend
    backend_class = get_backend_class(loc.store_name)
  File "/usr/lib/pymodules/python2.7/glance/store/__init__.py", line 72, in get_backend_class
    raise UnsupportedBackend("No backend found for '%s'" % backend)
UnsupportedBackend: No backend found for 'swift+https'
2011-07-26 21:31:47 DEBUG [eventlet.wsgi.server] 10.208.241.236 - - [26/Jul/2011 21:31:47] "GET /v1/images/1 HTTP/1.1" 200 0 0.020222

Attaching updated glance-api.conf and glance-registry.conf

[Adam Gandelman (gandelman-a)]

 ?field.comment=

[Adam Gandelman (gandelman-a)]

Please disregard previous comments regarding error "UnsupportedBackend: No backend found for 'swift+https'", looks like this was resolved as support for multiple image URIs stabilized.

[Brian Waldon (bcwaldon)]

Jay, I'm curious what the bug here actually is. This doesn't seem like something we need to address in Glance proper.

[Kevin L. Mitchell (klmitch)]

I believe he was thinking that we should have a less opaque error message reported in the case of configuration format changes like this.

[Jay Pipes (jaypipes)]

I don't either... I think we can probably close this one up and target https://bugs.launchpad.net/glance/+bug/815208 for Essex 1 and discuss the separation of paste deploy stuff from the other config stuff at the design summit.

[Jay Pipes (jaypipes)]

On Thu, Sep 15, 2011 at 6:30 PM, Kevin L. Mitchell
<email address hidden> wrote:
> I believe he was thinking that we should have a less opaque error
> message reported in the case of configuration format changes like this.

Yes, I agree. I was just suggesting we handle that in the fix that
separates paste from other config stuff...

-jay

[Jay Pipes (jaypipes)]

Adam Gandelman, can you confirm if this bug is still a bug?

Thanks!
-jay

[Adam Gandelman (gandelman-a)]

Hi Jay-

Upgrading Glance from a version earlier than when the new middleware was introduced will require manual config modification, as I don't think that stuff can't really be inserted into config via packaging. I believe we can close this, deal with what we can in packaging and at the summit discuss better ways of managing middleware config through something like a conf.d directory.

[Brian Waldon (bcwaldon)]

Marking invalid, please reopen if you feel otherwise.

[Justin Randall (jrlogicle)]

# glance --version
glance 2012.1

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04 LTS
Release: 12.04
Codename: precise

I am working through the OpenStack Installation Guide (http://docs.openstack.org/trunk/openstack-compute/install/content/images-verifying-install.html). In the glance/api.log I see an identical call stack (different line numbers since there have been some changes since this bug was filed).

This is likely user error. The real bug is that the output from the glance command to add an image is meaningless to anyone trying to troubleshoot the problem. It is difficult to tell which part of the stack is failing. It could be in glance, glance-api, keystone ...

Since I am on essex + precice, the workaround seems to be ineffective.