Upload broken with Keystone unless tenant is specified

[justinsb]

Works with SERVICE_TOKEN:
root@openstack1:~/images# glance -A c29a31e699717cdad649 add name=imagetest < disk.qcow2
Added new image with ID: baed02be-0bc8-48d9-b628-7cb1222a8607

Fails with credentials:
root@openstack1:~/images# glance --username=demo --password=supersecret --auth_strategy=keystone --auth_url=http://127.0.0.1:5000/v2.0/ add name=imagetest < disk.qcow2 Failed to add image. Got error:
Internal Server error: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 336, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 279, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/keystone/keystone/middleware/auth_token.py", line 343, in __call__
    return self._forward_request(env, start_response, proxy_headers)
  File "/opt/stack/keystone/keystone/middleware/auth_token.py", line 576, in _forward_request
    return self.app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 279, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 477, in __call__
    request, **action_args)
  File "/opt/stack/glance/glance/common/wsgi.py", line 494, in dispatch
    return method(*args, **kwargs)
  File "/opt/stack/glance/glance/api/v1/images.py", line 526, in create
    image_meta = self._upload_and_activate(req, image_meta)
  File "/opt/stack/glance/glance/api/v1/images.py", line 478, in _upload_and_activate
    location = self._upload(req, image_meta)
  File "/opt/stack/glance/glance/api/v1/images.py", line 329, in _upload
    {'status': 'saving'})
  File "/opt/stack/glance/glance/registry/__init__.py", line 126, in update_image_metadata
    return c.update_image(image_id, image_meta, purge_props)
  File "/opt/stack/glance/glance/registry/client.py", line 142, in update_image
    res = self.do_request("PUT", "/images/%s" % image_id, body, headers)
  File "/opt/stack/glance/glance/common/client.py", line 51, in wrapped
    return func(self, *args, **kwargs)
  File "/opt/stack/glance/glance/common/client.py", line 331, in do_request
    headers=headers)
  File "/opt/stack/glance/glance/common/client.py", line 68, in wrapped
    return func(self, method, url, body, headers)
  File "/opt/stack/glance/glance/common/client.py", line 435, in _do_request
    raise exception.NotFound(res.read())
NotFound: An object with the specified identifier was not found.
Details: Image not found

Note: Your image metadata may still be in the registry, but the image's status will likely be 'killed'.

[justinsb (justin-fathomdb)]

Even if I get a token from keystone I still get the same error...

root@openstack1:~/images# curl -d '{"auth":{"passwordCredentials":{"username": "demo", "password": "supersecret"}}}' -H "Content-type: application/json" http://localhost:5000/v2.0/tokens
{"access": {"token": {"expires": "2012-02-03T10:26:00", "id": "27288a04-e3c9-4fc4-a985-82fd22aded18"}, "serviceCatalog": [{"endpoints": [{"id": "2", "region": "RegionOne", "internalURL": "http://192.168.69.1:8773/services/Cloud", "publicURL": "http://192.168.69.1:8773/services/Cloud"}], "type": "ec2", "name": "ec2"}, {"endpoints": [{"id": "3", "region": "RegionOne", "internalURL": "http://192.168.69.1:9292/v1", "publicURL": "http://192.168.69.1:9292/v1"}], "type": "image", "name": "glance"}, {"endpoints": [{"id": "4", "region": "RegionOne", "internalURL": "http://192.168.69.1:5000/v2.0", "publicURL": "http://192.168.69.1:5000/v2.0"}], "type": "identity", "name": "keystone"}], "user": {"id": "f7a3552b662b4726bc110f1744de2daa", "roles": [], "name": "demo"}}}

root@openstack1:~/images# glance -A 27288a04-e3c9-4fc4-a985-82fd22aded18 add name=imagetest2 < disk.qcow2
(same error)

[Jay Pipes (jaypipes)]

Justin, if you supply a bogus location=http://example.com/images/123 instead of redirecting the image file, do you still get the same error?

[justinsb (justin-fathomdb)]

If it's relevant, I should have said that previously it looks like it uploads the image and then fails updating the metadata, because there's a pause for 10 seconds or so.

-----

With a dummy URL, no pause, different error;:
root@openstack1:~/images# glance --username=demo --password=supersecret --auth_strategy=keystone --auth_url=http://127.0.0.1:5000/v2.0/ add name=imagetest location=http://example.com/images/123
Failed to add image. Got error:
Internal Server error: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 336, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 279, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/keystone/keystone/middleware/auth_token.py", line 343, in __call__
    return self._forward_request(env, start_response, proxy_headers)
  File "/opt/stack/keystone/keystone/middleware/auth_token.py", line 576, in _forward_request
    return self.app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 279, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 477, in __call__
    request, **action_args)
  File "/opt/stack/glance/glance/common/wsgi.py", line 494, in dispatch
    return method(*args, **kwargs)
  File "/opt/stack/glance/glance/api/v1/images.py", line 530, in create
    image_meta = self._activate(req, image_id,...

[justinsb (justin-fathomdb)]

I think this is PEBKAC. If you specify a tenant, it does work!

Will close this and add a low-priority bug for when tenant not specified.

[justinsb (justin-fathomdb)]

Actually, I think it's probably better just to keep using this bug - no point rewriting those stack traces. I've changed the title to indicate the actual problem.

Priority should probably be Low, but I can't change it.

[Ewan Mellor (ewanmellor)]

I think that this is related to or the same as https://bugs.launchpad.net/glance/+bug/927870. On that bug, there's a discussion about whether this is by-design or not.

[Brian Waldon (bcwaldon)]

Ok, I'm not sure how we want this bug addressed, but I do want to point out that a lot has happened with glance/keystone integration over the past few weeks. Now it looks like we nicely handle the exception like this:

glance --os_username=demo --os_password=secrete --os_auth_url=http://192.168.27.100:5000/v2.0 add name=pants disk_format=ami container_format=ami < /opt/stack/glance/images/2571cbd6-3d2c-473d-adb4-833ef67003f8 Failed to add image. Got error:
Response from Keystone does not contain a Glance endpoint.
Note: Your image metadata may still be in the registry, but the image's status will likely be 'killed'.

What should we set as the criteria for fixing this bug?