Glance

500 error on image-download if image is not present at any location

Bug #1765309 reported by Abhishek Kekane
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Invalid
Undecided
Abhishek Kekane

Bug Description

As of now for downloading the image it iterates through all the locations and if data (image) is not present at any of the location in the end it Logs the error message as "Glance tried all active locations to get data for image <image-id> but all have failed." Further it fails with 500 error.

This issue can occur either if image is not available in any of the location or any error occured during the download process.

NOTE:
To reproduce this issue I have manually added 'continue' statement at [1] to assume that image is not available at any of the location.
[1] https://github.com/openstack/glance/blob/master/glance/location.py#L470

Steps to reproduce:
1. Modify glance/location.py as mentioned in NOTE
2. Restart glance-api service
3. Download image using:
   $ glance image-download <image-id> --file downloaded_image

Expected Result:
Image data should be downloaded and saved in 'downloaded_image' file

Actual Result:
glance image-download 780ffe26-b95a-4b6f-b2c0-c38803bed73a --file abcdUnable to download image '780ffe26-b95a-4b6f-b2c0-c38803bed73a'. (502 Proxy Error: Proxy Error: The proxy server received an invalid: response from an upstream server.: The proxy server could not handle the request GET&nbsp;/image/v2/images/780ffe26-b95a-4b6f-b2c0-c38803bed73a/file.: Reason: Error reading from remote server: Apache/2.4.18 (Ubuntu) Server at 192.168.0.6 Port 80 (HTTP 502))

Glance API logs:
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance.location [None req-cfca70d4-d9a7-495f-9762-0fb2a8eee769 admin admin] Glance tried all active locations to get data for image 780ffe26-b95a-4b6f-b2c0-c38803bed73a but all have failed.
Apr 19 06:37:58 signature <email address hidden>[5419]: CRITICAL glance [None req-cfca70d4-d9a7-495f-9762-0fb2a8eee769 admin admin] Unhandled error: TypeError: 'ImageProxy' object is not callable
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance Traceback (most recent call last):
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance resp = self.call_func(req, *args, **self.kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return self.func(req, *args, **kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance response = req.get_response(self.application)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance application, catch_exc_info=False)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance app_iter = application(self.environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance resp = self.call_func(req, *args, **self.kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return self.func(req, *args, **kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance response = req.get_response(self.application)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance application, catch_exc_info=False)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance app_iter = application(self.environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance resp = self.call_func(req, *args, **self.kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return self.func(req, *args, **kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance response = req.get_response(self.application)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance application, catch_exc_info=False)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance app_iter = application(self.environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance resp = self.call_func(req, *args, **self.kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return self.func(req, *args, **kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/opt/stack/glance/glance/common/wsgi.py", line 809, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance response = req.get_response(self.application)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance application, catch_exc_info=False)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance app_iter = application(self.environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance resp = self.call_func(req, *args, **self.kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return self.func(req, *args, **kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/osprofiler/web.py", line 119, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return request.get_response(self.application)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance application, catch_exc_info=False)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance app_iter = application(self.environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance resp = self.call_func(req, *args, **self.kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return self.func(req, *args, **kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 337, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance response = req.get_response(self._app)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance application, catch_exc_info=False)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance app_iter = application(self.environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance resp = self.call_func(req, *args, **self.kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return self.func(req, *args, **kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/opt/stack/glance/glance/common/wsgi.py", line 809, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance response = req.get_response(self.application)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance application, catch_exc_info=False)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance app_iter = application(self.environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance resp = self.call_func(req, *args, **self.kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return self.func(req, *args, **kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/opt/stack/glance/glance/common/wsgi.py", line 809, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance response = req.get_response(self.application)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance application, catch_exc_info=False)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance app_iter = application(self.environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance resp = self.call_func(req, *args, **self.kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return self.func(req, *args, **kwargs)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/opt/stack/glance/glance/common/wsgi.py", line 809, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance response = req.get_response(self.application)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance application, catch_exc_info=False)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance app_iter = application(self.environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 216, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return app(environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return resp(environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance response = self.app(environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return resp(environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance return resp(environ, start_response)
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance TypeError: 'ImageProxy' object is not callable
Apr 19 06:37:58 signature <email address hidden>[5419]: ERROR glance

Abhishek Kekane (abhishek-kekane)
Changed in glance:
assignee: nobody → Abhishek Kekane (abhishek-kekane)
Revision history for this message
Erno Kuvaja (jokke) wrote :

So if you look at the code there that you modified to introduce this bug, you are bypassing the try..catch block that will set the exception that would be raised at the failure causing it trying to raise None which obviously will fail.

Changed in glance:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.