Glance

If db deadlock occurs for some reason while deleting an image, no one can delete the image any more

Bug #1378215 reported by Ankit Agrawal
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
High
Ankit Agrawal
Glance 2015.1.0 "kilo"
Icehouse
Fix Released
Undecided
Unassigned
Glance 2014.1.4
Juno
Fix Released
Undecided
Unassigned
Glance 2014.2.1

Bug Description

Glance api returns 500 Internal Server Error, if db deadlock occurs in glance-registry for some reason while deleting an image.
The image 'status' is set to deleted and 'deleted' is set to False. As deleted is still False, the image is visible in image list but it can not be deleted any more.

If you try to delete this image it will raise 404 (Not Found) error for V1 api and 500 (HTTPInternalServerError) for V2 api.

Note:
To reproduce this issue I've explicitly raised "db_exception.DBDeadlock" exception from "_image_child_entry_delete_all" method under "\glance\db\sqlalchemy\api.py".

glance-api.log
--------------
2014-10-06 00:53:10.037 6827 INFO glance.registry.client.v1.client [2b47d213-6f80-410f-9766-dc80607f0224 7e7c3a413f184dbcb9a65404dbfcc0f0 309c5ff4082c423
1bcc17d8c55c83997 - - -] Registry client request DELETE /images/f9f8a40d-530b-498c-9fbc-86f29da555f4 raised ServerError
2014-10-06 00:53:10.045 6827 INFO glance.wsgi.server [2b47d213-6f80-410f-9766-dc80607f0224 7e7c3a413f184dbcb9a65404dbfcc0f0 309c5ff4082c4231bcc17d8c55c83997 - - -] Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 433, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/osprofiler/web.py", line 106, in __call__
    return request.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 748, in __call__
    return self._call_app(env, start_response)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 684, in _call_app
    return self._app(env, _fake_start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 206, in __call__
    return app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 683, in __call__
    request, **action_args)
  File "/opt/stack/glance/glance/common/wsgi.py", line 707, in dispatch
    return method(*args, **kwargs)
  File "/opt/stack/glance/glance/common/utils.py", line 449, in wrapped
    return func(self, req, *args, **kwargs)
  File "/opt/stack/glance/glance/api/v1/images.py", line 1058, in delete
    registry.delete_image_metadata(req.context, id)
  File "/opt/stack/glance/glance/registry/client/v1/api.py", line 175, in delete_image_metadata
    return c.delete_image(image_id)
  File "/opt/stack/glance/glance/registry/client/v1/client.py", line 207, in delete_image
    res = self.do_request("DELETE", "/images/%s" % image_id)
  File "/opt/stack/glance/glance/registry/client/v1/client.py", line 130, in do_request
    'exc_name': exc_name})
  File "/opt/stack/glance/glance/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/glance/glance/registry/client/v1/client.py", line 115, in do_request
    **kwargs)
  File "/opt/stack/glance/glance/common/client.py", line 68, in wrapped
    return func(self, *args, **kwargs)
  File "/opt/stack/glance/glance/common/client.py", line 388, in do_request
    headers=copy.deepcopy(headers))
  File "/opt/stack/glance/glance/common/client.py", line 85, in wrapped
    return func(self, method, url, body, headers)
  File "/opt/stack/glance/glance/common/client.py", line 546, in _do_request
    raise exception.ServerError()
ServerError: The request returned 500 Internal Server Error.
2014-10-06 00:53:10.055 6827 INFO glance.wsgi.server [2b47d213-6f80-410f-9766-dc80607f0224 7e7c3a413f184dbcb9a65404dbfcc0f0 309c5ff4082c4231bcc17d8c55c83997 - - -] 10.69.4.178 - - [06/Oct/2014 00:53:10] "DELETE /v1/images/f9f8a40d-530b-498c-9fbc-86f29da555f4 HTTP/1.1" 500 139 0.239514

Ankit Agrawal (ankitagrawal)
Changed in glance:
assignee: nobody → Ankit Agrawal (ankitagrawal)
Dolph Mathews (dolph)
Changed in glance:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/128301

Changed in glance:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/128301
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=dae0fa8f1071be41d5541e9a5254abea48f5ba36
Submitter: Jenkins
Branch: master

commit dae0fa8f1071be41d5541e9a5254abea48f5ba36
Author: ankitagrawal <email address hidden>
Date: Tue Oct 7 07:36:10 2014 -0700

    Can not delete images if db deadlock occurs

    Glance api returns 500 Internal Server Error, if db deadlock occurs
    in glance-registry for some reason while deleting an image.

    Added retry on image_destroy when db deadlock occures which will again
    try to delete the image from database.

    Closes-bug: 1378215
    Change-Id: Ifad403e363daf368e846b5b6838432a7bedbe81a

Changed in glance:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/130009

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/130029

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/juno)

Reviewed: https://review.openstack.org/130009
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=41e143b2172a36e4f49bc7e3a1ed97ee8477afb9
Submitter: Jenkins
Branch: stable/juno

commit 41e143b2172a36e4f49bc7e3a1ed97ee8477afb9
Author: ankitagrawal <email address hidden>
Date: Tue Oct 7 07:36:10 2014 -0700

    Can not delete images if db deadlock occurs

    Glance api returns 500 Internal Server Error, if db deadlock occurs
    in glance-registry for some reason while deleting an image.

    Added retry on image_destroy when db deadlock occures which will again
    try to delete the image from database.

    Closes-bug: 1378215
    Change-Id: Ifad403e363daf368e846b5b6838432a7bedbe81a
    (cherry picked from commit dae0fa8f1071be41d5541e9a5254abea48f5ba36)

tags: added: in-stable-juno
Erno Kuvaja (jokke)
tags: added: icehouse-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/icehouse)

Reviewed: https://review.openstack.org/130029
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=4b5cb74e5120decd618399d642ebf06ad45d26a6
Submitter: Jenkins
Branch: stable/icehouse

commit 4b5cb74e5120decd618399d642ebf06ad45d26a6
Author: ankitagrawal <email address hidden>
Date: Tue Oct 7 07:36:10 2014 -0700

    Can not delete images if db deadlock occurs

    Glance api returns 500 Internal Server Error, if db deadlock occurs
    in glance-registry for some reason while deleting an image.

    Added `_retry_on_deadlock` decorator from Nova since we don't depend
    on retrying library in Icehouse. Used this decorator on image_destroy
    when db deadlock occures during image delete, which will again try to
    delete the image from database.

    Conflicts:
            glance/tests/unit/test_db.py

    Note: In this patch, I've made few additional changes to add retrying
    logic because retrying library is not available in Icehouse.

    Closes-bug: 1378215
    Change-Id: Ifad403e363daf368e846b5b6838432a7bedbe81a
    (cherry picked from commit dae0fa8f1071be41d5541e9a5254abea48f5ba36)

tags: added: in-stable-icehouse
Louis Taylor (kragniz)
Changed in glance:
milestone: none → kilo-1
Thierry Carrez (ttx)
Changed in glance:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in glance:
milestone: kilo-1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.