blob stuck in saving status

Bug #1742729 reported by idan Narotzki
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Glare
High
Kushal Agrawal

Bug Description

blob stuck in "saving" status.

glare upload results with: INFO - ERROR (app) HTTPConflict (HTTP 409)

When going to glare_artifact_blobs (after more than 12 hours) we can find blob in "saving" status, and NULL values for url, sha1, key_name, sha256.

When Trying to delete the artifact we get 409 (as expected):

{"explanation": "There was a conflict when trying to complete your request.", "code": 409, "error": {"message": "You cannot change artifact status if it has uploading blobs.", "type": "Conflict"}, "title": "Conflict"}

ERROR (app) HTTPConflict (HTTP 409)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 137, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/glareclient/osc/v1/artifacts.py", line 344, in take_action
    type_name=parsed_args.type_name)
  File "/usr/lib/python2.7/site-packages/glareclient/v1/artifacts.py", line 239, in delete
    self.http_client.delete(url)
  File "/usr/lib/python2.7/site-packages/glareclient/common/http.py", line 275, in delete
    return self.request(url, "DELETE", **kwargs)
  File "/usr/lib/python2.7/site-packages/glareclient/common/http.py", line 227, in request
    raise exc.from_response(resp)
HTTPConflict: HTTPConflict (HTTP 409)
Traceback (most recent call last):
  File "/usr/bin/glare", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/glareclient/shell.py", line 336, in main
    return GlareShell().run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 137, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/glareclient/osc/v1/artifacts.py", line 344, in take_action
    type_name=parsed_args.type_name)
  File "/usr/lib/python2.7/site-packages/glareclient/v1/artifacts.py", line 239, in delete
    self.http_client.delete(url)
  File "/usr/lib/python2.7/site-packages/glareclient/common/http.py", line 275, in delete
    return self.request(url, "DELETE", **kwargs)
  File "/usr/lib/python2.7/site-packages/glareclient/common/http.py", line 227, in request
    raise exc.from_response(resp)
glareclient.exc.HTTPConflict: HTTPConflict (HTTP 409)

Revision history for this message
idan Narotzki (idanaroz) wrote :
description: updated
Revision history for this message
Mike Fedosin (mfedosin) wrote :

Usually the service automatically deletes a blob instance if there was an error during uploading process. For sure we can't guarantee this behavior always, because in some cases mysql db doesn't respond in time.

I agree that we must prevent such situations and allow users to delete blobs with 'saving' status.
But we must do it right and exclude the situations when malicious users will reupload their data, and therefore clog the storage.

My proposal is to add a new policy for deleting blobs with saving status and let only operators perform this action.

Changed in glare:
status: New → Confirmed
importance: Undecided → High
Changed in glare:
assignee: nobody → Kushal Agrawal (kushal2605)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glare (master)

Fix proposed to branch: master
Review: https://review.openstack.org/613023

Changed in glare:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glare (master)

Reviewed: https://review.openstack.org/613023
Committed: https://git.openstack.org/cgit/openstack/glare/commit/?id=80c2ec1932bb4e15f080e276d93b24feabf39c72
Submitter: Zuul
Branch: master

commit 80c2ec1932bb4e15f080e276d93b24feabf39c72
Author: Kushal Agrawal <email address hidden>
Date: Wed Oct 24 19:10:07 2018 +0530

    Removed tpool.execute to prevent upload failure

    As of now there no solution to the issue where thread is getting
    stuck in eventlet.
    Few other similar incidents and without proper solution:
    https://bugs.launchpad.net/cinder/+bug/1694509
    https://github.com/eventlet/eventlet/issues/432
    https://github.com/eventlet/eventlet/issues/492
    https://github.com/eventlet/eventlet/issues/395

    Change-Id: Ib278780ccb20b9cbef50f54ba1a1ad33761c8002
    closes-bug: #1742729

Changed in glare:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glare (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/642255

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glare (master)

Fix proposed to branch: master
Review: https://review.openstack.org/642259

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glare (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/642260

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glare (master)

Change abandoned by Idan Narotzki (<email address hidden>) on branch: master
Review: https://review.openstack.org/642259

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glare (stable/queens)

Reviewed: https://review.openstack.org/642260
Committed: https://git.openstack.org/cgit/openstack/glare/commit/?id=7112503f67ec293648ae8c440bac4fc107407e7e
Submitter: Zuul
Branch: stable/queens

commit 7112503f67ec293648ae8c440bac4fc107407e7e
Author: idanaroz <email address hidden>
Date: Sun Mar 10 09:23:37 2019 +0000

    Removed tpool.execute to prevent upload failure

    As of now there no solution to the issue where thread is getting
    stuck in eventlet.
    Few other similar incidents and without proper solution:
    https://bugs.launchpad.net/cinder/+bug/1694509
    https://github.com/eventlet/eventlet/issues/432
    https://github.com/eventlet/eventlet/issues/492
    https://github.com/eventlet/eventlet/issues/395

    Originally was taken from: https://review.openstack.org/#/c/613023/1

    Change-Id: Ic924f0ef0cb632b2439dfb7d1092bebf54adb863
    closes-bug: #1742729

tags: added: in-stable-queens
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.