Glance will leak staging data

Bug #1913625 reported by Dan Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Undecided
Dan Smith

Bug Description

In various situations, glance will leak (potentially very large) temporary files in the staging store.

One example is doing a web-download import, where glance initially downloads the image to its staging store. If the worker doing that activity crashes, loses power, etc, the user may delete the image and try again on another worker. When the crashed worker resumes, the staging data will remain but nothing will ever clean it up.

Another example would be a misconfigured glance that uses local staging directories, but glance-direct is used, where the user stages data, and then deletes the image from another worker.

Even in a situation where shared staging is properly configured, a failure to access the staging location during the delete call will result in the image being deleted, but the staging file not being purged.

IMHO, glance workers should clean their staging directories at startup, purging any data that is attributable to a previous image having been deleted.

Another option is to add a store location for each staged image, and make sure the scrubber can clean those things from the staging directory periodically (this requires also running the scrubber on each node, which may not be common practice currently).

Revision history for this message
Dan Smith (danms) wrote :

Marking this as invalid as the glance team thinks this is not a bug.

Changed in glance:
status: New → Invalid
Dan Smith (danms)
Changed in glance:
status: Invalid → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 21.0.0.0b3

This issue was fixed in the openstack/glance 21.0.0.0b3 development milestone.

Changed in glance:
status: Confirmed → Fix Released
assignee: nobody → Dan Smith (danms)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to glance (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/glance/+/785551
Committed: https://opendev.org/openstack/glance/commit/54060e8b72b048d9a6a2ff2ca01a5fc18b6eafec
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 54060e8b72b048d9a6a2ff2ca01a5fc18b6eafec
Author: Dan Smith <email address hidden>
Date: Tue Feb 23 18:59:07 2021 -0800

    Make functional tests set node_staging_uri

    Currently it is not possible to configure the staging directory
    URI of the functional workers. We need to be able to do that in order
    to enable the stage cleaning behavior. Right now, they're all sharing
    /tmp/staging, which will cause workers to conflict while running in
    parallel. This causes them to use their private test directory,
    which may also help some other spurious failures due to interaction.

    Related-Bug: #1913625
    Change-Id: Ic2ac2a528206c50c38e948a096daf9eb8e5eb715
    (cherry picked from commit e9852fb6251c655f0e384c6b0b8677bd556f17a1)
    (cherry picked from commit 8e216b4913e8b842cab983c100165949bed7def2)

tags: added: in-stable-ussuri
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers