backport GC AIO to Luminous

Bug #1838858 reported by Jesse Williamson on 2019-08-03
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Undecided
Unassigned
Pike
High
James Page
Queens
High
James Page
ceph (Ubuntu)
Undecided
Jesse Williamson
Bionic
High
James Page

Bug Description

[Impact]
In RGW deployments with large deltas for deleted objects, the garbage collector will easily fall behind on cleanup of deleted objects eventually resulting in OSD devices hitting storage capacity.

[Test Case]
Deploy Ceph + RADOS Gateway
Create millions of objects in an iterative loop
Start deleting objects in parallel at a high rate
Storage capacity will gradually reduce over time as RGW garbage collection falls behind.

[Tricky to reproduce without significant scale of objects + deletion but proposed fix has been tested in the field.]

[Regression Potential]
Low - fix has been accepted into Ceph Luminous upstream and will form part of the next point release. Change to use AIO for GC is also in later Ceph releases.

[Original Bug Report]
SRU for backport of the GC AIO feature to Ceph Luminous.

https://github.com/ceph/ceph/pull/28784

James Page (james-page) wrote :

This will be part of 12.2.13 - what's the urgency around this SRU?

We can cherry pick the fix OR wait for the point release.

Changed in ceph (Ubuntu):
status: New → Invalid
Changed in ceph (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → High
Changed in cloud-archive:
status: New → Invalid
James Page (james-page) on 2019-08-29
description: updated
description: updated
James Page (james-page) on 2019-08-29
Changed in ceph (Ubuntu Bionic):
assignee: nobody → James Page (james-page)
James Page (james-page) on 2019-08-30
Changed in ceph (Ubuntu Bionic):
status: Triaged → In Progress

Hello Jesse, or anyone else affected,

Accepted ceph into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/12.2.12-0ubuntu0.18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ceph (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Jesse Williamson (chardan) wrote :

There is an additional related upstream PR that we want to include at least parts of:
    https://github.com/ceph/ceph/pull/26601/files
    https://tracker.ceph.com/issues/38454

Jesse Williamson (chardan) wrote :
Dan Hill (hillpd) wrote :

The issues and fixes discuss in comments #3 and #4 are being tracked under lp#1843085.

They have AIO GC as a dependency, but are not required for this SRU.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.