luminous: bluestore rocksdb max_background_compactions regression in 12.2.13

Bug #1871820 reported by James Page on 2020-04-09
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ceph
Unknown
Unknown
ceph (Ubuntu)
Undecided
Unassigned
Bionic
High
James Page

Bug Description

There is a potential regression in the rocksdb max_background_compactions change in the 12.2.13 point release that is in bionic-proposed.

https://tracker.ceph.com/issues/44937 was raised as the rocksdb version in the bionic ceph source does not contain:

  https://github.com/facebook/rocksdb/pull/3926

which ensures that any overlap between background compactions is detected and handled.

We either need to revert to a single background compaction as the default setting, or pick the fix into the rocksdb source in src:ceph/bionic

James Page (james-page) on 2020-04-09
Changed in ceph (Ubuntu):
status: New → Invalid
James Page (james-page) on 2020-04-09
Changed in ceph (Ubuntu Bionic):
importance: Undecided → High
James Page (james-page) on 2020-05-19
Changed in ceph (Ubuntu Bionic):
assignee: nobody → James Page (james-page)
status: New → In Progress
James Page (james-page) wrote :

I'm going to revert the default configuration change - this is a minumal, least risk approach to this issue.

Its possible that we might want to pick the rocksdb fixes as well but that diverges Ubuntu from upstream in a core part of ceph which I'm not 100% comfortable with.

Feedback on the upstream tracker bug indicates there is a lot of evidence that increasing concurrency has not resulted in data loss.

Nick Niehoff (nniehoff) on 2020-05-26
tags: added: sts

Hello James, or anyone else affected,

Accepted ceph into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/12.2.13-0ubuntu0.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ceph (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
James Page (james-page) wrote :

standard verification of bionic-proposed passed:

======
Totals
======
Ran: 92 tests in 709.8995 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 989.9518 sec.

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
James Page (james-page) wrote :

xenial-queens/proposed

======
Totals
======
Ran: 92 tests in 707.9748 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 897.0776 sec.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 12.2.13-0ubuntu0.18.04.2

---------------
ceph (12.2.13-0ubuntu0.18.04.2) bionic; urgency=medium

  * d/p/bug1871820.patch: Revert change in default concurrency for
    rocksdb background compactions to avoid potential data loss
    (LP: #1871820).

ceph (12.2.13-0ubuntu0.18.04.1) bionic; urgency=medium

  * New upstream point release (LP: #1861793).
  * d/p/bug1847544.patch,ceph-volume-wait-for-lvs.patch,dont-validate-fs-
    caps-on-authorize.patch,issue37490.patch,issue38454.patch,rgw-gc-use-
    aio.patch: Drop, all included in upstream release.
  * d/p/*: Refresh as needed.
  * d/p/bug1804261.patch: Cherry pick fix to ensure that ceph-volume
    tries and interval environment variables are converted to int
    (LP: #1804261).

 -- James Page <email address hidden> Tue, 19 May 2020 08:40:13 +0100

Changed in ceph (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.