ceph-volume lvm list is O(n^2)

Bug #1850754 reported by Dariusz Gadomski on 2019-10-31
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph
Unknown
Unknown
Ubuntu Cloud Archive
Undecided
Unassigned
Queens
Undecided
Unassigned
Rocky
Undecided
Unassigned
Stein
Undecided
Unassigned
Train
Undecided
Unassigned
ceph (Ubuntu)
Undecided
Dariusz Gadomski
Bionic
Undecided
Unassigned
Disco
Undecided
Unassigned
Eoan
Undecided
Unassigned

Bug Description

[Impact]
ceph-volume lvm calls take too much time and the time grows exponentially.
A simple call may take several minutes depending on the LV configuration.

[Test Case]
1. Create a setup with a large number of disks and logical volumes (e.g. 24)
2. Call ceph-volume lvm list for one of the disks.
3. Wait for the call to finish.

[Regression Potential]
Patch introduces LV caching, so changing LVs while ceph-volume is running may lead to race conditions.

[Other Info]
Original bug description:

 * Anything else you think is useful to include
 * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
 * and address these questions in advance

ceph-volume lvm list calls take very long time due to the fact it has been implemented with O(n^2). This leads to unproportionally long execution time taking under consideration the fact that no complex computation is needed.

Steps to reproduce:
1. Create a setup with a large number of disks and logical volumes (e.g. 24)
2. Call ceph-volume lvm list for one of the disks.
3. Wait for the call to finish.

Expected result:
Call is executed within seconds at most.

Actual result:
Call may take even minutes in some circumstances (large number of disks, some OSDs stopped).

Changed in ceph (Ubuntu):
assignee: nobody → Dariusz Gadomski (dgadomski)
no longer affects: ceph (Ubuntu Xenial)
Dariusz Gadomski (dgadomski) wrote :

SRU proposal for focal

Changed in ceph (Ubuntu):
status: New → Incomplete
status: Incomplete → New
description: updated
Dariusz Gadomski (dgadomski) wrote :

SRU proposal for eoan.

Dariusz Gadomski (dgadomski) wrote :

SRU proposal for bionic.

The attachment "focal_ceph_14.2.4-0ubuntu2.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
tags: added: sts
James Page (james-page) wrote :

disco is still supported so we'll need to apply the same fixes for that ubuntu series.

James Page (james-page) wrote :

@dgadomski - your patches suffer from the fact that there are a load of symlinks in the ceph source tree that mean the same change shows up in three places.

I've resolved this for the focal upload.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 14.2.4-0ubuntu2

---------------
ceph (14.2.4-0ubuntu2) focal; urgency=medium

  [ Dariusz Gadomski ]
  * d/p/issue37490.patch: Cherry pick fix to optimize LVM queries in
    ceph-volume, resolving performance issues in systems under heavy load
    or with large numbers of disks (LP: #1850754).

  [ James Page ]
  * d/p/issue40114.patch: Cherry pick endian fixes to resolve issues
    using Ceph on big-endian architectures such as s390x (LP: #1851290).

 -- Dariusz Gadomski <email address hidden> Wed, 06 Nov 2019 11:08:53 +0100

Changed in ceph (Ubuntu):
status: New → Fix Released
James Page (james-page) on 2019-11-22
Changed in cloud-archive:
status: New → Fix Released

Hello Dariusz, or anyone else affected,

Accepted ceph into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/14.2.4-0ubuntu0.19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ceph (Ubuntu Eoan):
status: New → Fix Committed
tags: added: verification-needed verification-needed-eoan
Dariusz Gadomski (dgadomski) wrote :

James, I assume that debdiff is not symlink-aware and it does produce the duplicated output. From what I see dpkg-source --commit produced a correct patch.

Is there a trick to make debdiff skip it? Or simply removing duplicated lines from debdiff output is the correct way to go?

I'd appreciate any hints. Thanks!

James Page (james-page) on 2019-11-28
Changed in cloud-archive:
status: Fix Released → Fix Committed
Timo Aaltonen (tjaalton) wrote :

Hello Dariusz, or anyone else affected,

Accepted ceph into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/12.2.12-0ubuntu0.18.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ceph (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic
James Page (james-page) wrote :

Hello Dariusz, or anyone else affected,

Accepted ceph into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-queens-needed
Corey Bryant (corey.bryant) wrote :

Hello Dariusz, or anyone else affected,

Accepted ceph into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-train-needed
Corey Bryant (corey.bryant) wrote :

Hello Dariusz, or anyone else affected,

Accepted ceph into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Timo Aaltonen (tjaalton) wrote :

Hello Dariusz, or anyone else affected,

Accepted ceph into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/13.2.7-0ubuntu0.19.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ceph (Ubuntu Disco):
status: New → Fix Committed
tags: added: verification-needed-disco
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.