[SRU] "radosgw-admin bucket limit check" has duplicate entries if bucket count exceeds 1000 (max_entries)

[nikhil kshirsagar]

The "radosgw-admin bucket limit check" command has a bug in octopus.

Since we do not clear the bucket list in RGWRadosUser::list_buckets() before asking for the next "max_entries", they are appended to the existing list and we end up counting the first ones again. This causes duplicated entries in the output of "ragodgw-admin bucket limit check"

This bug is triggered if bucket count exceeds 1000 (default max_entries).

------

$ dpkg -l | grep ceph
ii ceph 15.2.12-0ubuntu0.20.04.1 amd64 distributed storage and file system
ii ceph-base 15.2.12-0ubuntu0.20.04.1 amd64 common ceph daemon libraries and management tools
ii ceph-common 15.2.12-0ubuntu0.20.04.1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mds 15.2.12-0ubuntu0.20.04.1 amd64 metadata server for the ceph distributed file system
ii ceph-mgr 15.2.12-0ubuntu0.20.04.1 amd64 manager for the ceph distributed file system
ii ceph-mgr-modules-core 15.2.12-0ubuntu0.20.04.1 all ceph manager modules which are always enabled
ii ceph-mon 15.2.12-0ubuntu0.20.04.1 amd64 monitor server for the ceph storage system
ii ceph-osd 15.2.12-0ubuntu0.20.04.1 amd64 OSD server for the ceph storage system
ii libcephfs2 15.2.12-0ubuntu0.20.04.1 amd64 Ceph distributed file system client library
ii python3-ceph-argparse 15.2.12-0ubuntu0.20.04.1 amd64 Python 3 utility libraries for Ceph CLI
ii python3-ceph-common 15.2.12-0ubuntu0.20.04.1 all Python 3 utility libraries for Ceph
ii python3-cephfs 15.2.12-0ubuntu0.20.04.1 amd64 Python 3 libraries for the Ceph libcephfs library

$ sudo radosgw-admin bucket list | jq .[] | wc -l
5572
$ sudo radosgw-admin bucket limit check | jq .[].buckets[].bucket | wc -l
20572
$ sudo radosgw-admin bucket limit check | jq '.[].buckets[] | select(.bucket=="bucket_1095")'
{
"bucket": "bucket_1095",
"tenant": "",
"num_objects": 5,
"num_shards": 3,
"objects_per_shard": 1,
"fill_status": "OK"
}
{
"bucket": "bucket_1095",
"tenant": "",
"num_objects": 5,
"num_shards": 3,
"objects_per_shard": 1,
"fill_status": "OK"
}
{
"bucket": "bucket_1095",
"tenant": "",
"num_objects": 5,
"num_shards": 3,
"objects_per_shard": 1,
"fill_status": "OK"
}
{
"bucket": "bucket_1095",
"tenant": "",
"num_objects": 5,
"num_shards": 3,
"objects_per_shard": 1,
"fill_status": "OK"
}
{
"bucket": "bucket_1095",
"tenant": "",
"num_objects": 5,
"num_shards": 3,
"objects_per_shard": 1,
"fill_status": "OK"
}
{
"bucket": "bucket_1095",
"tenant": "",
"num_objects": 5,
"num_shards": 3,
"objects_per_shard": 1,
"fill_status": "OK"
}

------------------------------------------------------------------------------

Fix proposed through https://github.com/ceph/ceph/pull/43381

diff --git a/src/rgw/rgw_sal.cc b/src/rgw/rgw_sal.cc
index 2b7a313ed91..65880a4757f 100644
--- a/src/rgw/rgw_sal.cc
+++ b/src/rgw/rgw_sal.cc
@@ -35,6 +35,7 @@ int RGWRadosUser::list_buckets(const string& marker, const string& end_marker,
   RGWUserBuckets ulist;
   bool is_truncated = false;
   int ret;
+ buckets.clear();

   ret = store->ctl()->user->list_buckets(info.user_id, marker, end_marker, max,
                                         need_stats, &ulist, &is_truncated);

------------------------------------------------------------------------------

tested and verified the fix works:

$ sudo dpkg -l | grep ceph
ii ceph 15.2.14-0ubuntu0.20.04.3 amd64 distributed storage and file system
ii ceph-base 15.2.14-0ubuntu0.20.04.3 amd64 common ceph daemon libraries and management tools
ii ceph-common 15.2.14-0ubuntu0.20.04.3 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mds 15.2.14-0ubuntu0.20.04.3 amd64 metadata server for the ceph distributed file system
ii ceph-mgr 15.2.14-0ubuntu0.20.04.3 amd64 manager for the ceph distributed file system
ii ceph-mgr-modules-core 15.2.14-0ubuntu0.20.04.3 all ceph manager modules which are always enabled
ii ceph-mon 15.2.14-0ubuntu0.20.04.3 amd64 monitor server for the ceph storage system
ii ceph-osd 15.2.14-0ubuntu0.20.04.3 amd64 OSD server for the ceph storage system
ii libcephfs2 15.2.14-0ubuntu0.20.04.3 amd64 Ceph distributed file system client library
ii python3-ceph-argparse 15.2.14-0ubuntu0.20.04.3 amd64 Python 3 utility libraries for Ceph CLI
ii python3-ceph-common 15.2.14-0ubuntu0.20.04.3 all Python 3 utility libraries for Ceph
ii python3-cephfs 15.2.14-0ubuntu0.20.04.3 amd64 Python 3 libraries for the Ceph libcephfs library
ubuntu@crush-ceph-rgw01:~$ sudo apt-cache policy ceph
ceph:
Installed: 15.2.14-0ubuntu0.20.04.3
Candidate: 15.2.14-0ubuntu0.20.04.3

$ sudo radosgw-admin bucket list | jq .[] | wc -l
5572
$ sudo radosgw-admin bucket limit check | jq .[].buckets[].bucket | wc -l
5572
$ sudo radosgw-admin bucket limit check | jq '.[].buckets[] | select(.bucket=="bucket_1095")'
{
"bucket": "bucket_1095",
"tenant": "",
"num_objects": 5,
"num_shards": 3,
"objects_per_shard": 1,
"fill_status": "OK"
}

----------

[Impact]

duplicated bucket name entries appear in the customers outputs when they script the `radosgw-admin bucket limit check` commands.

To reproduce:

Create more than 1000 (default value of max_entries) buckets in a cluster, and run 'radosgw-admin bucket limit check'

Duplicated entries are seen in the output on Octopus. For example,

$ sudo radosgw-admin bucket list | jq .[] | wc -l
5572

$ sudo radosgw-admin bucket limit check | jq .[].buckets[].bucket | wc -l
20572

[Test case]

Create more than 1000 buckets in a cluster, then run the 'radosgw-admin bucket limit check' command. There should be no duplicated entries in the output. Below is correct output, where the numbers match.

$ sudo radosgw-admin bucket limit check | jq .[].buckets[].bucket | wc -l
5572

$ sudo radosgw-admin bucket list | jq .[] | wc -l
5572

[Where problems could occur]

The duplicate entries could end up causing admins or even scripts to assume that there are more buckets than there really are.

[Other Info]
- The patch was provided by Nikhil Kshirsagar (attached here)
- Upstream tracker: https://tracker.ceph.com/issues/52813
- Upstream PR: https://github.com/ceph/ceph/pull/43381
- Patched into Octopus upstream release.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "0001-rgw-clear-buckets-before-calling-list_buckets.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[nikhil kshirsagar (nkshirsagar)]

Attaching debdiff built on focal for octopus

[James Page (james-page)]

Fix included in Pacific so marking bug tasks for newer releases as Fix Released.

[James Page (james-page)]

Uploaded to focal UNAPPROVED for SRU team review.

[Robie Basak (racb)]

Hello nikhil, or anyone else affected,

Accepted ceph into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/15.2.14-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi, I can confirm that the packages from -propose will resolve the issue. See evidences bellow:

- While using 15.2.12, we can see the bug:

https://pastebin.ubuntu.com/p/jYg8x4Q7vR/

- After installing 15.2.14 from -proposed, the bug is resolved:

https://pastebin.ubuntu.com/p/mKFrTjddX7/

Thank you for your help addressing this issue.

Regards,
Fabio Martins

[Łukasz Zemczak (sil2100)]

Hey! Sadly, it looks like the riscv64 binary FTBFS for the new version. Since this arch is very flaky right now, I'll retry the build and see if it succeeds. If it doesn't, someone will have to take a look at it before we can release the package to focal-updates.

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 15.2.14-0ubuntu0.20.04.2

---------------
ceph (15.2.14-0ubuntu0.20.04.2) focal; urgency=medium

  * d/p/rgw_clear_buckets_before_calling_list_buckets.patch:
    Cherry-pick upstream change to fix duplicate entries with
    large buckets (LP: #1946211).

 -- Chris MacNaughton <email address hidden> Wed, 08 Dec 2021 09:31:08 +0000

[Chris MacNaughton (chris.macnaughton)]

Hello nikhil, or anyone else affected,

Accepted ceph into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi,

I've verified that the package from ussuri-proposed fixes the issue.

Without the patch:

https://pastebin.ubuntu.com/p/PFrkX7BkCT/

With the patch:

https://pastebin.ubuntu.com/p/yytzD3sjV9/

Cheers and thank you for fixing it.

- Fabio

[Corey Bryant (corey.bryant)]

ceph 15.2.14-0ubuntu0.20.04.2~cloud1 has been released to ussuri-updates and will publish shortly. Thanks for testing Fabio!