Building Ceph packages with RelWithDebInfo

[Ponnuvel Palaniyappan]

[Impact]
This is to build the Ceph packages (UCA) with 'RelWithDebInfo' build option (CMAKE_BUILD_TYPE [0]) which can be supplied to the build script do_cmake.sh [0].

Currently, UCA builds default to 'Debug' [1] builds. The UCA's build process does debug build just for the rocksdb component - because the upstream builds (CMakeLists.txt) left it to individual projects using rocksdb. So UCA's build used the default (debug build). Upstream has since
updated this to do release build (with RelWithDebInfo option) [2].

In general, gcc's -O2 [3] (which will be enabled by RelWithDebInfo) build should generate better code as well as there's a specific case for Ceph [4] that it improves performance.

Building rocksdb with optimizations enabled improves the performance and
rocksdb being an important part of OSD and MON components of Ceph, this
enables better performance of Ceph cluster. Comments #17, #18, & #19
show relevant test results.

[Test Case]
Standard ceph-backed OpenStack regression will be run, consisting of juju-deployed OpenStack+ceph and tempest regression testing against the deployment.

[Regression Potential]
As Ponnuvel mentioned, the size of binaries will be slightly bigger and also the build/compilation will be longer.

There are no upstream bugs for this. rocksdb is one of the submodules in Ceph and how each of the submodule gets built is left to end-users. The bug [5] where this change was done in upstream is not relevant directly; this change was done as part of a different bug fix.

[0] https://github.com/ceph/ceph/blob/master/do_cmake.sh

[1] https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html

[2] https://github.com/facebook/rocksdb/pull/5916

[3] https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

[4] https://bugs.gentoo.org/show_bug.cgi?id=733316

[5] https://github.com/facebook/rocksdb/issues/2445

[Ponnuvel Palaniyappan (pponnuvel)]

Another thing to note is that the size of binaries will be slightly bigger and also the build/compilation will be longer.

[Ponnuvel Palaniyappan (pponnuvel)]

Attaching relevant chat transcript with a Ceph dev on RelWithDebInfo.

[Trent Lloyd (lathiat)]

Are we sure it's actually building as Debug?

At least 15.2.3 on focal seems to build with RelWithDebugInfo.. I see -O2 .. only do_cmake.sh had logic for this (it would set Debug if a .git directory exists) but the debian rules file doesn't seem to use that script but cmake directly.

[Tyler Stachecki (tstachecki)]

Checked out ceph-15.2.3 source on a up-to-date focal VM.

When running dpkg-buildpackage, CMakeCache.txt gets generated as expected. However, the generated CMAKE_BUILD_TYPE is "None".

A bit odd, but going a step further, one can grep for CMAKE_C_FLAGS in that same file and see something to the effect of:
CMAKE_C_FLAGS_DEBUG:STRING=<sensible compiler settings for debug builds>
CMAKE_C_FLAGS_NONE:STRING=
...
...
CMAKE_C_FLAGS_RELWITHDEBINFO:STRING=<sensible compiler settings for release builds>

It seems, though, that since "None" is used, all of those are actually ignored and
CMAKE_C_FLAGS:STRING=<compiler flags from dpkg-buildpackage>
is used

That particular set of flags seems to be missing -DNDEBUG. It's worth noting that -DNDEBUG is defined within CMAKE_C_FLAGS_RELWITHDEBINFO:STRING and including it may result in a more optimized build.

This is not quite the Gentoo issue as Trent alluded.

[Tyler Stachecki (tstachecki)]

Oof - so I hacked up debian/rules a little so that "RelWithDebInfo" is used instead of "None" for CMAKE_BUILD_TYPE. I then ^C'd the dpkg-buildpackage after it started compiling the first few source files and observed that:

  * When doing make V=1 VERBOSE=1 in obj-x86_64-linux-gnu/src/rocksdb *without* any changes, the source is compiled with *neither* -g nor -O2 -DNDEBUG.

  * When doing make V=1 VERBOSE=1 in obj-x86_64-linux-gnu/src/rocksdb *with* my changes, rocksdb is built with *both* -g and -O2 -DNDEBUG.

-O2 -DNDEBUG should result in significantly better performance/lower utilization for metadata operations...

[Ponnuvel Palaniyappan (pponnuvel)]

Looking some of the recent build logs, most of the Ceph packages are in fact compiled with -O2 and -g as Trent alluded to. Only rocksdb is compiled without them as Tyler noted.

However, the upstream rocksdbhas been changed to honour CMAKE_BUILD_TYPE recently [2]. So it's only the rocksdb in UCA that appear to be old; src/rocksdb/CMakeLists.txt [3] is different from the one in UCA [5]. So I believe that source rocksdb needs to be updated to use latest version and that'd naturally build rocksdb RelWithDebInfo, too, as upstream already has the changes for that.

[0] https://launchpadlibrarian.net/475127725/buildlog_ubuntu-focal-amd64.ceph_15.2.1-0ubuntu1_BUILDING.txt.gz

[1] https://launchpadlibrarian.net/494025636/buildlog_ubuntu-groovy-amd64.ceph_15.2.3-0ubuntu3_BUILDING.txt.gz

[2] https://github.com/facebook/rocksdb/pull/5916/files

[3] https://github.com/ceph/rocksdb/blob/f86d378860f399bf595bd1135f39f03a9eaca59f/CMakeLists.txt

[4] https://git.launchpad.net/ubuntu/+source/ceph/tree/src/rocksdb/CMakeLists.txt

[5] https://git.launchpad.net/ubuntu/+source/ceph/tree/src/rocksdb/CMakeLists.txt

[Tyler Stachecki (tstachecki)]

Good to know it's just rocksdb!

Has Canonical also considered compiling the source code with -DNDEBUG to align with what appears to be the upstream default set of RelWithDebInfo flags? The Ceph source code is quasi-judicious about its use of `ceph_assert`, which is effectively compiled out when -DNDEBUG is specified.

I am unsure of the performance benefit of this in particular.

[Corey Bryant (corey.bryant)]

@Ponnuvel, I'm not fully familiar with how ceph deals with rocksdb code so this is mostly observations from my end.

Specifically looking at 15.2.4 for groovy. In https://github.com/ceph/ceph/tree/v15.2.4/src, it appears to pick up rocksdb from a specific commit (rocksdb @ e24b452) that exists in ceph's forked rocksdb repo (https://github.com/ceph/rocksdb @ ceph-octopus-v5.8-1436). The commit for ac304ad doesn't appear to exist in the forked repo, but does exist in the fb repo at https://github.com/facebook/rocksdb/.

I'm not sure what the process is but it seems that ac304ad needs to get into ceph's forked repo. Perhaps a cherry-pick with a PR is enough. James probably knows the process.

[Ponnuvel Palaniyappan (pponnuvel)]

@Tyler, Canonical's Ceph builds don't deal with build options as such (like -DNDEBUG) but uses the
upstream provided CMakeLists.txt files. So if we build with RelWithDebInfo, it should automatically use -DNDEBUG, too. The question was whether our build uses RelWithDebInfo -- which turns out to be case for all but rocksdb.

[Ponnuvel Palaniyappan (pponnuvel)]

@Corey Bryant, I think Ceph simply specifies rocksdb version to be used in the submodule information.

1. I checked out the tag v15.2.3 (upstream): git checkout tags/v15.2.3 -b 15.2.3
  (which src/rocksdb is an empty directory)

2. Initialize submodule indexes: git submodule update --init

3. The commit HEAD of rocksdb that Ceph 15.2.3 uses:
 $ git submodule status | grep rocksdb
 4c736f177851cbf9fb7a6790282306ffac5065f8 rocksdb (v5.8-1436-g4c736f177)

I think this is how Ceph upstream "matches" the versions of various submodules (rocksdb is one of them) for a given Ceph release.

Comparing the above cloned rocksdb matches exactly with UCA's repo. So I think the UCA's repo is upto date and there's no such problem of using "older" version of rocksdb as I speculated.

So the question is whether we want to cherry-pick build change from upstream so as compile rocksdb with RelWithDebInfo or wait until upstream defaults to a newer rocksdb submodule that does build with RelWithDebInfo by default.

P.S: One thing I don't know is what mechanism upstream follow before they decide to use
a newer version of rocksdb. For example, the rocksdb version 6.8.1 contains the change
to use 'RelWithDebInfo' - which was released/tagged on April 17th [0] as part of Ceph repo. But the Ceph 15.2.3 release [1] that was done on May 29 didn't include that but uses an older version of rocksdb (as noted initially in the comment). My guess is that upstream always integrate recent rocksdb commits in Ceph master for testing/development purposes. But when it comes to release, they choose the one that they think is stable.

[0] https://github.com/ceph/rocksdb/releases

[1] https://github.com/ceph/ceph/releases

[Corey Bryant (corey.bryant)]

@Ponnuvel, Thanks for the details.

4c736f177851cbf9fb7a6790282306ffac5065f8 (the current git submodule hash for octopus) is in origin/ceph-octopus-v5.8-1435. ac304adf46ff073a943d1ab2f6de0623c8ad6d86 (the RelWithDebInfo patch) is in origin/ceph-pacific-v6.8.1-1. So it seems we'll pick this fix up naturally when the ceph pacific package is uploaded, but pacific isn't due to release until March 2021.

Ideally we would pick this up from upstream rather than cherry-picking to the package. I'm not sure what the process is for cherry-picking patches to the ceph-octopus-v5.8-1435 rocksdb branch is upstream. The last update is from Apr 2019 so obviously it remains quite stable. I imagine they have a process for accepting cherry-picks into stable branches but we'd have to see if this would qualify.

[Corey Bryant (corey.bryant)]

@Ponnuvel, if I cherry pick this to the package for groovy can you give it a test?

[Ponnuvel Palaniyappan (pponnuvel)]

@Corey, Yes, I am happy to test this for Groovy.

[Ponnuvel Palaniyappan (pponnuvel)]

An update:

I have built rocksdb with & without RelWithDebInfo. Some differences are:

 - the exact set of additional flags used with RelWithDebInfo are: -O2 -NDEBUG -fno-omit-frame-pointer -momit-leaf-frame-pointer.

 - the size of rocksdb shared library (librocksdb.so): default-build = 156M, with-RelWithDebInfo = 235M (In the case of Ceph binaries, this shared library isn't used separately but compiled into ceph-osd and others. Similar comparison of ceph-osd binary size stripped: default-build = 16M, with-RelWithDebInfo = 25M).

- There are about ~5700 asserts in rocksdb which would disappear with RelWithDebInfo.

I am doing further performance benchmarks between the two versions and will update once done.

[Corey Bryant (corey.bryant)]

@Ponnuvel,

I have a package built for groovy with RelWithDebInfo enabled. Can you give it a test?

sudo add-apt-repository ppa:corey.bryant/groovy-victoria

The individual building of rocksdb files doesn't give off much information in the package builds to see compiler flags set. For example:
[ 0%] Building CXX object CMakeFiles/rocksdb.dir/db/compacted_db_impl.cc.o

But comparing the cmake call shows that we do have RelWithDebInfo set for rocksdb:

== Before RelWithDebInfo was enabled ==
cd /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/rocksdb && /usr/bin/cmake -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DWITH_GFLAGS=OFF -DCMAKE_PREFIX_PATH= -DCMAKE_CXX_COMPILER=/usr/bin/c++ -DWITH_SNAPPY=TRUE -DWITH_LZ4=TRUE -DLZ4_INCLUDE_DIR=/usr/include -DLZ4_LIBRARIES=/usr/lib/x86_64-linux-gnu/liblz4.so -DWITH_ZLIB=TRUE -DPORTABLE=ON -DCMAKE_AR=/usr/bin/ar -DCMAKE_BUILD_TYPE=None -DFAIL_ON_WARNINGS=OFF -DUSE_RTTI=1 "-GUnix Makefiles" -DCMAKE_C_FLAGS=-Wno-stringop-truncation "-DCMAKE_CXX_FLAGS='-Wno-deprecated-copy -Wno-pessimizing-move'" "-GUnix Makefiles" /<<PKGBUILDDIR>>/src/rocksdb

== After RelWithDebInfo was enabled ==
cd /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/rocksdb && /usr/bin/cmake -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DWITH_GFLAGS=OFF -DCMAKE_PREFIX_PATH= -DCMAKE_CXX_COMPILER=/usr/bin/c++ -DWITH_SNAPPY=TRUE -DWITH_LZ4=TRUE -DLZ4_INCLUDE_DIR=/usr/include -DLZ4_LIBRARIES=/usr/lib/x86_64-linux-gnu/liblz4.so -DWITH_ZLIB=TRUE -DPORTABLE=ON -DCMAKE_AR=/usr/bin/ar -DCMAKE_BUILD_TYPE=RelWithDebInfo -DFAIL_ON_WARNINGS=OFF -DUSE_RTTI=1 "-GUnix Makefiles" -DCMAKE_C_FLAGS=-Wno-stringop-truncation "-DCMAKE_CXX_FLAGS='-Wno-deprecated-copy -Wno-pessimizing-move'" "-GUnix Makefiles" /<<PKGBUILDDIR>>/src/rocksdb

[Ponnuvel Palaniyappan (pponnuvel)]

@Corey

> The individual building of rocksdb files doesn't give off much information in the package builds to see compiler flags set.

VERBOSE flag would show full flags - however, that'd need changing rules & rebuilding again. That we can RelWithDebInfo is sufficient to confirm that it does pass those flags as I've locally built rocksdb with UCA/Ceph source using your cherry-picked patch (https://paste.ubuntu.com/p/YyMVq33BgD/). So this is fine.

I've installed the Ceph packages from your PPA - they install ok. However, I can't deploy a Ceph cluster as the default ceph-deploy package conflicts with the packages in your PPA.

This is from a lxd container (ubuntu-daily:groovy):

Unpacking ceph-base (15.2.3-0ubuntu4~ubuntu20.10.1~ppa202009221514) ...
dpkg: error processing archive /tmp/apt-dpkg-install-3hzpv6/32-ceph-base_15.2.3-0ubuntu4~ubuntu20.10.1~ppa202009221514_amd64.deb (--unpack):
 trying to overwrite '/usr/share/man/man8/ceph-deploy.8.gz', which is also in package ceph-deploy 2.0.1-0ubuntu1
Selecting previously unselected package smartmontools.
Preparing to unpack .../33-smartmontools_7.1-1build1_amd64.deb ...
Unpacking smartmontools (7.1-1build1) ...
Errors were encountered while processing:
 /tmp/apt-dpkg-install-3hzpv6/32-ceph-base_15.2.3-0ubuntu4~ubuntu20.10.1~ppa202009221514_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Ponnuvel Palaniyappan (pponnuvel)]

Hi @Corey,

(Ignore my comments as I've managed to setup a cluster without needing ceph-deploy).

I have installed the packages and setup a cluster. The cluster was functional and found no issues with the Groovy packages in my tests.

I have done some of the Ceph's benchmarks [0] using the Ceph cluster. Also did some separate tests on rocksdb with and without those flags. Overall, the Ceph Cluster tests which were mainly focusing on I/O didn't show anything out of the ordinary. But the rocksdb benchmarks [1] [2] showed that the optimized build works better in most cases.

Besides, we already that this has already been enabled in Ceph upstream as you know -- which we'll get in the future releases. So I am positive that there are no cause for concern with these packages that you've built for Groovy.

[0] https://tracker.ceph.com/projects/ceph/wiki/Benchmark_Ceph_Cluster_Performance
[1] https://github.com/facebook/rocksdb/wiki/Read-Modify-Write-Benchmarks
[2] https://github.com/facebook/rocksdb/wiki/RocksDB-In-Memory-Workload-Performance-Benchmarks

[Tyler Stachecki (tstachecki)]

Ponnuvel, don't the fio tests show significantly better write IOPS and latency in the optimized case?

========== rocksdb%20optimized.txt
...
  write: IOPS=1269, BW=5076KiB/s (5198kB/s)(2048MiB/413114msec); 0 zone
...
     | 99.00th=[ 180], 99.50th=[ 384], 99.90th=[ 1003], 99.95th=[ 1636],
     | 99.99th=[ 3641]
========== rocksdb%20UNoptimized.txt
...
  write: IOPS=383, BW=1533KiB/s (1570kB/s)(2048MiB/1367654msec); 0 zone
...
     | 99.00th=[ 1099], 99.50th=[ 1536], 99.90th=[ 3239], 99.95th=[ 4178],
     | 99.99th=[ 6544]

[Ponnuvel Palaniyappan (pponnuvel)]

@Tyler, Indeed it does. My cluster tests were done in a "cloud-on-cloud" environment, so I didn't want to stress too much on the actual numbers (whereas rocksdb tests were done locally in a tightly controlled environment). As a whole, we can definitely say that RelWithDebInfo is the way to go for rocksdb, too.

[Corey Bryant (corey.bryant)]

@Ponnuvel, thanks for testing. I've pushed the changes to the ubuntu/groovy package branch and will ask James to +1 before we upload to groovy. If this is low risk we can consider backporting via SRU.

[Ponnuvel Palaniyappan (pponnuvel)]

@Corey,

Had a discussion with Daniel Hill in SEG, and we think the risk is low. So, yes, we'd like to backport this to all releases. I am happy to do SRU (and/or tests) once this is ready.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 15.2.5-0ubuntu1

---------------
ceph (15.2.5-0ubuntu1) groovy; urgency=medium

  [ Corey Bryant ]
  * d/control: Remove Breaks/Replaces that are older than Focal (LP: #1878419).
  * d/p/enable-strsignal.patch: Enable HAVE_REENTRANT_STRSIGNAL for groovy
    to fix FTBFS with glibc 2.32.
  * d/p/rules: Enable RelWithDebInfo (LP: #1894453).

  [ James Page ]
  * New upstream point release (LP: #1898200).
  * d/p/fix-crash-in-committed-osd-maps.patch: Drop, included upstream.
  * d/p/*: Refresh.
  * d/libcephfs-dev.install: Drop ceph_statx.h.

  [ Dimitri John Ledkov ]
  * d/rules,control: Enable boost context usage on s390x (LP: #1694926).

 -- James Page <email address hidden> Tue, 06 Oct 2020 12:01:52 +0100

[Brian Murray (brian-murray)]

From the bug description its not clear to me why this change should be SRU'ed to Focal or what the benefit to end users is. Could someone please elaborate on that? Thanks!

[Ponnuvel Palaniyappan (pponnuvel)]

Hi Brian,

I have updated the description. Does it help?

On Tue, Oct 27, 2020 at 10:01 PM Brian Murray <email address hidden>
wrote:

> >From the bug description its not clear to me why this change should be
> SRU'ed to Focal or what the benefit to end users is. Could someone
> please elaborate on that? Thanks!
>
> ** Changed in: ceph (Ubuntu Focal)
> Status: Triaged => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1894453
>
> Title:
> Building Ceph packages with RelWithDebInfo
>
> Status in Ubuntu Cloud Archive:
> Invalid
> Status in Ubuntu Cloud Archive queens series:
> Triaged
> Status in Ubuntu Cloud Archive stein series:
> Triaged
> Status in Ubuntu Cloud Archive train series:
> Triaged
> Status in Ubuntu Cloud Archive ussuri series:
> Triaged
> Status in ceph package in Ubuntu:
> Fix Released
> Status in ceph source package in Bionic:
> Triaged
> Status in ceph source package in Focal:
> Incomplete
>
> Bug description:
> [Impact]
> This is to consider/discuss the possibility of whether the Ceph packages
> (UCA) should be built with RelWithDebInfo build option (CMAKE_BUILD_TYPE
> [0]) which can be supplied to the build script do_cmake.sh [1].
>
> Currently, Upstream as well as UCA builds default to 'Debug' [0]
> builds. But is any reason why shouldn't go for RelWithDebInfo (i.e.
> "Release with Debug information")?
>
> There's one issue noted with RelWithDebInfo which has a workaround too
> [2].
>
> In general, gcc's -O2 [3] build should generate better code as well as
> there's a specific case for Ceph [4] that it improves performance.
>
> [0] https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html
>
> [1] https://github.com/ceph/ceph/blob/master/do_cmake.sh
>
> [2] https://docs.ceph.com/docs/master/dev/perf/#common-issues
>
> [3] https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html
>
> [4] https://bugs.gentoo.org/show_bug.cgi?id=733316
>
> [Test Case]
> Standard ceph-backed OpenStack regression will be run, consisting of
> juju-deployed OpenStack+ceph and tempest regression testing against the
> deployment.
>
> [Regression Potential]
> As Ponnuvel mentioned, the size of binaries will be slightly bigger and
> also the build/compilation will be longer.
> Ponnuvel also did a search of pponnuvel i'd search of upstream bugs at
> https://tracker.ceph.com and didn't find any relevant bugs.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-archive/+bug/1894453/+subscriptions
>

--
Regards,
Ponnuvel

[Corey Bryant (corey.bryant)]

I've uploaded this change to train-staging, stein-staging and the bionic unapproved queue. Note, for these versions there is no change to armhf as enabling RelWithDebInfo for armhf was leading to build timeouts.

[Łukasz Zemczak (sil2100)]

Hello Ponnuvel, or anyone else affected,

Accepted ceph into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/15.2.5-0ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

An upload of ceph to bionic-proposed has been rejected from the upload queue for the following reason: "Upload contains garbage/leftover control.orig. Please re-upload after cleanup.".

[Corey Bryant (corey.bryant)]

Hello Ponnuvel, or anyone else affected,

Accepted ceph into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Ponnuvel, or anyone else affected,

Accepted ceph into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Ponnuvel, or anyone else affected,

Accepted ceph into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Ponnuvel, or anyone else affected,

Accepted ceph into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/12.2.13-0ubuntu0.18.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ponnuvel Palaniyappan (pponnuvel)]

SRU tests are oood for ussuri (observed perf numbers are better). Did a short rados benchmark and attached the results (before and after upgrade).

The steps taken are:
- Upgrade the packages on monitor, and OSDs in order.
- Restart monitor, manager and OSDs in order.
- Repeat the same benchmark test.

Attached files have the output of the commands:
 - lsb_release -a
 - dpkg -l | grep ceph
 - ceph versions
 - ceph osd pool create scbench 128 128
 - rados bench -p scbench 10 write

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ceph (12.2.13-0ubuntu0.18.04.5) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

libvirt/4.0.0-1ubuntu8.17 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#ceph

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ponnuvel Palaniyappan (pponnuvel)]

The smoke-lxc test failure (reported in comment #37) could be due to https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180.

LP#1899180 suggests this could be treated as flaky. If it's easy, coud try re-running the regression tests for 12.2.13-0ubuntu0.18.04.5 to see if it's indeed a Heisenbug.

[James Page (james-page)]

verification of this fix on focal is currently blocking 15.2.5 release into focal, and the release of the same update into the bionic/ussuri UCA.

@pponnuvel - can you complete the verification work for focal-proposed as well please?

[Ponnuvel Palaniyappan (pponnuvel)]

Performed same as for Ussuri (comment #34) and attached results here.

[Ponnuvel Palaniyappan (pponnuvel)]

^^ The above is for Stein verification.

[Ponnuvel Palaniyappan (pponnuvel)]

Test results for Train.

[James Page (james-page)]

I just tested focal-proposed - significant increase in iops and throughput and massive reduction in latency.

[Ponnuvel Palaniyappan (pponnuvel)]

Re-run of the regression tests is OK for 12.2.13-0ubuntu0.18.04.5 [0].

Test results for 12.2.13-0ubuntu0.18.04.5 is good. Attached before & after test results.

[0] https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#ceph

[Ponnuvel Palaniyappan (pponnuvel)]

James Page has done focal-proposed verification already (comment #43) - with that, all tests are done and verified. Please let me know if you have any questions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 15.2.5-0ubuntu0.20.04.1

---------------
ceph (15.2.5-0ubuntu0.20.04.1) focal; urgency=medium

  [ Corey Bryant ]
  * d/p/rules: Enable RelWithDebInfo (LP: #1894453).

  [ James Page ]
  * New upstream point release (LP: #1898200).
    - d/p/fix-crash-in-committed-osd-maps.patch: Drop, included upstream.
    - d/libcephfs-dev.install: Drop ceph_statx.h.
    - d/libcephfs-dev.install: Add new header files.

 -- James Page <email address hidden> Mon, 02 Nov 2020 15:18:02 +0000

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 12.2.13-0ubuntu0.18.04.5

---------------
ceph (12.2.13-0ubuntu0.18.04.5) bionic; urgency=medium

  * d/p/rules: Enable RelWithDebInfo (LP: #1894453).

 -- Corey Bryant <email address hidden> Mon, 26 Oct 2020 09:17:19 -0400

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ceph - 15.2.5-0ubuntu0.20.04.1~cloud0
---------------

 ceph (15.2.5-0ubuntu0.20.04.1~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (15.2.5-0ubuntu0.20.04.1) focal; urgency=medium
 .
   [ Corey Bryant ]
   * d/p/rules: Enable RelWithDebInfo (LP: #1894453).
 .
   [ James Page ]
   * New upstream point release (LP: #1898200).
     - d/p/fix-crash-in-committed-osd-maps.patch: Drop, included upstream.
     - d/libcephfs-dev.install: Drop ceph_statx.h.
     - d/libcephfs-dev.install: Add new header files.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ceph - 14.2.11-0ubuntu0.19.10.1~cloud4
---------------

 ceph (14.2.11-0ubuntu0.19.10.1~cloud4) bionic-train; urgency=medium
 .
   * d/p/rules: Enable RelWithDebInfo (LP: #1894453).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ceph - 13.2.9-0ubuntu0.19.04.1~cloud2
---------------

 ceph (13.2.9-0ubuntu0.19.04.1~cloud2) bionic-stein; urgency=medium
 .
   * d/p/rules: Enable RelWithDebInfo (LP: #1894453).

[Corey Bryant (corey.bryant)]

Hello Ponnuvel, or anyone else affected,

Accepted ceph into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ponnuvel Palaniyappan (pponnuvel)]

Tested queens-proposed following the same steps noted in #34. Like others results previously, this shows better results as well. Attached the test notes.

[Ponnuvel Palaniyappan (pponnuvel)]

Tested queens-proposed following the same steps noted in #34. Like others results previously, this shows better results as well. Attached the test notes.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ceph - 12.2.13-0ubuntu0.18.04.5~cloud0
---------------

 ceph (12.2.13-0ubuntu0.18.04.5~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (12.2.13-0ubuntu0.18.04.5) bionic; urgency=medium
 .
   * d/p/rules: Enable RelWithDebInfo (LP: #1894453).