[SRU] ceph 17.2.7 point release

[Christian Rohmann]

[Impact]
This release fixes several bugs. We would like to make sure all of our users have access to these improvements.

The update contains the following package updates:

   * ceph 17.2.7

[Test Case]
The following SRU process was followed:

https://wiki.ubuntu.com/OpenStackUpdates

In order to avoid regression of existing users, the OpenStack team will run their continuous integration test against the packages that are in -proposed. A successful run of all available tests will be required before the proposed packages can be let into -updates.

The OpenStack team will be in charge of attaching the output summary of the executed tests. The OpenStack team members will not mark ‘verification-done’ until this has happened.

[Regression Potential]
In order to mitigate the regression potential, the results of the aforementioned tests will be attached to this bug.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Peter Sabaini (peter-sabaini)]

There is this upstream issue still present in 17.2.7 I believe? https://tracker.ceph.com/issues/63391
We may want to wait for this fix being released

[Christian Rohmann (christian-rohmann)]

Thanks Peter for looking into this. The fix is already merged, see https://github.com/ceph/ceph/pull/54392.
I wonder if there will be a bugfix release to the 17.2.7 point release to tackle this regression? Do you know more? The merge was 2 weeks ago.

Would you otherwise cherry-pick this patch and build packages for 17.2.7?

[Markus Schade (lp-markusschade)]

There is also a fix for a year-long issue (https://tracker.ceph.com/issues/16767) still waiting to be merged in quincy: https://github.com/ceph/ceph/pull/51976

The corresponding fix has been merged in reef and main.

[Christian Rohmann (christian-rohmann)]

I was asking for packages for 17.2.7 and the HUGE amount of fixes that come with that point release.
Sure there will always be more (very interesting) bugfixes ... but those will then come with 17.2.8 likely.

Just look at the sheer size of the changelog and the list of bugfixes in that point release: https://docs.ceph.com/en/latest/releases/quincy/#v17-2-7-quincy

To me the mClock fix (in "notable changes") is especially important as that causes VERY slow recovery times, even with beefy servers and fast storage media. But quintessentially I'd simply like to quote the Ceph people:

"This is the seventh backport release in the Quincy series. We recommend that all users update to this release."

[Peter Sabaini (peter-sabaini)]

Agreed we do want to make 17.2.7 available, especially for the mClock fix.

There doesn't seem to be an ETA for 17.2.8 so we will look to backport the patch for the ceph-volume breakage

On 29.11.23 10:26, Christian Rohmann wrote:
> I was asking for packages for 17.2.7 and the HUGE amount of fixes that come with that point release.
> Sure there will always be more (very interesting) bugfixes ... but those will then come with 17.2.8 likely.
>
> Just look at the sheer size of the changelog and the list of bugfixes in
> that point release:
> https://docs.ceph.com/en/latest/releases/quincy/#v17-2-7-quincy
>
> To me the mClock fix (in "notable changes") is especially important as
> that causes VERY slow recovery times, even with beefy servers and fast
> storage media. But quintessentially I'd simply like to quote the Ceph
> people:
>
>
> "This is the seventh backport release in the Quincy series. We recommend that all users update to this release."
>

[Christian Rohmann (christian-rohmann)]

Hey Peter! Is there any ETA on packages for 17.2.7 for Jammy then?

[Christian Rohmann (christian-rohmann)]

May I kindly ask again what you plans regarding the release of packages for 17.2.7 are?

[Peter Sabaini (peter-sabaini)]

Christian, we plan to upload packages for 17.2.7 first weeks of January. I do not have an ETA of the actual release date (this depends on verification outcomes).

[Christian Rohmann (christian-rohmann)]

Thanks Peter. Did not mean to pull on the grass (as they say in German if someone is impatiently asking for things over and over), I am just anxious (*sic*) to get the mClock fix myself.

[Peter Sabaini (peter-sabaini)]

No worries, the mClock fix certainly is important!

[Ponnuvel Palaniyappan (pponnuvel)]

https://tracker.ceph.com/issues/57244 is also fixed in 17.2.7 which was reported by a customer.

[Tyler Stachecki (tstachecki)]

Hi, just wanted to report that mClock may still broken in this release (17.2.7). There are already some ML posts suggesting that backfill can lock up and stop making progress in some cases with it enabled.

[Christian Rohmann (christian-rohmann)]

Tyler I suppose you are referring to e.g. https://<email address hidden>/thread/IPHBE3DLW5ABCZHSNYOBUBSI3TLWVD22/ ?

Yes, mClock seems, also reading through the other recent posts, to be something operators have issues with.
Either with slow convergence (recovery), influence of scrubbing, ....

And there are more fixes incoming: https://github.com/ceph/ceph/pull/51171

But considering the SRU / update from 17.2.6 to 17.2.7 those might also not be problems / regressions of this particular point release. All of 17.x (Quincy) is using mClock scheduler by default. I believe it's more about more installations moving to mClock that we now see more and more issues.

What I am trying to say is that 17.2.7 might actually not make things any worse, but could actually deliver fixes / improvements that Ceph has done.

[Luciano Lo Giudice (lmlogiudice)]

Note: This SRU includes the bugfix for this issue: https://tracker.ceph.com/issues/63391
Which was pushed via this PR: https://github.com/ceph/ceph/pull/54392

The reason for doing so is that this SRU is otherwise unusable to us - Any Ceph-related solution, such as charmed Ceph would be completely useless without this bugfix.

[James Page (james-page)]

Uploaded for SRU team review (subscribed).

[Łukasz Zemczak (sil2100)]

Before I accept this into -proposed: as this SRU includes the aforementioned bugfix from comment #15, does the upload require some additional testing to be performed? Or will this bugfix be covered by the usual MRE testing?

[Tyler Stachecki (tstachecki)]

Christian re: post #14, yes - I am referring to posts and feedback of that form that I have received or read. I agree that mClock probably has not regressed any _more_ here -- just suggesting that this release likely will not solve all outstanding issues with mClock.

[James Page (james-page)]

@sil2100 - the fix would be covered by the usual MRE testing - infact without it our usual testing won't work!

[Andreas Hasenack (ahasenack)]

This upload contains some changes that are not mentioned here or in d/changelog:
- new file debian/ceph.patch. Not referenced anywhere, according to grep. Cruft?
- dropped CVE-2023-43040.patch, presumably upstream (confirmed in the changelog at https://docs.ceph.com/en/latest/releases/quincy/#v17-2-7-quincy), but not mentioned in d/changelog

Additionally, the d/p/fix-lvm-devices.patch could use better DEP3 headers, like an upstream reference, shown in comment #15 (and that PR is merged now).

This last one is not a blocker, but the d/ceph.patch file needs some explanation please.

[Andreas Hasenack (ahasenack)]

On the CVE point, looks like you missed rebasing your change on top of what is in jammy-security/updates right now, which is 17.2.6-0ubuntu0.22.04.3, which is what added that patch.

[Luciano Lo Giudice (lmlogiudice)]

I've pushed the changes locally to include the CVE fix and improve the commit message for the LVM patch. I'll wait for James' approval.

OTOH, I haven't found any ceph.patch file. Can you point to me where you found it, Andreas?

[Andreas Hasenack (ahasenack)]

From https://launchpadlibrarian.net/713408703/ceph_17.2.6-0ubuntu0.22.04.2_17.2.7-0ubuntu0.22.04.1.diff.gz. a debian/ceph.patch file is being added:

diff -Nru ceph-17.2.6/debian/ceph.patch ceph-17.2.7/debian/ceph.patch
--- ceph-17.2.6/debian/ceph.patch 1970-01-01 00:00:00.000000000 +0000
+++ ceph-17.2.7/debian/ceph.patch 2024-01-31 17:10:43.000000000 +0000
@@ -0,0 +1,58 @@
+diff --git a/debian/control b/debian/control
+index 683ad2df6c..469a1bb1c3 100644
+--- a/debian/control
++++ b/debian/control
+@@ -13,7 +13,7 @@ Vcs-Browser: https://salsa.debian.org/ceph-team/ceph
+ Build-Depends:
+ clang [armhf mipsel armel m68k sh4],
+ cmake,
+- cython3-legacy,
++ cython3,
+ debhelper (>= 10~),
+ default-jdk,
+ dh-exec,
+@@ -26,21 +26,6 @@ Build-Depends:
+ libbabeltrace-ctf-dev,
+ libbabeltrace-dev,
+ libblkid-dev (>= 2.17),
+- libboost-atomic-dev (>= 1.74.0),
+- libboost-chrono-dev (>= 1.74.0),
+- libboost-context-dev (>= 1.74.0) [!mips64el !ia64 !m68k !ppc64 !sh4 !sparc64 !x32],
+- libboost-coroutine-dev (>= 1.74.0) [!mips64el !ia64 !m68k !ppc64 !sh4 !sparc64 !x32],
+- libboost-date-time-dev (>= 1.74.0),
+- libboost-filesystem-dev (>= 1.74.0),
+- libboost-iostreams-dev (>= 1.74.0),
+- libboost-program-options-dev (>= 1.74.0),
+- libboost-python-dev (>= 1.74.0),
+- libboost-random-dev (>= 1.74.0),
+- libboost-regex-dev (>= 1.74.0),
+- libboost-system-dev (>= 1.74.0),
+- libboost-test-dev (>= 1.74.0),
+- libboost-thread-dev (>= 1.74.0),
+- libboost-timer-dev (>= 1.74.0),
+ libbz2-dev,
+ libc-ares-dev,
+ libcap-dev,
+diff --git a/debian/patches/series b/debian/patches/series
+index 608f0eb491..c8a98c3f0d 100644
+--- a/debian/patches/series
++++ b/debian/patches/series
+@@ -16,4 +16,4 @@ fix-CheckCxxAtomic-riscv64.patch
+ patch-out-exporter.patch
+ fix-mantic-misc.patch
+ cmake-allow-empty-submodules.patch
+-fix-boost-1.82-build.patch
++#fix-boost-1.82-build.patch
+diff --git a/debian/rules b/debian/rules
+index 5babc6c9fc..e56e2ca829 100755
+--- a/debian/rules
++++ b/debian/rules
+@@ -63,7 +63,7 @@ extraopts += -DWITH_PYTHON3=3
+ extraopts += -DWITH_CEPHFS_JAVA=ON
+ extraopts += -DWITH_CEPHFS_SHELL=ON
+ extraopts += -DWITH_TESTS=OFF
+-extraopts += -DWITH_SYSTEM_BOOST=ON
++extraopts += -DWITH_SYSTEM_BOOST=OFF
+ extraopts += -DWITH_LTTNG=OFF -DWITH_EMBEDDED=OFF
+ extraopts += -DCMAKE_INSTALL_LIBEXECDIR=/usr/lib
+ extraopts += -DWITH_MGR_DASHBOARD_FRONTEND=OFF
diff -Nru ceph-17.2.6/debian/changelog ceph-17.2.7/debian/changelog
--- ceph-17.2.6/debian/changelog 2023-08-31 22:44:27.000000000 +0000
+++ ceph-17.2.7/debian/changelog 2024-02-08 11:33:49.000000000 +0000
@@ -1,3 +1,12 @@
+ceph (17.2.7-0ubuntu0.22.04.1) jammy; urgency=medium
+
+ * New upstream point release (LP: #2043336):
+ - d/p/fix-lvm-devices.patch: Cherry pick committed fix for issues
+ using LVM PV/LV for OSDs introduced in 17.2.7 release.
+ - d/p/patch-out-exporter.patch: Refresh for new release.
+
+ -- Luciano Lo Giudice <email address hidden> Thu, 08 Feb 2024 11:33:49 +0000
+
 ceph (17.2.6-0ubuntu0.22.04.2) jammy; urgency=medium

   [ Peter Sabaini]

[Luciano Lo Giudice (lmlogiudice)]

Ack. I'll track from where this file came from, but it's definitely cruft. All the needed patches are in d/p already.

[Andreas Hasenack (ahasenack)]

Ok, I'll reject that upload from the unapproved queue then.

[James Page (james-page)]

Apologies - the ceph.patch was hiding in my local working directory when I generated the source package.

Luciano and I met today to review generally - the CVE patch can be safely dropped as it was included in the upstream release - I've merged and updated the changelog to that effect in the version I just uploaded.

[James Page (james-page)]

@ahasenack please could we get the refreshed upload reviewed and accepted into proposed if it looks good now?

[Andreas Hasenack (ahasenack)]

On it

[Andreas Hasenack (ahasenack)]

Would have been good to have a bug reference for the fix-lvm-devices.patch. I think it's https://tracker.ceph.com/issues/63391 ? And for the patch as well, although once it has an Author, the Origin tag is not mandatory.

[Andreas Hasenack (ahasenack)]

Hello Christian, or anyone else affected,

Accepted ceph into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/17.2.7-0ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Luciano Lo Giudice (lmlogiudice)]

Tested on jammy and the tests ran fine (attaching full logs).

2024-03-15 16:08:53 [INFO] Events:
  Deploy Bundle:
    Start: 1710515992.8099723
    Finish: 1710516086.5489063
    Elapsed Time: 93.73893404006958
    PCT Of Run Time: 4
  Prepare Environment:
    Start: 1710515988.2187657
    Finish: 1710515992.8092358
    Elapsed Time: 4.5904700756073
    PCT Of Run Time: 1
  Test zaza.charm_tests.lifecycle.tests.UpgradeCharmsToPath;ceph-mon:
    Start: 1710518663.496419
    Finish: 1710518668.3889372
    Elapsed Time: 4.8925182819366455
    PCT Of Run Time: 1
  Test zaza.openstack.charm_tests.ceph.mon.tests.CephPermissionUpgradeTest:
    Start: 1710518882.371464
    Finish: 1710518933.5469208
    Elapsed Time: 51.17545676231384
    PCT Of Run Time: 2
  Test zaza.openstack.charm_tests.ceph.osd.tests.SecurityTest:
    Start: 1710518839.4351707
    Finish: 1710518841.249194
    Elapsed Time: 1.8140232563018799
    PCT Of Run Time: 1
  Test zaza.openstack.charm_tests.ceph.tests.CephAuthTest:
    Start: 1710518842.9432013
    Finish: 1710518855.94271
    Elapsed Time: 12.999508619308472
    PCT Of Run Time: 1
  Test zaza.openstack.charm_tests.ceph.tests.CephLowLevelTest:
    Start: 1710518764.7217534
    Finish: 1710518773.7229624
    Elapsed Time: 9.001209020614624
    PCT Of Run Time: 1
  Test zaza.openstack.charm_tests.ceph.tests.CephMonActionsTest:
    Start: 1710518855.9428604
    Finish: 1710518882.3712504
    Elapsed Time: 26.42839002609253
    PCT Of Run Time: 1
  Test zaza.openstack.charm_tests.ceph.tests.CephMonJujuPersistent:
    Start: 1710518668.3891659
    Finish: 1710518762.7588964
    Elapsed Time: 94.3697304725647
    PCT Of Run Time: 4
  Test zaza.openstack.charm_tests.ceph.tests.CephPrometheusTest:
    Start: 1710518841.2492943
    Finish: 1710518842.943128
    Elapsed Time: 1.693833827972412
    PCT Of Run Time: 1
  Test zaza.openstack.charm_tests.ceph.tests.CephTest:
    Start: 1710518773.7234535
    Finish: 1710518839.4349415
    Elapsed Time: 65.71148800849915
    PCT Of Run Time: 3
  Test zaza.openstack.charm_tests.ceph.tests.CheckPoolTypes:
    Start: 1710518762.7592592
    Finish: 1710518764.7217026
    Elapsed Time: 1.9624433517456055
    PCT Of Run Time: 1
  Wait for Deployment:
    Start: 1710516086.548988
    Finish: 1710518541.8577774
    Elapsed Time: 2455.308789253235
    PCT Of Run Time: 84
Metadata: {}

Ceph version:
(venv) ubuntu@lmlogiudice-bastion:~/src/$ juju ssh ceph-mon/0 -- 'ceph -v'
ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 17.2.7-0ubuntu0.22.04.1

---------------
ceph (17.2.7-0ubuntu0.22.04.1) jammy; urgency=medium

  * New upstream point release (LP: #2043336):
    - d/p/fix-lvm-devices.patch: Cherry pick committed fix for issues
      using LVM PV/LV for OSDs introduced in 17.2.7 release.
    - d/p/patch-out-exporter.patch: Refresh for new release.
    - d/p/CVE-2023-43040.patch: Drop, included in release.

 -- Luciano Lo Giudice <email address hidden> Thu, 08 Feb 2024 11:33:49 +0000

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

Update release to the Yoga UCA as well.

[James Page (james-page)]

 ceph (17.2.7-0ubuntu0.22.04.1~cloud0) focal-yoga; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 ceph (17.2.7-0ubuntu0.22.04.1) jammy; urgency=medium
 .
   * New upstream point release (LP: #2043336):
     - d/p/fix-lvm-devices.patch: Cherry pick committed fix for issues
       using LVM PV/LV for OSDs introduced in 17.2.7 release.
     - d/p/patch-out-exporter.patch: Refresh for new release.
     - d/p/CVE-2023-43040.patch: Drop, included in release.

[Christian Rohmann (christian-rohmann)]

Ceph has started to produce (Quincy /17.2.x) packages for Jammy,
https://<email address hidden>/thread/NIALMLMGVQVZW5INEABS6W4LDIQYI3UC/,

They are available at: https://download.ceph.com/debian-quincy/dists/jammy/