[SRU] Squid: Ceph new point release 19.2.3

[Ponnuvel Palaniyappan]

[Impact]
This release fixes several bugs. We would like to make sure all of our users have access to these improvements.

The update contains the following package updates:
   * ceph 19.2.2 https://ceph.io/en/news/blog/2025/v19-2-2-squid-released/
   * ceph 19.2.3 https://ceph.io/en/news/blog/2025/v19-2-3-squid-released/

Notable changes:
    * This fixes a silent RGW data loss regression when CopyObject is used to copy an object onto itself. The regression results in objects past the original 4mb head object being added to the garbage collection queue and deleted permanently, and are not recoverable.
    * LTTNG has been enabled for easier tracing and debugging.

Key differences from upstream:
    * libnbd has been disabled from the build system. libnbd is in universe in Ubuntu, and cannot be a dependency for 19.2.3.
    * On questing, adjustments had to be made to build with boost1.88. On plucky and noble, these adjustments have been removed to build with standard boost 1.83 and 1.82.
    * python-cryptography uses PyO3 to call some rust based bcrypt and TLS functions, and has problems being called in parallel and in sub-interpreters, necessary for ceph-dashboard to work correctly. This was fixed by backporting:
  commit 717d0a6f3530ad3e07f4423002810327b2addcf1
  From: "Paulo E. Castro" <email address hidden>
  Date: Sat, 5 Apr 2025 21:47:55 +0100
  Subject: pybind/mgr: Hack around the 'ImportError: PyO3 modules
   may only be initialized once per interpreter process' issue.
  Link: https://github.com/ceph/ceph/commit/7094a5a44d90e705141dbae9739e6c0835bf7ce3
  See also:
   https://tracker.ceph.com/issues/64213
   https://github.com/bazaah/aur-ceph/issues/20
   https://bugs.launchpad.net/charm-ceph-dashboard/+bug/2073115
    * Noble also receives the bugfix for hardened CFLAGS for python modules. See https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2078244

[Test Case]

This SRU will run all functional tests with the packages in 'proposed' for every Ceph charm maintained by Canonical's Ceph team, so that there's a higher chance of catching bugs associated with any Ceph component.

The charms for which the functional tests will be run are: ceph-mon, ceph-osd,
ceph-radosgw, ceph-proxy, ceph-nfs, ceph-nvme, ceph-fs and ceph-rbd-mirror.

The functional test suite includes upgrade testing from the current stable
release to the proposed (in this case, from 19.2.1 to 19.2.3)

Since our functional tests do not include testing for LTTNG, this will be
tested manually. This involves the following steps:

- Deploy a Ceph cluster with 19.2.3.
- Enable OSD tracing by adding the option `osd_tracing` to /etc/ceph.conf
- Run `lttng create test`
- Run `lttng enable-event --userspace osd:*`
- At this point, tracing is enabled. Do some writes to an OSD, e.g: By creating an RBD object or via ceph bench.
- Stop tracing with `lttng stop`
- View the logs with `lttng view`.
- We can then do some more writes on an OSD and verify that no more tracing output is generated.

[Regression Potential]

ceph is a critical package as it underpins large scale data storage and integrity checking, and care must be taken to not introduce any data loss regressions. ceph also is a common backing store for OpenStack virtual machines through ceph-rbd, and care must be taken to not break any API or ABI compatibility with current OpenStack releases.

In order to mitigate the regression potential, the results of the aforementioned tests will be attached to this bug.

[Other info]

Microrelease exception: https://documentation.ubuntu.com/project/SRU/reference/exception-OpenStack-Updates/

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Darth Shadow (anaghk-dos)]

Just to get clarity, what kind of timelines are we looking at to have this version available on Noble? Is it in the order of days, weeks or even months?

Asking because 19.2.3 has a few critical fixes for various hangs in the client and I would like to upgrade my servers ASAP.

Will need to look into alternative sources (like building my own packages) if the timeline is longer for this to be available.

[Matthew Ruffell (mruffell)]

Luciano Lo Giudice (lmlogiudice) wrote on 2025-09-02:

As with before, I kindly ask anyone with the right permissions to set the affected targets to Noble, Plucky and Questing.

The PPA's for those respective versions can be seen here:
https://launchpad.net/~lmlogiudice/+archive/ubuntu/ceph-noble-1923
https://launchpad.net/~lmlogiudice/+archive/ubuntu/ceph-plucky-1923
https://launchpad.net/~lmlogiudice/+archive/ubuntu/ceph-questing-1923

Once in proposed, I will run the full suite of functional tests for verification.

[Matthew Ruffell (mruffell)]

Hi everyone,

I was talking to Dan Hill today, and we noticed that ceph-exporter exists in
plucky and questing and not in noble:

$ rmadison ceph-exporter
 ceph-exporter | 19.2.1-0ubuntu1 | plucky/universe | amd64, arm64, ppc64el, riscv64, s390x
 ceph-exporter | 19.2.1-0ubuntu3 | questing/universe | amd64, arm64, ppc64el, riscv64, s390x

It seems that this package was introduced in oracular:

https://launchpad.net/ubuntu/+source/ceph/19.2.0~is.really.19.1.0-0ubuntu1

    - d/p/patch-out-exporter.patch: Drop, enable exporter.
    - d/control,ceph-exporter.install: Enable ceph-exporter binary
      package.
    - d/p/ceph-exporter-systemd.patch: Cherry pick commit for ceph-
      exporter systemd configuration.

Dan Hill and I talked about the different exporters in use over the years
including their pros and cons.

* Digital ocean implementation (external), can overrun the MGR causing outages
to the manager, can cause cluster instability, not ideal.
* Then was moved inside the manager, in the mgr-module prometheus. Solved
manager overload issues. However, When there is control plane failure, metrics
stop working, no good.
* Then was moved? to a separate binary / systemd service called ceph-exporter
which when run, disables the mgr-module prometheus. Best solution.

I talked this over with Dan H, and also Christopher Halse Rogers during our SRU
shift. We both came to the same conclusion that ceph-exporter should probably
be fully enabled with proper debian packaging and a systemd service if we are
going to make it work properly.

So, here is what we are going to do.

For noble 19.2.3 and jammy 17.2.9, we will add the debian/control entries for
ceph-exporter, add d/ceph-exporter.install and then likely look at if
d/p/ceph-exporter-systemd.patch is necessary.

Christopher and I rejected 17.2.9 for the time being, and we will enable the
exporter properly in a new upload.

Dan Hill and I will work on the packaging for 19.2.3 tomorrow morning. From
there, lets try and get this uploaded soon.

Luciano, if you don't want this to happen or think it is a bad idea, let us
know.

Thanks,
Matthew

[Luciano Lo Giudice (lmlogiudice)]

Hello Matthew,

Regardless of what I think, since Oracular introduced those changes, I believe we should apply them to the branches affected by this SRU as well. I've re-uploaded the PPA's with the changes now (the .install file and the addition of ceph-exporter to d/control; the systemd file was already there).

Let me know if there's anything else missing,
Luciano.

[Matthew Ruffell (mruffell)]

Hi Luciano,

This is going to need a lot of rework.

I diffed the upstream tarball https://download.ceph.com/tarballs/ceph-19.2.3.tar.gz
with ceph_19.2.3.orig.tar.xz from the uploads, and noticed there is a new
directory, src/python-common/ceph/cryptotools/* added.

Dan Hill and I managed to determine that it was a part of d/p/pyo3-fix.patch,
but I am very uncomfortable with this being a part of the actual upstream orig
tarball as it was never intended to be there as part of the 19.2.3 release.
This really should be a part of d/p/pyo3-fix.patch, mostly so we can we can see
that it all belongs together as one patch that does one thing, that we aren't
sneaking random bits of upstream code in the upstream tarball that can easily
be missed and forgotten when we do 19.2.4 or whatever next time.

I think it needs to have proper dep3 tagging and attribution, as well as more
documentation around the actual problem. Please consider filing a Launchpad
bug explaining the packaging issue and talking about it more in the debian
changelog. Also, consider something like the following in the patch itself:

So, A one line Description tag with "Fix pyo3 issues" just is not enough, it
needs to be like:

Subject: <Talk about the reasons why we need this patch, definitely talk about
 sub interpreter issues. Subject can be multi line, while description is only a
 single line.>
Author: John Mulligan <email address hidden>
Author: Paulo E. Castro <email address hidden>
Author: Ernesto Puerta <email address hidden>
Origin: backport, https://github.com/ceph/ceph/pull/62951
Bug: https://tracker.ceph.com/issues/64213
Bug-Ubuntu: https://bugs.launchpad.net/bugs/<lp bug>
Bug-Arch: https://github.com/bazaah/aur-ceph/issues/20

We really should carry the entire patchset, even the changes to the unit tests,
so if anyone wishes to pull the ceph packages from the archive and execute them,
they work. Yes, people really do this.

So again, all changes to src/python-common/ceph/cryptotools/* need to be a part
of d/p/pyo3-fix.patch.

I also noticed that there is a hunk at the bottom of d/p/pyo3-fix.patch which
does not belong to this patchset:

diff --git a/src/rocksdb/db/blob/blob_file_meta.h b/src/rocksdb/db/blob/blob_file_meta.h
index d7c8a12433..2e47726f8d 100644
--- a/src/rocksdb/db/blob/blob_file_meta.h
+++ b/src/rocksdb/db/blob/blob_file_meta.h
@@ -6,6 +6,7 @@
 #pragma once

 #include <cassert>
+#include <cstdint>
 #include <iosfwd>
 #include <memory>
 #include <string>

This seems to belong to questing-boost-fix.patch? Please ensure it goes to the
correct file.

Why isn't questing-boost-fix.patch mentioned in debian/changelog?

The other major issue, which must be addressed, is that you took the noble
debian directory, and repurposed it for plucky and questing. This loses all
plucky and questings debian/changelog entries, and pushes forward nobles
flawed debian directory to questing. If you debdiff

debdiff questing/ceph_19.2.1-0ubuntu4.dsc questing/ceph_19.2.3-0ubuntu0.24.04.1~bpo25.10.1~ppa202508291528.dsc > diff.debdiff

you will quickly see that noble's debian directory is missing d/control and
d/ceph-exporter.install files, so they get "deleted" in plucky and q...

[Matthew Ruffell (mruffell)]

I have reviewed the questing package in https://launchpad.net/~lmlogiudice/+archive/ubuntu/ceph-proposed-1923 and I give it my +1.

- All patches are documented in debian/changelog
- New patches have correct and proper dep3 tagging
- Builds with -updates, Luciano is doing -proposed now.
- Orig tarball is removals only, notably of upstream debian dir, blog posts, fonts.

Once it completes a build in -proposed, the version can be adjusted and sponsored to questing by a coredev.

[Guillaume Boutry (gboutry)]

@mruffell, this is very late in the cycle, I believe a feature freeze exception would be required as we are well past beta freeze. In any case, I believe the conversation should be had with the release team :)

[Renan Rodrigo (rr)]

I have reviewed ceph 19.2.3-0ubuntu1 for uploading.
* Upstream code
- The diff from the upstream tarball makes sense as pointed out by @mrufell

* Build and test
- The package builds as expected in the PPA both with -proposed enabled and just with -updates
- Autopkgtests are passing for the version in the PPA

* Packaging
- Changes to d/control and d/rules make sense and are listed in the changelog
- Kept patches apply correctly, with no fuzz
- d/p/boost-url.patch was changed but it is not mentioned in the changelog. The change is adding 'process' to BOOST_COMPONENTS - which is used in one of the new patches (questing-boost-fix.patch). As this one targets 'url', it would make more sense for this change to be part of the new patch instead. I fixed this so the upload is not blocked.
- New patches:
 - d/p/questing-boost-fix.patch: The header specifies this does not need forwarding, it makes sense.
 - d/p/pyo3-fix.patch: The Origin metadata points to a pull request; generally I would say it's better to point at the commit itself. I took the liberty of changing that.
- All removed patches are now part of the upstream code and were correctly dropped. No other patches to drop.

* Exception
- FFe aproved by the release team (https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2126446)

[Renan Rodrigo (rr)]

That said,

Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading ceph_19.2.3-0ubuntu1.dsc: done.
  Uploading ceph_19.2.3.orig.tar.xz: done.
  Uploading ceph_19.2.3-0ubuntu1.debian.tar.xz: done.
  Uploading ceph_19.2.3-0ubuntu1_source.buildinfo: done.
  Uploading ceph_19.2.3-0ubuntu1_source.changes: done.
Successfully uploaded packages.

[Renan Rodrigo (rr)]

Something I overlooked - seems this upload fixes https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2120235, and thus that could be in the changelog and in the patch headers. Well too late now, but make sure to close the bug as fix committed as soon as the questing version lands.

[Renan Rodrigo (rr)]

Ok so
the new dependency in `libnbd` has introduced a whole bunch of component mismatches. Either all of those are MIRed so this migrates, or the dependency needs to be removed in a further patch. Coordinating with the Openstack team on what to do here.

[Matthias Klose (doko)]

the ocaml inclusion was a seed issue, that apparently nobody was looking at.

[Trent Lloyd (lathiat)]

Looking for a status update on this bug.

- What is the current status of the fixed package being sponsored by Renan for Resolute

- Boost1.88 is now in main for Resolute, Questing, but we need a solution for Noble/Plucky. The ultimate aim is to get this into Noble LTS where it's actually being consumed for most users. Have there been any discussions about that so far, otherwise who is going to follow that part up?

- I am unsure what the comment from Matthias about "the ocaml inclusion was a seed issue" is referring to - can you clarify? I didn't see ocaml mentioned before now.

[Renan Rodrigo (rr)]

Luciano has prepared a new patch removing the libnbd dependency; it looks good and it's uploaded now.

Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading ceph_19.2.3-0ubuntu2.dsc: done.
  Uploading ceph_19.2.3.orig.tar.xz: done.
  Uploading ceph_19.2.3-0ubuntu2.debian.tar.xz: done.
  Uploading ceph_19.2.3-0ubuntu2_source.buildinfo: done.
  Uploading ceph_19.2.3-0ubuntu2_source.changes: done.
Successfully uploaded packages.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 19.2.3-0ubuntu2

---------------
ceph (19.2.3-0ubuntu2) resolute; urgency=medium

  * d/control: remove dependency on libnbd-dev.
  * d/p/patch-out-libnbd.patch: build without libnbd.

ceph (19.2.3-0ubuntu1) questing; urgency=medium

  * New upstream stable release (LP: #2119024)
  * d/control: Update dependencies.
  * d/rules: Enable LTTNG.
  * d/p/pyo3-fix.patch: Fix issues with PyO3.
  * d/p/questing-boost-fix.patch: Fix issues with Boost on Questing.
  * d/p/ceph-exporter-systemd.patch: Removed, no longer needed.
  * d/ppy313-compat/0006-mgr-allow-specifying-module-option-level.patch: Removed, no longer needed.
  * d/p/CVE-2024-48916.patch: Removed, no longer needed.
  * No-change backport to questing.

 -- Luciano Lo Giudice <email address hidden> Wed, 22 Oct 2025 22:52:04 -0300

[Matthew Ruffell (mruffell)]

Sponsor notes for questing:

- I removed the change to d/p/boost-url.patch and synced the copy of d/p/questing-boost-fix.patch from resolute with the resulting CMakeLists.txt change for "process", and to reduce the delta from the resolute package.
- I also synced pyo3-fix.patch from resolute to pick up the commit reference change.
- Modified debian/changelog to have the new resolute entry, due to not being able to reuse the questing entry due to it already being in questing-proposed. Added LP bug reference.

I have attached a tarball of debdiffs from resolute to questing, questing-proposed to questing, and ppa to questing.

I have now sponsored questing to -unapproved. Thanks Luciano.

Uploading ceph using ftp to ubuntu
Uploading ceph_19.2.3-0ubuntu1.25.10.1.dsc
Uploading ceph_19.2.3-0ubuntu1.25.10.1.debian.tar.xz
Uploading ceph_19.2.3-0ubuntu1.25.10.1_source.buildinfo
Uploading ceph_19.2.3-0ubuntu1.25.10.1_source.changes

[Matthew Ruffell (mruffell)]

Sponsor notes for plucky:

- Changed d/changelog to add a bunch new entries.
- Synced changes to d/clean.
- d/control: changed pkgconf to pkg-config to match every other ceph release
- Synced pyo3-fix.patch from resolute
- Enabled LTTNG in d/rules, as this had been missed, after talking with Luciano.

I have attached a tarball of debdiffs from resolute to plucky, and ppa to plucky.

I have now sponsored plucky to -unapproved. Thanks Luciano.

Uploading ceph_19.2.3-0ubuntu0.25.04.1.dsc
Uploading ceph_19.2.3.orig.tar.xz
Uploading ceph_19.2.3-0ubuntu0.25.04.1.debian.tar.xz
Uploading ceph_19.2.3-0ubuntu0.25.04.1_source.buildinfo
Uploading ceph_19.2.3-0ubuntu0.25.04.1_source.changes

[Matthew Ruffell (mruffell)]

Sponsor notes for noble:

- Changed d/changelog to add a bunch new entries.
- d/control: Removed ceph-exporter, Luciano and I agreed we will SRU this later.
- Added d/p/pass-build-flags.patch to fix bug 2078244
- Synced d/p/patch-out-libnbd.patch from resolute
- Enabled LTTNG in d/rules after talking with Luciano since this was missed.
- Synced d/source/lintian-overrides from resolute.

I have attached a tarball of debdiffs from resolute to noble, and ppa to noble.

I have now sponsored noble to -unapproved. Thanks Luciano.

Uploading ceph_19.2.3-0ubuntu0.24.04.1.dsc
Uploading ceph_19.2.3.orig.tar.xz
Uploading ceph_19.2.3-0ubuntu0.24.04.1.debian.tar.xz
Uploading ceph_19.2.3-0ubuntu0.24.04.1_source.buildinfo
Uploading ceph_19.2.3-0ubuntu0.24.04.1_source.changes

[Chris Halse Rogers (raof)]

I've reviewed the packaging changes; they mostly look good, I have only a few questions/comments:

The added Build-Depends:
 liblttng-ust-dev,
 liblttng-ust1t64,

are duplicative; liblttng-ust-dev already Depends on liblttng-ust1t64. Is there a reason for explicitly pulling liblttng-ust1t64 in?

The debian/rules change:

- extraopts += -DWITH_LTTNG=OFF -DWITH_EMBEDDED=OFF
+ extraopts += -DWITH_LTTNG=ON -DWITH_EMBEDDED=ON

changes both the WITH_LTTNG and WITH_EMBEDDED settings. (This appears to be harmless because WITH_EMBEDDED appears to be an option the build system no longer recognises)

Finally, the Noble upload has a weird diff from the other uploads:

diff -Nru ceph-19.2.3/debian/patches/snapshot-upgrade-fix.patch ceph-19.2.3/debian/patches/snapshot-upgrade-fix.patch
--- ceph-19.2.3/debian/patches/snapshot-upgrade-fix.patch 2025-08-12 03:02:00.000000000 +1000
+++ ceph-19.2.3/debian/patches/snapshot-upgrade-fix.patch 2025-10-01 07:02:28.000000000 +1000
@@ -5,14 +5,15 @@
  the right order

 Reference: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2089565
-Origin: vendor
-Bug-Ubuntu: https://bugs.launchpad.net/bugs/2089565
 ---
- diff --git a/src/mds/MDSMap.cc b/src/mds/MDSMap.cc
-index cd5cb3a98a..5a68a2ef0b 100644
+ src/mds/MDSMap.cc | 120 ++++++++++++++++++++++++++++++++++++++++++++--
+ 1 file changed, 116 insertions(+), 4 deletions(-)
+
+diff --git a/src/mds/MDSMap.cc b/src/mds/MDSMap.cc
+index cd5cb3a98a..297f0da50b 100644
 --- a/src/mds/MDSMap.cc
 +++ b/src/mds/MDSMap.cc
-@@ -961,11 +961,114 @@ void MDSMap::decode(bufferlist::const_iterator& p)
+@@ -961,11 +961,123 @@ void MDSMap::decode(bufferlist::const_iterator& p)
    }

    if (ev >= 17) {
@@ -94,7 +95,10 @@
 + *
 + * Quincy 17.2.7 has neither of the 2 fields.
 + */
++ dout(20) << "ev=" << ev << dendl;
++
 + unsigned off = p.get_off();
++ dout(20) << "off=" << off << dendl;
 +
 + /*
 + * The smallest size is a single decimal digit bal_rank_mask,
@@ -108,18 +112,24 @@
 + uint8_t byte;
 + decode(byte, p);
 + p.seek(off);
-
-- if (ev >= 18) {
-- decode(max_xattr_size, p);
++
++ dout(20) << "byte=" << static_cast<int>(byte) << dendl;
++
 + if (byte < 0x10) {
++ dout(20) << "decoding in the normal squid order" << dendl;
++
 + if (ev >= 17) {
 + decode(max_xattr_size, p);
 + }
-+
+
+- if (ev >= 18) {
+- decode(max_xattr_size, p);
 + if (ev >= 18) {
 + decode(bal_rank_mask, p);
 + }
 + } else {
++ dout(20) << "decoding in the squid git snapshot order" << dendl;
++
 + if (ev >= 17) {
 + decode(bal_rank_mask, p);
 + }

However, this is not a change from the status-quo - this is not a diff introduced by this upload. It might be worth working out why the Noble version of this patch is slightly different, though.

[Matthew Ruffell (mruffell)]

RE d/p/snapshot-upgrade-fix.patch on noble:

The differences between N and P,Q,R are the dout logging statements. This was
intentionally removed between 19.2.1-0ubuntu0.24.04.1 and 19.2.1-0ubuntu0.24.04.2
in noble to fix breakage that occurs with nfs-ganesha.

https://launchpadlibrarian.net/801618016/ceph_19.2.1-0ubuntu0.24.04.1_19.2.1-0ubuntu0.24.04.2.diff.gz

The history of that patch is documented in:

https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2089565
(Entire bug)

https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2097605
(Comments 15 to 31, RE dout logging removal)

Open questions for future ceph uploads:

Do these dout statements for d/p/snapshot-upgrade-fix.patch break nfs-ganesha on
plucky, questing and resolute?

Should d/p/snapshot-upgrade-fix.patch be dropped on these releases at some
point?

Regardless, for the moment, d/p/snapshot-upgrade-fix.patch must remain as-is
on noble.

[Matthew Ruffell (mruffell)]

Hi Luciano,

Could you please answer some questions on the SRU template that I added to the [Testcase] section?

 * can you add details of what ceph-charms will be run?
 * will you do any upgrade testing from 19.2.1 to 19.2.3?
 * will you test LTTNG being enabled? How? What will you trace to check it works?
 * is it possible to verify that the data loss regression has been fixed?

Christopher and I are mostly interested in upgrade testing and how LTTNG will be tested.

If you could adjust the [Testcase] section with your plan, that would help a lot.

[Luciano Lo Giudice (lmlogiudice)]

Hello Matthew and Chris,

I've replied to the questions in the original message. To wit:

- All of our Ceph charms will be tested. These are: ceph-mon, ceph-osd, ceph-radosgw, ceph-fs, ceph-nfs, ceph-nvme, ceph-proxy, ceph-rbd-mirror, ceph-dashboard.

- Our test suite includes upgrade testing from the current stable version (19.2.1) to proposed (19.2.3). We also perform integration testing with Openstack charms.

- We will test it manually. We will deploy a Ceph cluster with 19.2.3, then enable tracing to make sure it works. Afterwards, we will disable it to make sure no more tracing output is produced and that everything works as normal (Tracing is enabled/disabled via a simple config option).

- It's not possible to test this reliably as this data loss was caused under very hard to reproduce circumstances. We can leave radosgw running for a bit and verify that no data loss happened, but that would not imply that the bug is no longer present (FWIW, the ceph-radosgw tests already do something like this).

[Luciano Lo Giudice (lmlogiudice)]

I've edited the original message with a more detailed testplan for the LTTNG feature.

[Chris Halse Rogers (raof)]

Hello Ponnuvel, or anyone else affected,

Accepted ceph into questing-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/19.2.3-0ubuntu1.25.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-questing to verification-done-questing. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-questing. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Ponnuvel, or anyone else affected,

Accepted ceph into plucky-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/19.2.3-0ubuntu0.25.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-plucky to verification-done-plucky. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-plucky. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Ponnuvel, or anyone else affected,

Accepted ceph into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/19.2.3-0ubuntu0.24.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ceph (19.2.3-0ubuntu0.25.04.1) for plucky have finished running.
The following regressions have been reported in tests triggered by the package:

qemu/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/plucky/update_excuses.html#ceph

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Christian Rishøj (christian-rishoj)]

Would like to verify that the Ceph dashboard is working with 19.2.3, but we using the microceph snap, not the apt package directly.

Is a "proposed" update channel available for snaps?

Apologies for my ignorance.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ceph (19.2.3-0ubuntu1.25.10.1) for questing have finished running.
The following regressions have been reported in tests triggered by the package:

cinder/2:27.0.0-0ubuntu1 (amd64)
libvirt/unknown (amd64)
qemu/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/questing/update_excuses.html#ceph

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Wesley Hershberger (whershberger)]

Hi Christian - the proposed packages unfortunately aren't available in the microceph snap channels until after the package has been released to -updates. It seems reasonable to request that feature in the MicroCeph bugtracker [1].

[1] https://github.com/canonical/microceph/issues

[Luciano Lo Giudice (lmlogiudice)]

@christian-rishoj: Microceph will be tested with these packages before a new release, but not on this bug report, but rather as part of the regular process.

For SRU reviewers: The issues with autopkgtests seems to have been spurious and no further issues are showing up after the riscv packages have been built.

[Luciano Lo Giudice (lmlogiudice)]

Hello,
The whole test-suite for noble-proposed has completed and all tests have passed. The results can be seen here: https://github.com/canonical/ceph-charms/actions/runs/19547255675?pr=120
(Please ignore the COS integration test, it isn't related to this SRU and it's been broken for a while now).

As for the tests regarding LTTNG, the following procedure was used:

- After deploying a Ceph cluster with 19.2.3, we install the lttng binaries:
> sudo apt install lttng-tools

- Next, we run the lttng daemons:
> lttng-sessiond --daemonize

- We can then list the available trace points:
> lttng list -u

  UST events:
  -------------

  PID: 14531 - Name: /usr/bin/ceph-osd

  ust_baddr_statedump:soinfo (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  mutex:lock_exit (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  mutex:lock_enter (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
  osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)

- We can now create a session, and begin tracing:

> lttng create
  Session auto-31240637-925731 created.

> lttng enable-event -u -a
  All UST events are enabled in channel channel0

> lttng start
  Tracing started for session auto-31240637-925731

- Next, we'll generate some activity on the Ceph cluster. We'll use `rados bench` but any other type of I/O over objects can be applied:

> sudo rados -p data bench 10 write

- After the above is done, we'll stop tracing and verify the output:

> lttng stop
  Waiting for data availability.
  Tracing stopped for session auto-31240637-925731

> lttng view | head
[18:42:17.027535265] (+0.000001786) mutex:lock_enter: { cpu_id = 0 }, { addr = 0x1fc2408, name = "SimpleMessenger::DispatchQeueu::lock" }
[18:42:17.027535846] (+0.000000581) mutex:lock_exit: { cpu_id = 0 }, { addr = 0x1fc2408, name = "SimpleMessenger::DispatchQeueu::lock" }
[18:42:23.027538829] (+0.000000576) mutex:lock_exit: { cpu_id = 0 }, { addr = 0x205a610, name = "OSD::recovery_tp::lock" }

Finally, we can note that if we disable tracing and run the `rados bench` program again, no tracing output will be generated.

[Guillaume Boutry (gboutry)]

Hello Ponnuvel, or anyone else affected,

Accepted ceph into caracal-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:caracal-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-caracal-needed to verification-caracal-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-caracal-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Luciano Lo Giudice (lmlogiudice)]

Our infrastructure wrt Github runners is having issues with connectivity, so I'm running the verification myself. I've run the entire functional test suite locally and all the tests passed. I'm pasting the results of one such run and attaching the full logs as proof, but I can do the same for all the tests run:

2025-11-26 16:16:23 [INFO] Events:
  Deploy Bundle:
    Start: 1764183921.2315564
    Finish: 1764183934.9559
    Elapsed Time: 13.724343538284302
    PCT Of Run Time: 3
  Prepare Environment:
    Start: 1764183920.6793232
    Finish: 1764183921.2313614
    Elapsed Time: 0.5520381927490234
    PCT Of Run Time: 1
  Test tests.target.CephAuthTest:
    Start: 1764184540.159873
    Finish: 1764184548.451297
    Elapsed Time: 8.291424036026001
    PCT Of Run Time: 2
  Test tests.target.CephLowLevelTest:
    Start: 1764184456.572862
    Finish: 1764184478.533339
    Elapsed Time: 21.960477113723755
    PCT Of Run Time: 4
  Test tests.target.CephMonActionsTest:
    Start: 1764184548.4513416
    Finish: 1764184566.2094848
    Elapsed Time: 17.758143186569214
    PCT Of Run Time: 3
  Test tests.target.CephMonJujuPersistent:
    Start: 1764184573.6686642
    Finish: 1764184583.0324569
    Elapsed Time: 9.363792657852173
    PCT Of Run Time: 2
  Test tests.target.CephMonKeyRotationTests:
    Start: 1764184566.2095222
    Finish: 1764184573.6686409
    Elapsed Time: 7.459118604660034
    PCT Of Run Time: 2
  Test tests.target.CephPrometheusTest:
    Start: 1764184540.158875
    Finish: 1764184540.1598554
    Elapsed Time: 0.000980377197265625
    PCT Of Run Time: 1
  Test tests.target.CephTest:
    Start: 1764184478.5333674
    Finish: 1764184540.158836
    Elapsed Time: 61.625468492507935
    PCT Of Run Time: 10
  Test zaza.charm_tests.lifecycle.refresh.CharmRefreshAll:
    Start: 1764184456.5684538
    Finish: 1764184456.5728464
    Elapsed Time: 0.0043926239013671875
    PCT Of Run Time: 1
  Wait for Deployment:
    Start: 1764183934.9559112
    Finish: 1764184456.5666878
    Elapsed Time: 521.6107766628265
    PCT Of Run Time: 79
Metadata: {}

The Ceph version matches the expectations:

> lmlg@lmlg-pc:~/src/ceph-charms/ceph-mon$ juju ssh ceph-mon/0 -- sudo ceph --version
ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)

I've additionally repeated the process for testing LTTNG and got similar (i.e: positive) results.

[Christian Ehrhardt (paelzer)]

@Luciano

Since this blocks to release a qemu SRU could could you please also verify this on questring and plucky?

And I think also the actual release on noble would be blocked by verifying Q&P as well.

[Luciano Lo Giudice (lmlogiudice)]

@Christian,

Yes, of course. It's been delayed due to issues with our infrastructure.

[Luciano Lo Giudice (lmlogiudice)]

Hello,
The whole test-suite for plucky-proposed has completed and all tests have passed. The results can be seen here: https://github.com/canonical/ceph-charms/actions/runs/19940696570/job/57178742811?pr=126

I've also run the LTTNG tests by hand on plucky and got similar results to the ones in noble.

[Luciano Lo Giudice (lmlogiudice)]

Hello,
The whole test suite for questing-proposed has completed and all tests have passed. The results can be seen here: https://github.com/canonical/ceph-charms/actions/runs/20104215871/job/57684601905?pr=128

The LTTNG tests were run by hand similarly to how they were done for Noble/Plucky and they have also succeeded.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 19.2.3-0ubuntu1.25.10.1

---------------
ceph (19.2.3-0ubuntu1.25.10.1) questing; urgency=medium

  * d/control: remove dependency on libnbd-dev.
  * d/p/patch-out-libnbd.patch: build without libnbd.
  * No-change backport to questing. (LP: #2119024)

 -- Luciano Lo Giudice <email address hidden> Wed, 22 Oct 2025 22:52:04 -0300

[Nick Rosbrook (enr0n)]

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 19.2.3-0ubuntu0.25.04.1

---------------
ceph (19.2.3-0ubuntu0.25.04.1) plucky; urgency=medium

  * New upstream stable release (LP: #2119024)
  * d/control: Update dependencies.
  * d/rules: Enable LTTNG.
  * d/p/pyo3-fix.patch: Fix issues with PyO3.
  * d/p/ceph-exporter-systemd.patch: Removed, no longer needed.
  * d/p/py313-compat/0006-mgr-allow-specifying-module-option-level.patch: Removed, no longer needed.
  * d/p/CVE-2024-48916.patch: Removed, no longer needed.
  * d/p/patch-out-libnbd.patch: build without libnbd.
  * d/clean: Add btrfs and dashboard removals.
  * Backported from resolute to plucky. Differences:
  * d/p/questing-boost-fix.patch: Removed, not needed on plucky.

 -- Luciano Lo Giudice <email address hidden> Thu, 30 Oct 2025 21:49:13 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 19.2.3-0ubuntu0.24.04.1

---------------
ceph (19.2.3-0ubuntu0.24.04.1) noble; urgency=medium

  [ Luciano Lo Giudice ]
  * New upstream stable release (LP: #2119024)
  * d/control: Update dependencies.
  * d/rules: Enable LTTNG.
  * d/p/pyo3-fix.patch: Fix issues with PyO3.
  * d/p/CVE-2024-48916.patch: Removed, no longer needed.
  * d/p/patch-out-libnbd.patch: build without libnbd.
  * d/clean: Add btrfs and dashboard removals.
  * d/source/lintian-overrides: Fixup changed directory.
  * Backported from resolute to noble. Differences:
  * d/p/questing-boost-fix.patch: Removed, not needed on noble.
  * d/p/py313-compat/*.patch: Removed, not needed on noble.

  [ Graham Inggs ]
  * d/p/pass-build-flags.patch: Pass buildflags to Python extensions
    (LP: #2078244).

 -- Luciano Lo Giudice <email address hidden> Wed, 29 Oct 2025 22:09:45 +0000

[Nick Rosbrook (enr0n)]

As a side note - it would be nice if someone would just change d/control to stop trying to build ceph on armhf everywhere. It's depwait eveywhere, which creates just enough noise in the pending-sru report so that at first glance it looks unreleaseable.

In reality, it has been this way a long time, so the fact that it is still depwait in -proposed is not a regression.

[Edward Hope-Morley (hopem)]

@lmlogiudice can you please run your test suite against the Caracal Ubuntu Cloud Archive as we also need that validated in order to release. Thanks!

[Edward Hope-Morley (hopem)]

ok I'm going to assume that your tests run in https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2119024/comments/37 are run against Caracal UCA so will take that as a green light.

[Edward Hope-Morley (hopem)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Edward Hope-Morley (hopem)]

This bug was fixed in the package ceph - 19.2.3-0ubuntu0.24.04.1~cloud0
---------------

 ceph (19.2.3-0ubuntu0.24.04.1~cloud0) jammy-caracal; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (19.2.3-0ubuntu0.24.04.1) noble; urgency=medium
 .
   [ Luciano Lo Giudice ]
   * New upstream stable release (LP: #2119024)
   * d/control: Update dependencies.
   * d/rules: Enable LTTNG.
   * d/p/pyo3-fix.patch: Fix issues with PyO3.
   * d/p/CVE-2024-48916.patch: Removed, no longer needed.
   * d/p/patch-out-libnbd.patch: build without libnbd.
   * d/clean: Add btrfs and dashboard removals.
   * d/source/lintian-overrides: Fixup changed directory.
   * Backported from resolute to noble. Differences:
   * d/p/questing-boost-fix.patch: Removed, not needed on noble.
   * d/p/py313-compat/*.patch: Removed, not needed on noble.
 .
   [ Graham Inggs ]
   * d/p/pass-build-flags.patch: Pass buildflags to Python extensions
     (LP: #2078244).