[SRU] Ceph Squid release candidate

[James Page]

[Impact]
Ubuntu 24.04 LTS users are currently using a development snapshot of Ceph; the upstream project are working towards a release and produced a 19.1.0 RC which we should include in 24.04 as a stepping stone towards release. There are current know upgrade issues with the development snapshot included.

[Test Case]
No reproducer for this bug; RC will be tested in the following ways:

- As part of a core24 based microceph built from proposed
- Using the standard OpenStack Regression Testing utilised for validation of OpenStack and Ceph updates under the MRE already in place:

https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates

Test results will all be attached to this bug report.

[Where things might go wrong]
As this is an RC there will likely be outstanding bugs that will need to be resolved by the upstream project before release; This is expected but the RC will be an improvement on the current development snapshot in terms of quality.

[Rgpublic (rgpublic)]

Existing Ceph nodes on Ubuntu 23.10 cannot be upgraded due to the broken Ceph version currently in Noble:

https://tracker.ceph.com/issues/64440

It seems this was fixed at the end of March, but the GIT version in Noble is from March 1st. Considering that the support for Mantic is running out soon and there's no release of Ceph Reef for Noble that leaves all Ceph users in very dire straits, unfortunately. I don't know why it was decided to deliver a broken Ceph version with an LTS release, but at least please consider publishing another more recent in-between release so users have a decent upgrade path for now.

[James Page (james-page)]

@rgpublic the decision to target squid was made after discussion with the upstream Ceph project however the RC (which is now out) and subsequent release have taken long to appear than originally planned for.

We should have the RC into a PPA for testing in the next two weeks.

[Rgpublic (rgpublic)]

@james-page: I see. Thanks a lot for the update. Very reassuring to have at least a ballpark timeframe. We're obviously waiting very anxiously, because Mantic support is running out on July 11th.

[James Page (james-page)]

@rgpublic the short support periods of the non-LTS releases does make this
challenging which is why most production deployments use a LTS release - we
provide a fully supported Ceph Reef via the Ubuntu Cloud Archive for
OpenStack Bobcat which can be used on Ubuntu 22.04 LTS.

On Tue, Jul 9, 2024 at 5:35 PM Rgpublic <email address hidden> wrote:

> @james-page: I see. Thanks a lot for the update. Very reassuring to have
> at least a ballpark timeframe. We're obviously waiting very anxiously,
> because Mantic support is running out on July 11th.
>
> --
> You received this bug notification because you are a member of Ubuntu
> OpenStack, which is subscribed to ceph in Ubuntu.
> https://bugs.launchpad.net/bugs/2065515
>
> Title:
> [SRU] Ceph Squid release
>
> Status in ceph package in Ubuntu:
> Triaged
> Status in ceph source package in Noble:
> Triaged
> Status in ceph source package in Oracular:
> Triaged
>
> Bug description:
> TBC
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2065515/+subscriptions
>
>
> --
> Ubuntu-openstack-bugs mailing list
> <email address hidden>
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-openstack-bugs
>

[James Page (james-page)]

RC tarball: https://download.ceph.com/tarballs/ceph_19.1.0.orig.tar.gz

[Rgpublic (rgpublic)]

@james-page: Thanks! I see where you're coming from. Just let me point out: I understand that Ceph is targeting for the most part rather huge installations where Ceph is running on its own servers or in a Docker container and that's what the Ceph folks suggest to use. And I certainly understand that that's the more "exicting" part of Ceph's abilities. Development-wise as well as business-wise. And it's amazing that Ceph is able to run on such huge installations. And while it certainly is more efficient to run Ceph like that, please don't forget us SMEs completely.

For us, every single rather miniscule complexity that's added means more costs. For us, running Ceph and the web services on just two nodes, connected via crosslink means: A very efficient and comfortable way to have a small server but still have failover. Fact is: Nothing comes even close to the performance, stability and reliability of Ceph for that. No Gluster, no NFS - nothing. And when everything is on one the same server, we often need to be able to use a recent ubuntu version Apache, PHP etc. Ceph packages are in fact available on Ubuntu Mantic and you allow a do-release-upgrade on 23.04 even with Ceph packages installed.

So this is obviously not sth. Canonical totally blocks. And over the years, this worked just perfectly. The problem arose when there were buggy GIT packages delivered with the Update without any warning - crashing the installation. I *almost* got the Reef packages running on Noble by using lots of APT-ignores but eventually failed only with the MGR service due to Python dependency hell. So, I guess the whole problem could have been easily avoided if you just provided Reef packages for 24.04 until the Squid packages arrive. I might be wrong, but I assume the infrastructure to create these packages should be there. I'm a developer myself, though, so I understand that things sometimes just don't run as planned. So, "no front", just wanted to add another perspective on this issue.

[James Page (james-page)]

Packages for noble and oracular can be found in the following PPA's:

https://launchpad.net/~james-page/+archive/ubuntu/caracal (noble)
https://launchpad.net/~james-page/+archive/ubuntu/dalmation (oracular)

This is based on the 19.1.0 RC released last week.

[James Page (james-page)]

Deploy test for Noble packages:

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
ceph-noble-testing serverstack3.5 serverstack/serverstack 3.5.1 unsupported 10:17:11Z

App Version Status Scale Charm Channel Rev Exposed Message
ceph-mon 19.2.0~is.re... active 3 ceph-mon squid/candidate 226 no Unit is ready and clustered
ceph-osd 19.2.0~is.re... active 3 ceph-osd squid/candidate 601 no Unit is ready (3 OSD)
ceph-radosgw 19.2.0~is.re... active 3 ceph-radosgw squid/candidate 582 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
ceph-mon/0 active idle 0 10.5.3.236 Unit is ready and clustered
ceph-mon/1* active idle 1 10.5.3.137 Unit is ready and clustered
ceph-mon/2 active idle 2 10.5.3.30 Unit is ready and clustered
ceph-osd/0* active idle 3 10.5.0.82 Unit is ready (3 OSD)
ceph-osd/1 active idle 4 10.5.1.235 Unit is ready (3 OSD)
ceph-osd/2 active idle 5 10.5.2.119 Unit is ready (3 OSD)
ceph-radosgw/0 active idle 6 10.5.3.97 80/tcp Unit is ready
ceph-radosgw/1* active idle 7 10.5.0.193 80/tcp Unit is ready
ceph-radosgw/2 active idle 8 10.5.2.247 80/tcp Unit is ready

Machine State Address Inst id Base AZ Message
0 started 10.5.3.236 a0698994-40b0-4bb1-9a8e-942cf5bd77a9 ubuntu@24.04 nova ACTIVE
1 started 10.5.3.137 bee9b154-ced2-4e88-8c63-16215684d42b ubuntu@24.04 nova ACTIVE
2 started 10.5.3.30 f1f2f888-167b-4018-a97a-3da00b22dd3d ubuntu@24.04 nova ACTIVE
3 started 10.5.0.82 6e3021b1-6fd7-49bc-b4c3-3ceb92341f8f ubuntu@24.04 nova ACTIVE
4 started 10.5.1.235 692d46c8-7b6f-4bef-a312-29c66f228258 ubuntu@24.04 nova ACTIVE
5 started 10.5.2.119 b8eef6ed-d74c-4f36-a841-682b82db0055 ubuntu@24.04 nova ACTIVE
6 started 10.5.3.97 bdde8083-3f09-4a00-a780-4b80aafad0ee ubuntu@24.04 nova ACTIVE
7 started 10.5.0.193 4855a405-6970-4d08-aa85-36f111748fb6 ubuntu@24.04 nova ACTIVE
8 started 10.5.2.247 fa676a4d-bd73-42ca-b6e3-da50ab21e59d ubuntu@24.04 nova ACTIVE

[James Page (james-page)]

Upload for noble blocked until the current SRU completed (all bugs are now verified); update for oracular uploaded today.

[James Page (james-page)]

Upload for Oracular is currently blocked in proposed pending a binary only promotion for a boost library dependency.

Upload for Noble in the UNAPPROVED queue for the SRU team to review; we'd like to release the RC to Noble to improve the current snapshot while we wait for the final release of Ceph Squid.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 19.2.0~is.really.19.1.0-0ubuntu2

---------------
ceph (19.2.0~is.really.19.1.0-0ubuntu2) oracular; urgency=medium

  * d/p/dout-fix.patch: Fix build issues with gcc-14.

 -- Luciano Lo Giudice <email address hidden> Fri, 26 Jul 2024 13:55:13 +0000

[Chris Halse Rogers (raof)]

(SRU team hat)I've a couple of questions:

1. How long until we can expect a full ceph release? We've already missed the “people upgrading from 23.10 get a broken upgrade” window; should we be publishing an RC release now, rather than waiting for those bugs to be fixed and publishing the final release once it's ready?

2. What are the c++23-compatibility patches for? The build system doesn't seem to be switching to using the c++23 standard?

Not a question, but a comment: 19.2.0~is.really.19.1.0-0ubuntu0.24.04.1 doesn't seem like the right version string, as it's neither 19.2.0 nor is it really 19.1.0

[James Page (james-page)]

@raof

1) the million dollar question - I would expect in the next month or so based on prior experience. When we discussed as a team we agreed that we should push forward with the RC version now; its an improvement on the snapshot in archive already (which has some know bugs for upgraders running Ceph).

2) the two patches picked (logentry and snapmapper) where required to resolve FTBFS in this release - potentially in Oracular due to a different g++ version; we did all of the initial packaging in the development release and then backported to noble for the SRU so its possible they may not be required (I'd have to go build check without them to confirm either way).

Version comment - the internal version is 19.1.0 but we'd already managed to burn 19.2.0~ with a snapshot earlier in the year and I really didn't want to bump the epoch on the package to resolve - 19.2.0 will be the Squid release version. This gives us progression in versioning without the need for the epoch but I'm open to other suggestions on how to manage this.

[James Page (james-page)]

The upstream project announced a second release candidate yesterday - will discuss as to whether we should rebase the SRU, or proceed as is with the Ceph team.

[Chris Halse Rogers (raof)]

So, do we have a conclusion on whether to do a new RC or not?

Re: versions - oh, so this is a 19.2.0 RC, that internally reports as 19.1.0? Weird. I think that something like 19.2.0~is.really.19.2.0rc2 (or whatever the actual reported version is) would work, but really that's cosmetic.

[James Page (james-page)]

@raof - to confirm versioning - upstream use the odd minor release version of x.1.x for Release Candidates, with the final release being the first x.2.0 - the internal version reported in the update would be 19.1.0.

I think its worth bumping to 19.1.1 to pickup the fixes included.

[James Page (james-page)]

I've prepared and uploaded the second release candidate for the SRU team to review; I've tweak the upstream versioning for the package to 19.2.0~rc2 for the upstream version (although internally it will report 19.1.1 which is correct).

We're closely monitoring upstream for final releases - testing is underway but that is an extended process and may need to be repeated depending on any release blocking bugs discovered upstream.

[Chris Halse Rogers (raof)]

Hello James, or anyone else affected,

Accepted ceph into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/19.2.0~rc2-0ubuntu0.24.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Peter Sabaini (peter-sabaini)]

Seeing an error in the ceph-volume tool, reported as bug #2080208 (and upstream https://tracker.ceph.com/issues/68032)