[SRU] Ceph Squid release candidate

[James Page]

[Impact]
Ubuntu 24.04 LTS users are currently using a development snapshot of Ceph; the upstream project are working towards a release and produced a 19.1.0 RC which we should include in 24.04 as a stepping stone towards release. There are current know upgrade issues with the development snapshot included.

[Test Case]
No reproducer for this bug; RC will be tested in the following ways:

- As part of a core24 based microceph built from proposed
- Using the standard OpenStack Regression Testing utilised for validation of OpenStack and Ceph updates under the MRE already in place:

https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates

Test results will all be attached to this bug report.

[Where things might go wrong]
As this is an RC there will likely be outstanding bugs that will need to be resolved by the upstream project before release; This is expected but the RC will be an improvement on the current development snapshot in terms of quality.

[Rgpublic (rgpublic)]

Existing Ceph nodes on Ubuntu 23.10 cannot be upgraded due to the broken Ceph version currently in Noble:

https://tracker.ceph.com/issues/64440

It seems this was fixed at the end of March, but the GIT version in Noble is from March 1st. Considering that the support for Mantic is running out soon and there's no release of Ceph Reef for Noble that leaves all Ceph users in very dire straits, unfortunately. I don't know why it was decided to deliver a broken Ceph version with an LTS release, but at least please consider publishing another more recent in-between release so users have a decent upgrade path for now.

[James Page (james-page)]

@rgpublic the decision to target squid was made after discussion with the upstream Ceph project however the RC (which is now out) and subsequent release have taken long to appear than originally planned for.

We should have the RC into a PPA for testing in the next two weeks.

[Rgpublic (rgpublic)]

@james-page: I see. Thanks a lot for the update. Very reassuring to have at least a ballpark timeframe. We're obviously waiting very anxiously, because Mantic support is running out on July 11th.

[James Page (james-page)]

@rgpublic the short support periods of the non-LTS releases does make this
challenging which is why most production deployments use a LTS release - we
provide a fully supported Ceph Reef via the Ubuntu Cloud Archive for
OpenStack Bobcat which can be used on Ubuntu 22.04 LTS.

On Tue, Jul 9, 2024 at 5:35 PM Rgpublic <email address hidden> wrote:

> @james-page: I see. Thanks a lot for the update. Very reassuring to have
> at least a ballpark timeframe. We're obviously waiting very anxiously,
> because Mantic support is running out on July 11th.
>
> --
> You received this bug notification because you are a member of Ubuntu
> OpenStack, which is subscribed to ceph in Ubuntu.
> https://bugs.launchpad.net/bugs/2065515
>
> Title:
> [SRU] Ceph Squid release
>
> Status in ceph package in Ubuntu:
> Triaged
> Status in ceph source package in Noble:
> Triaged
> Status in ceph source package in Oracular:
> Triaged
>
> Bug description:
> TBC
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2065515/+subscriptions
>
>
> --
> Ubuntu-openstack-bugs mailing list
> <email address hidden>
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-openstack-bugs
>

[James Page (james-page)]

RC tarball: https://download.ceph.com/tarballs/ceph_19.1.0.orig.tar.gz

[Rgpublic (rgpublic)]

@james-page: Thanks! I see where you're coming from. Just let me point out: I understand that Ceph is targeting for the most part rather huge installations where Ceph is running on its own servers or in a Docker container and that's what the Ceph folks suggest to use. And I certainly understand that that's the more "exicting" part of Ceph's abilities. Development-wise as well as business-wise. And it's amazing that Ceph is able to run on such huge installations. And while it certainly is more efficient to run Ceph like that, please don't forget us SMEs completely.

For us, every single rather miniscule complexity that's added means more costs. For us, running Ceph and the web services on just two nodes, connected via crosslink means: A very efficient and comfortable way to have a small server but still have failover. Fact is: Nothing comes even close to the performance, stability and reliability of Ceph for that. No Gluster, no NFS - nothing. And when everything is on one the same server, we often need to be able to use a recent ubuntu version Apache, PHP etc. Ceph packages are in fact available on Ubuntu Mantic and you allow a do-release-upgrade on 23.04 even with Ceph packages installed.

So this is obviously not sth. Canonical totally blocks. And over the years, this worked just perfectly. The problem arose when there were buggy GIT packages delivered with the Update without any warning - crashing the installation. I *almost* got the Reef packages running on Noble by using lots of APT-ignores but eventually failed only with the MGR service due to Python dependency hell. So, I guess the whole problem could have been easily avoided if you just provided Reef packages for 24.04 until the Squid packages arrive. I might be wrong, but I assume the infrastructure to create these packages should be there. I'm a developer myself, though, so I understand that things sometimes just don't run as planned. So, "no front", just wanted to add another perspective on this issue.

[James Page (james-page)]

Packages for noble and oracular can be found in the following PPA's:

https://launchpad.net/~james-page/+archive/ubuntu/caracal (noble)
https://launchpad.net/~james-page/+archive/ubuntu/dalmation (oracular)

This is based on the 19.1.0 RC released last week.

[James Page (james-page)]

Deploy test for Noble packages:

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
ceph-noble-testing serverstack3.5 serverstack/serverstack 3.5.1 unsupported 10:17:11Z

App Version Status Scale Charm Channel Rev Exposed Message
ceph-mon 19.2.0~is.re... active 3 ceph-mon squid/candidate 226 no Unit is ready and clustered
ceph-osd 19.2.0~is.re... active 3 ceph-osd squid/candidate 601 no Unit is ready (3 OSD)
ceph-radosgw 19.2.0~is.re... active 3 ceph-radosgw squid/candidate 582 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
ceph-mon/0 active idle 0 10.5.3.236 Unit is ready and clustered
ceph-mon/1* active idle 1 10.5.3.137 Unit is ready and clustered
ceph-mon/2 active idle 2 10.5.3.30 Unit is ready and clustered
ceph-osd/0* active idle 3 10.5.0.82 Unit is ready (3 OSD)
ceph-osd/1 active idle 4 10.5.1.235 Unit is ready (3 OSD)
ceph-osd/2 active idle 5 10.5.2.119 Unit is ready (3 OSD)
ceph-radosgw/0 active idle 6 10.5.3.97 80/tcp Unit is ready
ceph-radosgw/1* active idle 7 10.5.0.193 80/tcp Unit is ready
ceph-radosgw/2 active idle 8 10.5.2.247 80/tcp Unit is ready

Machine State Address Inst id Base AZ Message
0 started 10.5.3.236 a0698994-40b0-4bb1-9a8e-942cf5bd77a9 ubuntu@24.04 nova ACTIVE
1 started 10.5.3.137 bee9b154-ced2-4e88-8c63-16215684d42b ubuntu@24.04 nova ACTIVE
2 started 10.5.3.30 f1f2f888-167b-4018-a97a-3da00b22dd3d ubuntu@24.04 nova ACTIVE
3 started 10.5.0.82 6e3021b1-6fd7-49bc-b4c3-3ceb92341f8f ubuntu@24.04 nova ACTIVE
4 started 10.5.1.235 692d46c8-7b6f-4bef-a312-29c66f228258 ubuntu@24.04 nova ACTIVE
5 started 10.5.2.119 b8eef6ed-d74c-4f36-a841-682b82db0055 ubuntu@24.04 nova ACTIVE
6 started 10.5.3.97 bdde8083-3f09-4a00-a780-4b80aafad0ee ubuntu@24.04 nova ACTIVE
7 started 10.5.0.193 4855a405-6970-4d08-aa85-36f111748fb6 ubuntu@24.04 nova ACTIVE
8 started 10.5.2.247 fa676a4d-bd73-42ca-b6e3-da50ab21e59d ubuntu@24.04 nova ACTIVE

[James Page (james-page)]

Upload for noble blocked until the current SRU completed (all bugs are now verified); update for oracular uploaded today.

[James Page (james-page)]

Upload for Oracular is currently blocked in proposed pending a binary only promotion for a boost library dependency.

Upload for Noble in the UNAPPROVED queue for the SRU team to review; we'd like to release the RC to Noble to improve the current snapshot while we wait for the final release of Ceph Squid.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 19.2.0~is.really.19.1.0-0ubuntu2

---------------
ceph (19.2.0~is.really.19.1.0-0ubuntu2) oracular; urgency=medium

  * d/p/dout-fix.patch: Fix build issues with gcc-14.

 -- Luciano Lo Giudice <email address hidden> Fri, 26 Jul 2024 13:55:13 +0000