ceph-volume needs "packaging" and "ceph" modules

[Peter Sabaini]

[ Impact ]
ceph-volume tool is not usable directly after install due to a missing dependencies.

[ Test Plan ]
sudo apt install ceph-volume
ceph-volume --help

[ Where problems could occur ]
The missing packaging module is immediately obvious - the ceph dependency less so as ceph-volume is usually installed with ceph-osd (which already pulls this in). Direct users of ceph-volume will get new depends pulled in.

[ Original Bug Report ]
The ceph-volume program needs python3-packaging but it looks like we're not installing it in jammy-caracal

https://github.com/ceph/ceph/pull/54423/commits/0985e201342fa53c014a811156aed661b4b8f994
https://openstack-ci-reports.ubuntu.com/artifacts/dcf/917920/4/check/jammy-caracal/dcf9973/index.html

Traceback excerpt:

2024-05-02 19:31:54.624912 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 INFO unit.ceph-osd/0.juju-log server.go:316 mon:27: osdize cmd: ['ceph-volume', 'lvm', 'create', '--osd-fsid', 'aef29aff-df24-4bb8-bfb3-bcd607761b2e', '--bluestore', '--data', 'ceph-aef29aff-df24-4bb8-bfb3-bcd607761b2e/osd-block-aef29aff-df24-4bb8-bfb3-bcd607761b2e']
2024-05-02 19:31:54.624972 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 Traceback (most recent call last):
2024-05-02 19:31:54.624990 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "/usr/sbin/ceph-volume", line 33, in <module>
2024-05-02 19:31:54.625002 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')())
2024-05-02 19:31:54.625014 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "/usr/sbin/ceph-volume", line 25, in importlib_load_entry_point
2024-05-02 19:31:54.625189 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 return next(matches).load()
2024-05-02 19:31:54.625210 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
2024-05-02 19:31:54.625222 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 module = import_module(match.group('module'))
2024-05-02 19:31:54.625234 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
2024-05-02 19:31:54.625247 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 return _bootstrap._gcd_import(name[level:], package, level)
2024-05-02 19:31:54.625259 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
2024-05-02 19:31:54.625541 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
2024-05-02 19:31:54.625598 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
2024-05-02 19:31:54.625611 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
2024-05-02 19:31:54.625622 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "<frozen importlib._bootstrap_external>", line 883, in exec_module
2024-05-02 19:31:54.625633 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2024-05-02 19:31:54.625975 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 9, in <module>
2024-05-02 19:31:54.626007 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 from ceph_volume import log, devices, configuration, conf, exceptions, terminal, inventory, drive_group, activate
2024-05-02 19:31:54.626057 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "/usr/lib/python3/dist-packages/ceph_volume/devices/__init__.py", line 1, in <module>
2024-05-02 19:31:54.626072 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 from . import lvm, simple, raw # noqa
2024-05-02 19:31:54.626280 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/__init__.py", line 1, in <module>
2024-05-02 19:31:54.626313 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 from .main import LVM # noqa
2024-05-02 19:31:54.626327 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 4, in <module>
2024-05-02 19:31:54.626339 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 from . import activate
2024-05-02 19:31:54.626359 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/activate.py", line 9, in <module>
2024-05-02 19:31:54.626371 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 from ceph_volume.util import encryption as encryption_utils
2024-05-02 19:31:54.626465 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 File "/usr/lib/python3/dist-packages/ceph_volume/util/encryption.py", line 9, in <module>
2024-05-02 19:31:54.626490 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 from packaging import version
2024-05-02 19:31:54.626502 | focal-medium | 2024-05-02 19:31:54 [ERROR] unit-ceph-osd-0.log: 2024-05-02 19:31:52 WARNING unit.ceph-osd/0.mon-relation-changed logger.go:60 ModuleNotFoundError: No module named 'packaging'

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Wesley Hershberger (whershberger)]

This also affects ceph-volume 19.2.0~git20240301.4c76c50-0ubuntu6 in Noble.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/reef
Review: https://review.opendev.org/c/openstack/charm-ceph-osd/+/919794

[Peter Sabaini (peter-sabaini)]

The dependency on python3-packaging was only introduced in squid, so marking reef as invalid

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Felipe Reyes <email address hidden>" on branch: stable/reef
Review: https://review.opendev.org/c/openstack/charm-ceph-osd/+/919794

[James Page (james-page)]

Note that ceph in oracular FTBFS due to a broken API in the snappy package which is pending a transition across Debian and Ubuntu.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Felipe Reyes <email address hidden>" on branch: stable/reef
Review: https://review.opendev.org/c/openstack/charm-ceph-osd/+/919794

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 19.2.0~git20240301.4c76c50-0ubuntu7

---------------
ceph (19.2.0~git20240301.4c76c50-0ubuntu7) oracular; urgency=medium

  [ Luciano Lo Giudice]
  * d/control: Add python3-{packaging,ceph-common} to (Build-)Depends
    as these are undocumented/detected runtime dependencies in
    ceph-volume (LP: #2064717).

  [ James Page ]
  * d/cephadm.install: Install cephadmlib Python module which the
    cephadm script uses (LP: #2063456).
  * d/control: cephadm linux-any -> all.
  * d/gbp.conf: Update for ubuntu/oracular packaging branch.
  * d/control: Update Vcs-* to point to Launchpad for Ubuntu packaging.
  * d/p/mgr-distutils.patch: Directly use vendored distutils from
    setuptools for Python that runs in the mgr daemon (LP: #2065867).

 -- James Page <email address hidden> Thu, 23 May 2024 16:30:19 +0100

[Timo Aaltonen (tjaalton)]

An upload of ceph to noble-proposed has been rejected from the upload queue for the following reason: "will be amended".

[Timo Aaltonen (tjaalton)]

Hello Peter, or anyone else affected,

Accepted ceph into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/19.2.0~git20240301.4c76c50-0ubuntu6.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Vadim Mishustin (vvak)]

Hello. I'm testing the proposed 6.1 package on Noble.
ceph-volume, ceph-osd, ceph-mgr, ceph-mon, ceph-radosgw work well. Thank you.

[James Page (james-page)]

# ceph-volume --help
usage: ceph-volume [-h] [--cluster CLUSTER] [--log-level {debug,info,warning,error,critical}] [--log-path LOG_PATH]

ceph-volume: Deploy Ceph OSDs using different device technologies like lvm or
physical disks.

Log Path: /var/log/ceph
Ceph Conf: /etc/ceph/ceph.conf

Available subcommands:

lvm Use LVM and LVM-based technologies to deploy OSDs
simple Manage already deployed OSDs with ceph-volume
raw Manage single-device OSDs on raw block devices
inventory Get this nodes available disk inventory
activate Activate an OSD
drive-group
    Deploy OSDs according to a drive groups specification.

    The DriveGroup specification must be passed in json.
    It can either be (preference in this order)
      - in a file, path passed as a positional argument
      - read from stdin, pass "-" as a positional argument
      - a json string passed via the --spec argument

    Either the path postional argument or --spec must be specifed.

options:
  -h, --help show this help message and exit
  --cluster CLUSTER Cluster name (defaults to "ceph")
  --log-level {debug,info,warning,error,critical}
                        Change the file log level (defaults to debug)
  --log-path LOG_PATH Change the log path (defaults to /var/log/ceph)

confirmed LGTM

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 19.2.0~git20240301.4c76c50-0ubuntu6.1

---------------
ceph (19.2.0~git20240301.4c76c50-0ubuntu6.1) noble; urgency=medium

  [ Luciano Lo Giudice]
  * d/control: Add python3-{packaging,ceph-common} to (Build-)Depends
    as these are undocumented/detected runtime dependencies in
    ceph-volume (LP: #2064717).

  [ James Page ]
  * d/cephadm.install: Install cephadmlib Python module which the
    cephadm script uses (LP: #2063456).
  * d/control: Update Vcs-* to point to Launchpad for Ubuntu packaging.
  * d/p/mgr-distutils.patch: Directly use vendored distutils from
    setuptools for Python that runs in the mgr daemon (LP: #2065867).

 -- James Page <email address hidden> Sat, 18 May 2024 12:40:36 +0200

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Peter Sabaini (peter-sabaini)]

The recent 18.2.4 SRU introduced a regression here, python3-packaging is now required for reef as well
https://github.com/ceph/ceph/commit/956305eb5caf323cfadb772a9a1f910a90aa7740

ubuntu@juju-69234d-0:~$ ceph-volume -h
Traceback (most recent call last):
  File "/usr/sbin/ceph-volume", line 33, in <module>
    sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')())
  File "/usr/sbin/ceph-volume", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 9, in <module>
    from ceph_volume import log, devices, configuration, conf, exceptions, terminal, inventory, drive_group, activate
  File "/usr/lib/python3/dist-packages/ceph_volume/devices/__init__.py", line 1, in <module>
    from . import lvm, simple, raw # noqa
  File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/__init__.py", line 1, in <module>
    from .main import LVM # noqa
  File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 4, in <module>
    from . import activate
  File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/activate.py", line 9, in <module>
    from ceph_volume.util import encryption as encryption_utils
  File "/usr/lib/python3/dist-packages/ceph_volume/util/encryption.py", line 9, in <module>
    from packaging import version
ModuleNotFoundError: No module named 'packaging'
ubuntu@juju-69234d-0:~$ dpkg -l |grep ceph-common
ii ceph-common 18.2.4-0ubuntu1~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster
ii python3-ceph-common 18.2.4-0ubuntu1~cloud0 all Python 3 utility libraries for Ceph
ubuntu@juju-69234d-0:~$ apt-cache policy ceph-common
ceph-common:
  Installed: 18.2.4-0ubuntu1~cloud0
  Candidate: 18.2.4-0ubuntu1~cloud0
  Version table:
 *** 18.2.4-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu jammy-updates/bobcat/main amd64 Packages
        100 /var/lib/dpkg/status
     17.2.7-0ubuntu0.22.04.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     17.2.6-0ubuntu0.22.04.3 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     17.1.0-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
ubuntu@juju-69234d-0:~$ dpkg -l |grep python.*packaging
ubuntu@juju-69234d-0:~$ dpkg -l |...

[Peter Sabaini (peter-sabaini)]

SRU link FTR
https://bugs.launchpad.net/cloud-archive/+bug/2075358

[Trent Lloyd (lathiat)]

I discovered this issue myself (for Reef, 18.2.4) today when running the zaza integration test for charm-glance-simplestreams-sync against jammy-bobcat.

According to the SRU, the charm-ceph-osd tests were run, and the package version was verified. The question is, why did those tests not catch this?

When I run the zaza test for charm-ceph-osd in the stable/reef branch, it also fails with the issue. I see the exact same version installed as reported in the SRU bug:
juju ssh ceph-osd/0 sudo ceph -v
ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)

So I am really curious to understand why the test passed previously.

[Trent Lloyd (lathiat)]

I suspect the reason this was not picked up in the SRU test, is possibly related to code from the Squid charm being used in the test instead of the Reef Charm.

The squid charm merged a "tactical fix" to manually install python3-packaging in this change: https://review.opendev.org/c/openstack/charm-ceph-osd/+/918992

But not for Reef, it was originally proposed but abandoned when we thought it wasn't needed for Reef: https://review.opendev.org/c/openstack/charm-ceph-osd/+/919794

The Squid charm supports running/installing reef because you're expected to upgrade the charm before Ceph itself, to orchestrate an upgrade. So both the Reef and Squid charm branches have a test for Reef (tests/bundles/jammy-bobcat).

IMHO merging this charm change was a bad idea and it should be reverted once all the packages are fixed. The package should simply have been fixed immediately in the first instance.

While I can appreciate this might have been done as a stop-gap to get the charm CI working while the issue was not yet fixed in an SRU, the problem is that we are using the charm tests to verify the SRU of the Ubuntu package which is potentially (and actually, even in the cloud-archive) used by people without the charms, so this is likely to hide such an issue as it did here. It also means we don't have a functional test to actually test that the issue is fixed, both in the Reef and Squid SRUs.

I can't quite figure out exactly how this test was done though. The original message said it was tested with the ceph-osd charm tests, but the zaza.openstack.charm_tests.ceph.tests.CephPrometheusTest test listed in the output only exists in charm-ceph-mon.

Then those tests all use the reef branch of the charm.. I am guessing maybe since we had to test with bobcat-proposed that the squid bundle from the squid branch was used but with openstack-origin overriden to bobcat-proposed or something like that?

Luciano: Would be great if you can clarify/reverse engineer exactly how you managed to run that test so we can learn for next time. I also wonder if we'd be better using charmed-openstack-tester or something like that, instead of purely the charm-ceph-mon tests, for validating SRUs?

A few possible lessons for future SRU verificaiton:
- We need to ensure we verify SRUs with all GIT/charmhub branches of the charm that support a release. So generally that would be both the matching and newer version. It's not sufficient to check with only one of those.
- Thinking more about the charm users that are the majority, I think ideally we also need to run both the charmhub stable AND candidate branches for both of those releases. Currently the test bundles use the '/edge' channel (which maps to candidate) and would only test the candidate charm, and won't show up if we're about to release a package that is broken wtith the stable charms. Especially for the latest release of Ceph, due to the Solutions QA process, sometimes the stable channel is lagging the edge channel by weeks or even months. So this is not unlikely.
- Using the charm tests to verify the Ubuntu package in general has some limits, in that it may miss scenarios that would still effect non-charm users. I am not...

[Luciano Lo Giudice (lmlogiudice)]

I think the process I used to test the SRU caused the issue to not be present in the tests, because I had a model up and running and then used `add-apt-repository` and `apt update` in the Ceph units, which can cause additional packages to be installed. The reason for doing this is that at the time of running the tests, Serverstack wasn't working reliably (still isn't), and so I made the choice of using an existing (non-clean) model to run the tests.

WRT to the tests being run, they are taken from the ceph-mon bundle. It was deemed enough as long as the Ceph units have the updated packages, since they encompass enough functionality (the Openstack tester tests run pretty much the same thing iirc).

I'd be delighted to have a standardized test plan for SRU's, and it's something I've looked at in the past, but never got to it due to lack of time. The process can feel a bit informal at times (I've had to make up test plans when Juju didn't support the needed series, for example), so any changes that formalize the process would be a welcome addition.

[Kellen Renshaw (krenshaw)]

Will this fix be made available in Bobcat? That is the only place to consume Reef AFAIK.

[Kellen Renshaw (krenshaw)]

This issue affects Reef in the Bobcat repo, a working 18.2.0 install broke when upgraded to the 18.2.4 packages. Workaround is to manually install python3-packaging.

[Trent Lloyd (lathiat)]

OK, well we learnt now that only upgrading and not doing a fresh deployment, and only doing the ceph-mon tests is not enough. Indeed, let's work on a more concrete/full test plan. I have some strong thoughts for that so will discuss with you and Utkarsh, etc.

Luciano: In the mean time, can you prioritise a ceph 18.2.4 SRU to fix this regression please (not the charm fix)? Like this week preferably. We have users actually using this and affected by it.

[Trent Lloyd (lathiat)]

Note: This issue is more impactful than I initially realised. I was thinking it was mainly an issue on initial deploy, but if you upgrade your deployment to 18.2.4 and then reboot a node, the OSDs won't start, because the ceph-volume tool is needed to activate the OSDs.

[James Page (james-page)]

This has been fixed upstream in the main branch:

 https://github.com/ceph/ceph/commit/729fd8e25ff2bfbcf99790d6cd08489d1c4e2ede

prepping update for bobcat now.

[James Page (james-page)]

Hello Peter, or anyone else affected,

Accepted ceph into bobcat-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:bobcat-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-bobcat-needed to verification-bobcat-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-bobcat-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Luciano Lo Giudice (lmlogiudice)]

I can confirm that ceph-volume works correctly now:

```
root@client:~# ceph-volume -h
usage: ceph-volume [-h] [--cluster CLUSTER] [--log-level {debug,info,warning,error,critical}] [--log-path LOG_PATH]

ceph-volume: Deploy Ceph OSDs using different device technologies like lvm or
physical disks.

Log Path: /var/log/ceph
Ceph Conf: Unable to load expected Ceph config at: /etc/ceph/ceph.conf

Available subcommands:

lvm Use LVM and LVM-based technologies to deploy OSDs
simple Manage already deployed OSDs with ceph-volume
raw Manage single-device OSDs on raw block devices
inventory Get this nodes available disk inventory
activate Activate an OSD
drive-group
    Deploy OSDs according to a drive groups specification.

    The DriveGroup specification must be passed in json.
    It can either be (preference in this order)
      - in a file, path passed as a positional argument
      - read from stdin, pass "-" as a positional argument
      - a json string passed via the --spec argument

    Either the path postional argument or --spec must be specifed.

options:
  -h, --help show this help message and exit
  --cluster CLUSTER Cluster name (defaults to "ceph")
  --log-level {debug,info,warning,error,critical}
                        Change the file log level (defaults to debug)
  --log-path LOG_PATH Change the log path (defaults to /var/log/ceph)
```

Ceph version:

```
root@client:~# ceph -v
ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
```

Apt policy:

```
root@client:~# apt policy ceph
ceph:
  Installed: 18.2.4-0ubuntu1~cloud1
  Candidate: 18.2.4-0ubuntu1~cloud1
  Version table:
 *** 18.2.4-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu jammy-proposed/bobcat/main amd64 Packages
        100 /var/lib/dpkg/status
     17.2.7-0ubuntu0.22.04.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     17.2.6-0ubuntu0.22.04.3 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     17.1.0-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
root@client:~# apt policy ceph-volume
ceph-volume:
  Installed: 18.2.4-0ubuntu1~cloud1
  Candidate: 18.2.4-0ubuntu1~cloud1
  Version table:
 *** 18.2.4-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu jammy-proposed/bobcat/main amd64 Packages
        100 /var/lib/dpkg/status
     17.2.7-0ubuntu0.22.04.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     17.2.6-0ubuntu0.22.04.3 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     17.1.0-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
```

[James Page (james-page)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package ceph - 18.2.4-0ubuntu1~cloud1
---------------

 ceph (18.2.4-0ubuntu1~cloud1) jammy-bobcat; urgency=medium
 .
   * d/control: Add python3-{packaging,ceph-common} to (Build-)Depends
     as these are undocumented/detected runtime dependencies in
     ceph-volume (LP: #2064717).

[James Page (james-page)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package ceph - 18.2.4-0ubuntu1~cloud1
---------------

 ceph (18.2.4-0ubuntu1~cloud1) jammy-bobcat; urgency=medium
 .
   * d/control: Add python3-{packaging,ceph-common} to (Build-)Depends
     as these are undocumented/detected runtime dependencies in
     ceph-volume (LP: #2064717).

[Ponnuvel Palaniyappan (pponnuvel)]

bobcat (Reef) is EOL, so won't fix for it.

[Ghadi Rahme (ghadi-rahme)]

Debdiff for Jammy